Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-99

Fix data loss due to no matching elasticsearch template

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Medium Medium
    • Dublin Release
    • None
    • LOG Sprint 8

      Currently data is stored in the index onaplogs-%{+YYYY.MM.dd}

      In Elastisearch the only template is:

      ubuntu@onap:~$ curl localhost:30254/_template/
      {"logstash":{"order":0,"version":50001,"template":"logstash-*","settings":{"index":{"refresh_interval":"5s"}},"mappings":{"default":{"dynamic_templates":[\\{"message_field":\\{"path_match":"message","mapping":

      {"norms":false,"type":"text"}

      ,"match_mapping_type":"string"}},\\{"string_fields":\\{"mapping":\\{"norms":false,"type":"text","fields":\\{"keyword":

      {"type":"keyword"}

      }},"match_mapping_type":"string","match":"*"}}],"_all":{"norms":false,"enabled":true},"properties":{"@timestamp":{"include_in_all":false,"type":"date"},"geoip":{"dynamic":true,"properties":{"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"}}},"@version":{"include_in_all":false,"type":"keyword"}}}},"aliases":{}}}ubuntu@onap:~$

       

      The fact that there is no matching template for the index onaplogs-%{+YYYY.MM.dd} seems to cause some data to be lost.

      This has been noted for logs that are written to rarely such as aai-ml metrics.log and audit.log.

       

      Possible solutions:

      1. Change index pattern in logstash conf to logstash-%{+YYYY.MM.dd} 
      2. Update Elasticsearch template by sending request to API after installation

            shanedaniel shanedaniel
            shanedaniel shanedaniel
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: