Uploaded image for project: 'Active and Available Inventory'
  1. Active and Available Inventory
  2. AAI-2092

aai-resources does excessive amounts of logging

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Medium
    • Resolution: Done
    • Casablanca Release
    • Frankfurt Release
    • None
    • A&AI Sprint 31

    Description

      While investigating AAI-2082, checked on the docker logs for aai-graphadmin, aai-resources and aai-traversal containers.

      Found that aai-resources logs was at a staggering 550MB:

      -rw-r--r-- 1 root root 555290838 Jan 21 11:12 log-dockerlog-resources.txt

      The bulk of the log file contents were lines of these forms:

      2018-12-22T03:53:13.564+0000 AUDIT [cadi] ID AAI converted to aai@aai.onap.org
2018-12-22T03:53:13.564+0000 INFO [cadi] Authenticated: aai@aai.onap.org authenticated by cached AAF password from 172.16.0.47:40680
2018-12-22T03:53:13.660+0000 WARN [cadi] Trans: user=aai@aai.onap.org[BAth],ip=172.16.0.47,ms=95.642197,validate=0.200038,code=95.422722

2018-12-22T03:55:12.139+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk

2018-12-22T04:02:52.296+0000 INFO [cadi] AAFLurPerm removed 1 and renewed 0 expired Permissions out of 1 and removed 0 password misses out of 0

      Removing lines with the pattern "AUDIT.*ID.*converted to.*onap.org" resulted in 1555922 fewer lines.

      Removing lines with the pattern "INFO.*Authenticated:.*onap.org" resulted in 1557878 fewer lines.

      Removing lines with the pattern "WARN.*Trans:.*onap.org" resulted in 1557877 fewer lines.

      Removing lines with the pattern "INFO.*AAFLurPerm" resulted in 43135 fewer lines.

      Removing lines with the pattern "INFO.*Persist Cache" resulted in 21591 fewer lines.

      The file size after removing those lines above was:

      -rw-r--r-- 1 root root     95187 Jan 21 11:32 log-dockerlog-resource-3.txt

      which is a reduction in size of 99.9998% !!

      Might be related to LOG-376 and OOM-1542 and similar problems where pods are saturating the logging or become unstable after some time.

       

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. OOM-1793 High CPU observed by Cassandra liveness/readiness probe

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. LOG-376 Logstash full saturation of 8 cores with AAI deployed on one of the quad 8 vCore vms for 30 logs/sec - up replicaSet 1 to 3 or use DaemonSet

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. AAI-2082 aai-resources gives incorrect output when aai-cassandra has shutdown with failure

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OOM-1542 ONAP vid/aai pods crashes automatically after sometime

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. AAF-722 aaf continues to be available to aai-resources even though aaf database appears to be down

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vk250x Venkata Harish Kajur
              keong Keong Lim
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: