Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-376

Logstash full saturation of 8 cores with AAI deployed on one of the quad 8 vCore vms for 30 logs/sec - up replicaSet 1 to 3 or use DaemonSet

    XMLWordPrintable

Details

    Description

      20190109
      from aai team
      https://wiki.onap.org/display/DW/2019-01-17+AAI+Developers+Meeting+Open+Agenda
      "hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling) "

      20181017: update - reopen or re-raise AAI/Logstash specific JIRA for Dublin - in LOG-707 - as the issue is more of an AAI to logstash issue

      Need to find out which container - it is the logstash one - mine

      12588 ubuntu 20 0 6397972 699192 22012 S 578.1 1.1 567:55.27 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbo+

      http://jenkins.onap.info/job/oom-cd-master/2897/console
      find out what the reason is for the saturation - is it excessive logs from example the cluster heartbeat from all the db clusters
      or a misconfiguration of the resource section

      it looks like logs are still being processed up to 4 min after they come into logstash - getting an average of 200-400 logs per 30 sec on

      http://18.188.238.244:30253/app/kibana#/discover?_g=(refreshInterval:('$$hashKey':'object:1198',display:'5%20seconds',pause:!f,section:1,value:5000),time:(from:now-15m,mode:quick,to:now))&_a=(columns:!(_source),index:'logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc))

      Attachments

        1. Screenshot 2018-05-08 01.34.16.png
          Screenshot 2018-05-08 01.34.16.png
          400 kB
        2. Screenshot 2018-05-08 19.04.25.png
          Screenshot 2018-05-08 19.04.25.png
          1.20 MB
        3. Screenshot 2018-05-15 15.27.26.png
          Screenshot 2018-05-15 15.27.26.png
          1.30 MB
        4. Screenshot 2018-05-16 18.08.35.png
          Screenshot 2018-05-16 18.08.35.png
          141 kB
        5. Screenshot 2018-05-16 18.15.09.png
          Screenshot 2018-05-16 18.15.09.png
          936 kB
        6. Screenshot 2018-05-16 18.33.08.png
          Screenshot 2018-05-16 18.33.08.png
          499 kB
        7. Screenshot 2018-05-16 18.33.20.png
          Screenshot 2018-05-16 18.33.20.png
          139 kB
        8. Screenshot 2018-05-16 18.34.03.png
          Screenshot 2018-05-16 18.34.03.png
          140 kB
        9. Screenshot 2018-05-16 18.35.24.png
          Screenshot 2018-05-16 18.35.24.png
          138 kB
        10. Screenshot 2018-05-18 08.07.18.png
          Screenshot 2018-05-18 08.07.18.png
          431 kB
        11. Screenshot 2018-05-28 01.26.27.png
          Screenshot 2018-05-28 01.26.27.png
          120 kB
        12. Screenshot 2018-05-29 07.39.02.png
          Screenshot 2018-05-29 07.39.02.png
          643 kB
        13. Screenshot 2018-05-29 07.39.13.png
          Screenshot 2018-05-29 07.39.13.png
          758 kB
        14. Screenshot 2018-05-30 13.26.34.png
          Screenshot 2018-05-30 13.26.34.png
          1012 kB
        15. Screenshot 2018-05-30 17.45.00.png
          Screenshot 2018-05-30 17.45.00.png
          489 kB
        16. Screenshot 2018-05-30 17.45.46.png
          Screenshot 2018-05-30 17.45.46.png
          1.20 MB
        17. Screenshot 2018-05-30 18.33.07.png
          Screenshot 2018-05-30 18.33.07.png
          1.16 MB
        18. Screenshot 2018-05-30 18.34.23.png
          Screenshot 2018-05-30 18.34.23.png
          40 kB
        19. Screenshot 2018-05-30 18.52.11.png
          Screenshot 2018-05-30 18.52.11.png
          1.23 MB
        20. Screenshot 2018-05-30 18.53.37.png
          Screenshot 2018-05-30 18.53.37.png
          1.35 MB
        21. values.yaml
          3 kB

        Issue Links

          blocks

          Task - A task that needs to be done. LOG-915 Reduce Logstash core limit to 1 from 3 until LOG-LS and AAI-CS perf issue on the same VM is determined

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Open

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LOG-841 Logstash container - use a label to distribute the ReplicaSet instead of DaemonSet

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OOM-1793 High CPU observed by Cassandra liveness/readiness probe

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. LOG-435 logstash consuming high CPU on host affects other components - pre onap-wide resource allocations

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-181 Use DaemonSets for logging - Logstash

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LOG-258 S3P ELK stack performance and clustering

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Task - A task that needs to be done. LOG-860 Lower logstash replicaSet from 5 to 3 and set to 1 in dev.yaml

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LOG-380 Platform Maturity: Performance, Stability, Resiliency, Scalability

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OOM-1120 values.yaml resource limits for cpu/ram have no effect - workaround is use JVM threadpool setting

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. LOG-862 Log project pods take a lot of CPU in full ONAP deployment

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-877 S3P: Logging streaming/format alignment for dublin - China Telecom, Deutsche Telekom, Vodafone

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. LOG-256 log-kibana deployment is timing out periodically

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-876 S3P: Logging for Core Service/VNF state and transition model - Deutsche Telekom and Vodafone

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. AAF-273 Cassandra pod running over 8G heap - or 10% of ONAP ram (for 135 other pods on 256G 4 node cluster)

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. AAI-2092 aai-resources does excessive amounts of logging

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LOG-157 S3P Events, Metrics, Monitoring

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. OOM-758 Common Mariadb Galera Helm Chart to be reused by many applications

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. OOM-927 Need a production grade configuration override file of ONAP deployment

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Sub-task - The sub-task of the issue OOM-1153 Resource Limits for log

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Task - A task that needs to be done. OOM-2794 Upgrade logstash image

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. VVP-130 DEV mode dev.yaml override for all remaining ReplicaSet counts still above 1

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. AAI-1860 champ service during startup using 9+ vCores until restarted

          • Low - It won't cause any major break-down of the system. It does not result in the termination of services, does not impact usability of the system and the desired results can be easily obtained by a workaround. Fix can be deferred until after more serious defect have been fixed.
          • Closed
          links to

          Web Link https://gerrit.onap.org/r/#/c/47941/1

          Web Link https://gerrit.onap.org/r/#/c/49553/

          Web Link https://www.elastic.co/blog/a-history-of-logstash-output-workers

          Web Link https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html

          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              michaelobrien Michael O'Brien
              michaelobrien Michael O'Brien
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: