Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-915

Reduce Logstash core limit to 1 from 3 until LOG-LS and AAI-CS perf issue on the same VM is determined


    • Icon: Task Task
    • Resolution: Unresolved
    • Icon: High High
    • Frankfurt Release
    • None

      from aai team
      "hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling) "

      Some VM architectures are using 4 vCores per VM - not 8 - in this case the current vCore limit of 3 for logstash is insufficient - reducing to 1 for now
      We will see some log processing hit but at least the vm ls and aai-cs are collocated won't saturate
      Actually I remember an issue with LOG-376 where running with less than 3 cores causes logstash not to deploy properly - I may just put the ReplicaSet to 1 or 2 from the current 5 (it is no longer a DaemonSet at 13)

      AAI-cassandra will also periodically reach 7 cores of 8 but only in spikes

      Discussion with @Sanjay Agraharam and pau2882 on checking how cassandra is running on the vm and if debug levels are on should be verified

      Proposal to use labelling to split aai-cs and ls - no DaemonSet in this case

            pau2882 pau2882
            michaelobrien michaelobrien
            0 Vote for this issue
            1 Start watching this issue