-
Bug
-
Resolution: Won't Do
-
Medium
-
None
-
Casablanca Release, Casablanca Maintenance Release
-
OOM deployment in windriver lab
In ONAP full deployment, log project uses a lot of CPU power and push node CPU utilization over 90% in 2+12 cluster (8vCPU/16GB for each node). If SDC pods happen to run on the same node as log pods, SDC GUI will experience very slow response.
Here is the top command output comparing node CPU before and after deleting log project from ONAP deployment. The measurement is done with light traffic in ONAP (model distribution), the similar issue was also observed with no ONAP workload.
root@oom-rancher:~/oom/kubernetes/robot# kubectl -n onap top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
oom-etcd-1 269m 13% 1563Mi 40%
oom-etcd-2 432m 21% 1576Mi 40%
oom-etcd-3 310m 15% 1658Mi 43%
oom-k8s-01 607m 7% 12112Mi 75%
oom-k8s-02 7329m 91% 15024Mi 94%
oom-k8s-03 575m 7% 2792Mi 17%
oom-k8s-04 7236m 90% 11296Mi 70%
oom-k8s-05 3020m 37% 10936Mi 68%
oom-k8s-06 3789m 47% 10038Mi 62%
oom-k8s-07 2577m 32% 13164Mi 82%
oom-k8s-08 1140m 14% 4975Mi 31%
oom-k8s-09 5079m 63% 13763Mi 86%
oom-k8s-10 893m 11% 8370Mi 52%
oom-k8s-11 5705m 71% 7510Mi 47%
oom-k8s-12 1900m 23% 11200Mi 70%
oom-orch-1 942m 47% 1080Mi 28%
oom-orch-2 432m 21% 2671Mi 69%
root@oom-rancher:~/oom/kubernetes/robot# helm delete dev-log --purge
root@oom-rancher:~/oom/kubernetes/robot# kubectl -n onap top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
oom-etcd-1 252m 12% 1524Mi 39%
oom-etcd-2 474m 23% 1618Mi 42%
oom-etcd-3 273m 13% 1706Mi 44%
oom-k8s-01 561m 7% 12146Mi 76%
oom-k8s-02 5165m 64% 14816Mi 92%
oom-k8s-03 532m 6% 2804Mi 17%
oom-k8s-04 5075m 63% 10246Mi 64%
oom-k8s-05 1225m 15% 10416Mi 65%
oom-k8s-06 614m 7% 9545Mi 59%
oom-k8s-07 2340m 29% 13188Mi 82%
oom-k8s-08 1112m 13% 4988Mi 31%
oom-k8s-09 5227m 65% 11367Mi 71%
oom-k8s-10 861m 10% 7768Mi 48%
oom-k8s-11 5250m 65% 7433Mi 46%
oom-k8s-12 1380m 17% 11169Mi 70%
oom-orch-1 639m 31% 1085Mi 28%
oom-orch-2 436m 21% 2709Mi 70%
- duplicates
-
LOG-376 Logstash full saturation of 8 cores with AAI deployed on one of the quad 8 vCore vms for 30 logs/sec - up replicaSet 1 to 3 or use DaemonSet
- Closed