-
Bug
-
Resolution: Done
-
Low
-
Casablanca Release
this single test is not scientific enough to lay the blame on Kubernetes 1.11 yet - will update it later after I test more configs
Ran into this one verifying 1.6.22 (integration is using a higher version than the 1.6.18 rancher version that supplies k8s 1.10) - the cluster is also used for my pairwise testing work for Logging RC2
Ran 2 identical 450G vms but installed k8s 1.11 (the problem vm) and 1.10 on the other (Standard)
The system is idling 1 hour after a full install
the 2nd one the issue under a consistent 8-9.23 vCores
26386 root 20 0 9.921g 814456 16456 S 923.7 0.2 864:31.86 java -Xmx1024m -DAPP_HOME=/opt/app/champ-service -DCONFIG_HOME=/opt/app/champ-service/appconfig -Dlogging.config=/opt/app/champ-service/bundleconfi+
in this case the pod is still starting up
ubuntu@a-osn-cd:~$ kubectl get pods --all-namespaces | grep champ
onap onap-aai-aai-champ-54f7986b6b-ksrz7 1/2 Running 0 3h
ran
ubuntu@a-osn-cd:~$ kubectl delete pod onap-aai-aai-champ-54f7986b6b-ksrz7 -n onap --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "onap-aai-aai-champ-54f7986b6b-ksrz7" deleted
fixed
ubuntu@a-osn-cd:~$ kubectl get pods --all-namespaces | grep champ
onap onap-aai-aai-champ-54f7986b6b-trt65 2/2 Running 0 4m
working system
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
45727 ubuntu 20 0 21.157g 904764 21528 S 291.8 0.2 283:17.55 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jru+
7721 root 20 0 7454928 409080 16992 S 100.0 0.1 133:33.76 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xms50m -Xmx128m -classpath /home/uui/bin/..//:/home/uui/bin/..//usecase-ui-server.jar -jar /home/uui/bi+
64510 root 20 0 2365396 79412 17996 S 100.0 0.0 0:01.10 /usr/bin/java -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -ea -cp /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra+
15060 root 20 0 9888056 769728 16480 S 99.1 0.2 108:18.28 java -Xmx1024m -DAPP_HOME=/opt/app/spike -DCONFIG_HOME=/opt/app/spike/config -Dlogging.config=/opt/app/spike/bundleconfig/etc/logback.xml -DKEY_STO+
22521 root 20 0 36.901g 105356 18340 S 99.1 0.0 92:17.21 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dhttps.protocols=TLSv1.1,TLSv1.2 -jar /opt/onap/sdnc/ueb-listener/lib/ueb-listener-0.3.1.jar -Dlog4j.co+
29418 root 20 0 5033584 79368 7288 S 81.8 0.0 1:14.88 /usr/bin/python /root/ready.py --container-name dmaap-dr-prov
65166 root 20 0 2142696 71088 17928 S 80.0 0.0 0:00.88 /usr/bin/java -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -ea -cp /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/+
33521 root 20 0 3908756 432004 52312 S 51.8 0.1 75:43.67 kubelet --kubeconfig=/etc/kubernetes/ssl/kubeconfig --register-node=true --cloud-provider=rancher --allow-privileged=true --healthz-bind-address=0.+
64989 systemd+ 20 0 103404 82080 26488 S 50.0 0.0 0:00.55 /consul/bin/kubectl -n onap get pod
51743 root 20 0 46.627g 1.993g 20536 S 41.8 0.5 17:58.26 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
31264 ubuntu 20 0 17.345g 1.099g 19976 S 36.4 0.3 47:36.11 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=tru+
35522 root 20 0 22.826g 166460 4272 S 34.5 0.0 28:42.96 conductor-controller: Template Translator worker(0)
35516 root 20 0 22.826g 164616 4264 S 33.6 0.0 28:02.41 conductor-controller: Music-RPC Server, topic(controller) worker(0)
60524 root 20 0 22.825g 164128 4716 S 32.7 0.0 26:50.40 conductor-data: Music-RPC Server, topic(data) worker(0)
22905 999 20 0 3706312 1.314g 26948 S 26.4 0.3 24:02.26 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
34770 999 20 0 3718096 1.321g 27008 S 24.5 0.3 24:58.19 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
39834 999 20 0 3731752 1.318g 28004 S 20.9 0.3 25:37.19 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
51549 root 20 0 46.670g 1.891g 20472 S 20.9 0.4 18:55.53 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
24357 root 20 0 41.115g 851320 27536 S 18.2 0.2 80:12.62 /usr/bin/dockerd -H fd://
ubuntu@a-cd-cd:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.3-rancher1", GitCommit:"f6320ca7027d8244abb6216fbdb73a2b3eb2f4f9", GitTreeState:"clean", BuildDate:"2018-05-29T22:28:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26386 root 20 0 9.921g 814456 16456 S 923.7 0.2 864:31.86 java -Xmx1024m -DAPP_HOME=/opt/app/champ-service -DCONFIG_HOME=/opt/app/champ-service/appconfig -Dlogging.config=/opt/app/champ-service/bundleconfi+
13931 ubuntu 20 0 21.029g 907384 21932 S 286.0 0.2 343:43.21 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jru+
26818 root 20 0 7454928 403140 16836 S 98.2 0.1 171:59.50 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xms50m -Xmx128m -classpath /home/uui/bin/..//:/home/uui/bin/..//usecase-ui-server.jar -jar /home/uui/bi+
65464 root 20 0 9954620 762884 16736 S 97.4 0.2 126:10.20 java -Xmx1024m -DAPP_HOME=/opt/app/spike -DCONFIG_HOME=/opt/app/spike/config -Dlogging.config=/opt/app/spike/bundleconfig/etc/logback.xml -DKEY_STO+
61949 root 20 0 36.901g 110100 18264 S 96.5 0.0 102:35.31 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dhttps.protocols=TLSv1.1,TLSv1.2 -jar /opt/onap/sdnc/ueb-listener/lib/ueb-listener-0.3.1.jar -Dlog4j.co+
58956 root 20 0 5033924 79764 7408 S 57.0 0.0 1:36.20 /usr/bin/python /root/ready.py --container-name dmaap-dr-prov
46237 root 20 0 5204460 608248 58008 S 42.1 0.1 95:40.12 kubelet --kubeconfig=/etc/kubernetes/ssl/kubeconfig --register-node=true --cloud-provider=rancher --allow-privileged=true --healthz-bind-address=0.+
17127 ubuntu 20 0 17.278g 1.114g 19668 S 33.3 0.3 55:35.70 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=tru+
41218 root 20 0 46.474g 1.898g 20496 S 31.6 0.4 23:42.57 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
43813 root 20 0 22.825g 174844 4884 S 30.7 0.0 31:42.74 conductor-data: Music-RPC Server, topic(data) worker(0)
27020 root 20 0 22.826g 167824 5356 S 29.8 0.0 33:22.06 conductor-controller: Template Translator worker(0)
27016 root 20 0 22.826g 165956 5348 S 28.9 0.0 32:53.05 conductor-controller: Music-RPC Server, topic(controller) worker(0)
56427 999 20 0 3726104 1.316g 29480 S 27.2 0.3 30:46.20 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
63668 999 20 0 3710512 1.315g 28852 S 23.7 0.3 30:30.57 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
36686 root 20 0 23.460g 288844 7604 S 20.2 0.1 21:27.43 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docke+
65289 997 20 0 2994296 86312 6844 S 20.2 0.0 21:14.27 /usr/lib64/erlang/erts-6.3/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /us+
36665 root 20 0 41.233g 1.015g 27420 S 19.3 0.2 89:19.35 /usr/bin/dockerd -H fd://
36305 root 20 0 46.686g 1.876g 20336 S 18.4 0.4 23:50.15 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
6367 999 20 0 3704408 1.306g 28456 S 16.7 0.3 30:02.00 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
32126 root 20 0 2140324 31388 16488 S 13.2 0.0 0:00.15 /usr/bin/java -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -ea -cp /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/+
46276 root 20 0 1176136 967124 61496 S 13.2 0.2 44:59.92 kube-apiserver --storage-backend=etcd2 --storage-media-type=application/json --service-cluster-ip-range=10.43.0.0/16 --etcd-servers=https://etcd.ku+
18971 syslog 20 0 33.935g 121536 5720 S 11.4 0.0 17:17.04 /usr/lib/erlang/erts-8.2.1/bin/beam.smp -W w -A 1024 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname er+
20542 ubuntu 20 0 34708 6020 2912 R 10.5 0.0 0:01.47 top
36516 999 20 0 2531264 475140 39916 S 8.8 0.1 7:39.10 java -Dcassandra.consistent.rangemovement=false -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThread+
32125 999 20 0 48664 16812 6476 R 7.9 0.0 0:00.09 python /usr/bin/cqlsh.py -u cassandra -p cassandra -f /docker-entrypoint-initdb.d/zzz_portalsdk.cql
ubuntu@a-osn-cd:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.2-rancher1", GitCommit:"386fc0481824878c3dacf830ec605776f112467e", GitTreeState:"clean", BuildDate:"2018-08-20T18:29:45Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
