Uploaded image for project: 'Active and Available Inventory'
  1. Active and Available Inventory
  2. AAI-1860

champ service during startup using 9+ vCores until restarted

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Low
    • Resolution: Done
    • Affects Version/s: Casablanca Release
    • Fix Version/s: Dublin Release
    • Labels:

      Description

      this single test is not scientific enough to lay the blame on Kubernetes 1.11 yet - will update it later after I test more configs
      Ran into this one verifying 1.6.22 (integration is using a higher version than the 1.6.18 rancher version that supplies k8s 1.10) - the cluster is also used for my pairwise testing work for Logging RC2

      Ran 2 identical 450G vms but installed k8s 1.11 (the problem vm) and 1.10 on the other (Standard)

      The system is idling 1 hour after a full install

      the 2nd one the issue under a consistent 8-9.23 vCores
      26386 root      20   0  9.921g 814456  16456 S 923.7  0.2 864:31.86 java -Xmx1024m -DAPP_HOME=/opt/app/champ-service -DCONFIG_HOME=/opt/app/champ-service/appconfig -Dlogging.config=/opt/app/champ-service/bundleconfi+
      in this case the pod is still starting up
      ubuntu@a-osn-cd:~$ kubectl get pods --all-namespaces | grep champ
      onap          onap-aai-aai-champ-54f7986b6b-ksrz7                            1/2       Running            0          3h
      ran
      ubuntu@a-osn-cd:~$ kubectl delete pod onap-aai-aai-champ-54f7986b6b-ksrz7 -n onap --grace-period=0 --force
      warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
      pod "onap-aai-aai-champ-54f7986b6b-ksrz7" deleted
      
      fixed
      ubuntu@a-osn-cd:~$ kubectl get pods --all-namespaces | grep champ
      onap          onap-aai-aai-champ-54f7986b6b-trt65                            2/2       Running            0          4m
      
      
      
      working system
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                             
      45727 ubuntu    20   0 21.157g 904764  21528 S 291.8  0.2 283:17.55 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jru+
       7721 root      20   0 7454928 409080  16992 S 100.0  0.1 133:33.76 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xms50m -Xmx128m -classpath /home/uui/bin/..//:/home/uui/bin/..//usecase-ui-server.jar -jar /home/uui/bi+
      64510 root      20   0 2365396  79412  17996 S 100.0  0.0   0:01.10 /usr/bin/java -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -ea -cp /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra+
      15060 root      20   0 9888056 769728  16480 S  99.1  0.2 108:18.28 java -Xmx1024m -DAPP_HOME=/opt/app/spike -DCONFIG_HOME=/opt/app/spike/config -Dlogging.config=/opt/app/spike/bundleconfig/etc/logback.xml -DKEY_STO+
      22521 root      20   0 36.901g 105356  18340 S  99.1  0.0  92:17.21 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dhttps.protocols=TLSv1.1,TLSv1.2 -jar /opt/onap/sdnc/ueb-listener/lib/ueb-listener-0.3.1.jar -Dlog4j.co+
      29418 root      20   0 5033584  79368   7288 S  81.8  0.0   1:14.88 /usr/bin/python /root/ready.py --container-name dmaap-dr-prov                                                                                       
      65166 root      20   0 2142696  71088  17928 S  80.0  0.0   0:00.88 /usr/bin/java -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -ea -cp /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/+
      33521 root      20   0 3908756 432004  52312 S  51.8  0.1  75:43.67 kubelet --kubeconfig=/etc/kubernetes/ssl/kubeconfig --register-node=true --cloud-provider=rancher --allow-privileged=true --healthz-bind-address=0.+
      64989 systemd+  20   0  103404  82080  26488 S  50.0  0.0   0:00.55 /consul/bin/kubectl -n onap get pod                                                                                                                 
      51743 root      20   0 46.627g 1.993g  20536 S  41.8  0.5  17:58.26 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
      31264 ubuntu    20   0 17.345g 1.099g  19976 S  36.4  0.3  47:36.11 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=tru+
      35522 root      20   0 22.826g 166460   4272 S  34.5  0.0  28:42.96 conductor-controller: Template Translator worker(0)                                                                                                 
      35516 root      20   0 22.826g 164616   4264 S  33.6  0.0  28:02.41 conductor-controller: Music-RPC Server, topic(controller) worker(0)                                                                                 
      60524 root      20   0 22.825g 164128   4716 S  32.7  0.0  26:50.40 conductor-data: Music-RPC Server, topic(data) worker(0)                                                                                             
      22905 999       20   0 3706312 1.314g  26948 S  26.4  0.3  24:02.26 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
      34770 999       20   0 3718096 1.321g  27008 S  24.5  0.3  24:58.19 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
      39834 999       20   0 3731752 1.318g  28004 S  20.9  0.3  25:37.19 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
      51549 root      20   0 46.670g 1.891g  20472 S  20.9  0.4  18:55.53 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
      24357 root      20   0 41.115g 851320  27536 S  18.2  0.2  80:12.62 /usr/bin/dockerd -H fd://                                                                                                                           
      ubuntu@a-cd-cd:~$ kubectl version
      Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
      Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.3-rancher1", GitCommit:"f6320ca7027d8244abb6216fbdb73a2b3eb2f4f9", GitTreeState:"clean", BuildDate:"2018-05-29T22:28:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
      
      
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                             
      26386 root      20   0  9.921g 814456  16456 S 923.7  0.2 864:31.86 java -Xmx1024m -DAPP_HOME=/opt/app/champ-service -DCONFIG_HOME=/opt/app/champ-service/appconfig -Dlogging.config=/opt/app/champ-service/bundleconfi+
      13931 ubuntu    20   0 21.029g 907384  21932 S 286.0  0.2 343:43.21 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jru+
      26818 root      20   0 7454928 403140  16836 S  98.2  0.1 171:59.50 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xms50m -Xmx128m -classpath /home/uui/bin/..//:/home/uui/bin/..//usecase-ui-server.jar -jar /home/uui/bi+
      65464 root      20   0 9954620 762884  16736 S  97.4  0.2 126:10.20 java -Xmx1024m -DAPP_HOME=/opt/app/spike -DCONFIG_HOME=/opt/app/spike/config -Dlogging.config=/opt/app/spike/bundleconfig/etc/logback.xml -DKEY_STO+
      61949 root      20   0 36.901g 110100  18264 S  96.5  0.0 102:35.31 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dhttps.protocols=TLSv1.1,TLSv1.2 -jar /opt/onap/sdnc/ueb-listener/lib/ueb-listener-0.3.1.jar -Dlog4j.co+
      58956 root      20   0 5033924  79764   7408 S  57.0  0.0   1:36.20 /usr/bin/python /root/ready.py --container-name dmaap-dr-prov                                                                                       
      46237 root      20   0 5204460 608248  58008 S  42.1  0.1  95:40.12 kubelet --kubeconfig=/etc/kubernetes/ssl/kubeconfig --register-node=true --cloud-provider=rancher --allow-privileged=true --healthz-bind-address=0.+
      17127 ubuntu    20   0 17.278g 1.114g  19668 S  33.3  0.3  55:35.70 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=tru+
      41218 root      20   0 46.474g 1.898g  20496 S  31.6  0.4  23:42.57 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
      43813 root      20   0 22.825g 174844   4884 S  30.7  0.0  31:42.74 conductor-data: Music-RPC Server, topic(data) worker(0)                                                                                             
      27020 root      20   0 22.826g 167824   5356 S  29.8  0.0  33:22.06 conductor-controller: Template Translator worker(0)                                                                                                 
      27016 root      20   0 22.826g 165956   5348 S  28.9  0.0  32:53.05 conductor-controller: Music-RPC Server, topic(controller) worker(0)                                                                                 
      56427 999       20   0 3726104 1.316g  29480 S  27.2  0.3  30:46.20 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
      63668 999       20   0 3710512 1.315g  28852 S  23.7  0.3  30:30.57 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
      36686 root      20   0 23.460g 288844   7604 S  20.2  0.1  21:27.43 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docke+
      65289 997       20   0 2994296  86312   6844 S  20.2  0.0  21:14.27 /usr/lib64/erlang/erts-6.3/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /us+
      36665 root      20   0 41.233g 1.015g  27420 S  19.3  0.2  89:19.35 /usr/bin/dockerd -H fd://                                                                                                                           
      36305 root      20   0 46.686g 1.876g  20336 S  18.4  0.4  23:50.15 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.jul+
       6367 999       20   0 3704408 1.306g  28456 S  16.7  0.3  30:02.00 java -Dcassandra.consistent.rangemovement=false -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+Hea+
      32126 root      20   0 2140324  31388  16488 S  13.2  0.0   0:00.15 /usr/bin/java -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -ea -cp /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/+
      46276 root      20   0 1176136 967124  61496 S  13.2  0.2  44:59.92 kube-apiserver --storage-backend=etcd2 --storage-media-type=application/json --service-cluster-ip-range=10.43.0.0/16 --etcd-servers=https://etcd.ku+
      18971 syslog    20   0 33.935g 121536   5720 S  11.4  0.0  17:17.04 /usr/lib/erlang/erts-8.2.1/bin/beam.smp -W w -A 1024 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname er+
      20542 ubuntu    20   0   34708   6020   2912 R  10.5  0.0   0:01.47 top                                                                                                                                                 
      36516 999       20   0 2531264 475140  39916 S   8.8  0.1   7:39.10 java -Dcassandra.consistent.rangemovement=false -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThread+
      32125 999       20   0   48664  16812   6476 R   7.9  0.0   0:00.09 python /usr/bin/cqlsh.py -u cassandra -p cassandra -f /docker-entrypoint-initdb.d/zzz_portalsdk.cql                                                 
      ubuntu@a-osn-cd:~$ kubectl version
      Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
      Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.2-rancher1", GitCommit:"386fc0481824878c3dacf830ec605776f112467e", GitTreeState:"clean", BuildDate:"2018-08-20T18:29:45Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
      
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              Assignee:
              jimmydot James Forsyth
              Reporter:
              michaelobrien Michael O'Brien
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: