22:58:51 9 pending > 0 at the 362th 15 sec interval
22:58:51 
22:59:08 onap          onap-brmsgw-5b6848777c-plrpn                                      0/1       Running            0          2h        10.42.97.127    ip-10-0-0-227.us-east-2.compute.internal
22:59:08 onap          onap-drools-0                                                     0/1       Init:0/1           0          2h        10.42.75.191    ip-10-0-0-111.us-east-2.compute.internal
22:59:08 onap          onap-nexus-54ddfc9497-cghfs                                       0/1       CrashLoopBackOff   44         2h        10.42.133.4     ip-10-0-0-13.us-east-2.compute.internal
22:59:08 onap          onap-portal-app-558d99bcd4-b4mdw                                  0/2       Init:0/1           12         2h        10.42.216.217   ip-10-0-0-227.us-east-2.compute.internal
22:59:08 onap          onap-sdnc-dmaap-listener-7d69d49655-p95fz                         0/1       Init:0/1           0          2h        10.42.108.238   ip-10-0-0-80.us-east-2.compute.internal
22:59:08 onap          onap-sdnc-ueb-listener-578d65f9f5-kxkdd                           0/1       Init:0/1           0          2h        10.42.186.114   ip-10-0-0-13.us-east-2.compute.internal
22:59:08 onap          onap-so-5b8c878f95-96qjs                                          0/2       Init:0/1           13         2h        10.42.116.143   ip-10-0-0-80.us-east-2.compute.internal
22:59:08 onap          onap-so-db-5665b9bb6d-p6fjc                                       0/1       Running            31         2h        10.42.228.247   ip-10-0-0-111.us-east-2.compute.internal
22:59:08 onap          onap-uui-67cc4b6f5f-pd85d                                         0/1       ImagePullBackOff   0          2h        10.42.147.253   ip-10-0-0-13.us-east-2.compute.internal



  Type     Reason                 Age                 From                                              Message
  ----     ------                 ----                ----                                              -------
  Warning  FailedScheduling       43m (x2 over 43m)   default-scheduler                                 PersistentVolumeClaim is not bound: "onap-dcae-redis-data-onap-dcae-redis-5" (repeated 4 times)
  Normal   Scheduled              43m                 default-scheduler                                 Successfully assigned onap-dcae-redis-5 to ip-10-0-0-13.us-east-2.compute.internal
  Normal   SuccessfulMountVolume  43m                 kubelet, ip-10-0-0-13.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "onap-dcae-redis-config"
  Normal   SuccessfulMountVolume  43m                 kubelet, ip-10-0-0-13.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "onap-dcae-redis4"
  Normal   SuccessfulMountVolume  43m                 kubelet, ip-10-0-0-13.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "localtime"
  Normal   SuccessfulMountVolume  43m                 kubelet, ip-10-0-0-13.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "onap-dcae-redis-scripts"
  Normal   SuccessfulMountVolume  43m                 kubelet, ip-10-0-0-13.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-bcdgr"
  Normal   Pulling                42m (x3 over 43m)   kubelet, ip-10-0-0-13.us-east-2.compute.internal  pulling image "nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.redis-cluster-container:latest"
  Normal   Pulled                 41m (x3 over 43m)   kubelet, ip-10-0-0-13.us-east-2.compute.internal  Successfully pulled image "nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.redis-cluster-container:latest"
  Normal   Created                41m (x3 over 43m)   kubelet, ip-10-0-0-13.us-east-2.compute.internal  Created container
  Normal   Started                41m (x3 over 43m)   kubelet, ip-10-0-0-13.us-east-2.compute.internal  Started container
  Warning  BackOff                23m (x89 over 43m)  kubelet, ip-10-0-0-13.us-east-2.compute.internal  Back-off restarting failed container
  Warning  FailedSync             3m (x180 over 43m)  kubelet, ip-10-0-0-13.us-east-2.compute.internal  Error syncing pod
ubuntu@ip-10-0-0-169:~$ kubectl describe pod  onap-dcae-redis-5 -n onap


  Normal   Scheduled              53m                 default-scheduler                                 Successfully assigned onap-nexus-54ddfc9497-njft6 to ip-10-0-0-80.us-east-2.compute.internal
  Normal   SuccessfulMountVolume  53m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "localtime"
  Normal   SuccessfulMountVolume  52m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "onap-nexus"
  Normal   SuccessfulMountVolume  52m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-bcdgr"
  Normal   Pulling                52m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  pulling image "oomk8s/ubuntu-init:1.0.0"
  Normal   Created                52m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  Created container
  Normal   Pulled                 52m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  Successfully pulled image "oomk8s/ubuntu-init:1.0.0"
  Normal   Started                52m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  Started container
  Normal   Killing                51m                 kubelet, ip-10-0-0-80.us-east-2.compute.internal  Killing container with id docker://nexus:Container failed liveness probe.. Container will be killed and recreated.
  Normal   Pulling                51m (x2 over 52m)   kubelet, ip-10-0-0-80.us-east-2.compute.internal  pulling image "nexus3.onap.org:10001/sonatype/nexus:2.14.8-01"
  Normal   Created                50m (x2 over 52m)   kubelet, ip-10-0-0-80.us-east-2.compute.internal  Created container
  Normal   Pulled                 50m (x2 over 52m)   kubelet, ip-10-0-0-80.us-east-2.compute.internal  Successfully pulled image "nexus3.onap.org:10001/sonatype/nexus:2.14.8-01"
  Normal   Started                50m (x2 over 52m)   kubelet, ip-10-0-0-80.us-east-2.compute.internal  Started container
  Warning  Unhealthy              49m (x4 over 52m)   kubelet, ip-10-0-0-80.us-east-2.compute.internal  Readiness probe failed: Get http://10.42.2.34:8081/nexus/service/local/status: dial tcp 10.42.2.34:8081: getsockopt: connection refused
  Warning  Unhealthy              17m (x31 over 52m)  kubelet, ip-10-0-0-80.us-east-2.compute.internal  Liveness probe failed: dial tcp 10.42.2.34:8081: getsockopt: connection refused
  Warning  FailedSync             7m (x162 over 48m)  kubelet, ip-10-0-0-80.us-east-2.compute.internal  Error syncing pod
  Warning  BackOff                2m (x180 over 48m)  kubelet, ip-10-0-0-80.us-east-2.compute.internal  Back-off restarting failed container
ubuntu@ip-10-0-0-169:~$ kubectl describe pod onap-nexus-54ddfc9497-njft6   -n onap

  Normal   Scheduled              53m                 default-scheduler                                  Successfully assigned onap-brmsgw-5b6848777c-4k42x to ip-10-0-0-111.us-east-2.compute.internal
  Normal   SuccessfulMountVolume  53m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "pe-scripts"
  Normal   SuccessfulMountVolume  53m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "localtime"
  Normal   SuccessfulMountVolume  53m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "pe"
  Normal   SuccessfulMountVolume  53m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "pe-brmsgw"
  Normal   SuccessfulMountVolume  53m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-bcdgr"
  Normal   Pulling                53m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  pulling image "oomk8s/readiness-check:2.0.0"
  Normal   Created                50m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  Created container
  Normal   Pulled                 50m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  Successfully pulled image "oomk8s/readiness-check:2.0.0"
  Normal   Started                50m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  Started container
  Normal   Pulling                49m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  pulling image "nexus3.onap.org:10001/onap/policy-pe:1.2.0"
  Normal   Pulled                 48m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  Successfully pulled image "nexus3.onap.org:10001/onap/policy-pe:1.2.0"
  Normal   Created                48m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  Created container
  Normal   Started                48m                 kubelet, ip-10-0-0-111.us-east-2.compute.internal  Started container
  Warning  Unhealthy              3m (x266 over 47m)  kubelet, ip-10-0-0-111.us-east-2.compute.internal  Readiness probe failed: dial tcp 10.42.164.154:9989: getsockopt: connection refused
ubuntu@ip-10-0-0-169:~$ kubectl describe pod onap-brmsgw-5b6848777c-4k42x  -n onap

  Type    Reason                 Age   From                                              Message
  ----    ------                 ----  ----                                              -------
  Normal  Scheduled              54m   default-scheduler                                 Successfully assigned onap-drools-0 to ip-10-0-0-80.us-east-2.compute.internal
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "localtime"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "policy-data-filebeat"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "policy-logs"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "drools-settingsxml"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "policy-logback"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "drools-config"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "filebeat-conf"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-bcdgr"
  Normal  SuccessfulMountVolume  54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "drools-secret"
  Normal  Pulling                54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  pulling image "oomk8s/readiness-check:2.0.0"
  Normal  Pulled                 54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  Successfully pulled image "oomk8s/readiness-check:2.0.0"
  Normal  Created                54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  Created container
  Normal  Started                54m   kubelet, ip-10-0-0-80.us-east-2.compute.internal  Started container
ubuntu@ip-10-0-0-169:~$ kubectl describe pod onap-drools-0   -n onap

  Type     Reason                 Age                  From                                               Message
  ----     ------                 ----                 ----                                               -------
  Normal   Scheduled              54m                  default-scheduler                                  Successfully assigned onap-policydb-c8f56cd86-g6j7t to ip-10-0-0-227.us-east-2.compute.internal
  Normal   SuccessfulMountVolume  54m                  kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "localtime"
  Normal   SuccessfulMountVolume  54m                  kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "mariadb-conf"
  Normal   SuccessfulMountVolume  54m                  kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "onap-policydb"
  Normal   SuccessfulMountVolume  54m                  kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-bcdgr"
  Warning  Unhealthy              51m                  kubelet, ip-10-0-0-227.us-east-2.compute.internal  Readiness probe failed: dial tcp 10.42.26.2:3306: getsockopt: connection refused
  Normal   Pulling                50m (x3 over 54m)    kubelet, ip-10-0-0-227.us-east-2.compute.internal  pulling image "nexus3.onap.org:10001/mariadb:10.2.14"
  Normal   Created                50m (x3 over 54m)    kubelet, ip-10-0-0-227.us-east-2.compute.internal  Created container
  Normal   Started                50m (x3 over 54m)    kubelet, ip-10-0-0-227.us-east-2.compute.internal  Started container
  Normal   Pulled                 44m (x7 over 54m)    kubelet, ip-10-0-0-227.us-east-2.compute.internal  Successfully pulled image "nexus3.onap.org:10001/mariadb:10.2.14"
  Warning  BackOff                14m (x156 over 51m)  kubelet, ip-10-0-0-227.us-east-2.compute.internal  Back-off restarting failed container
  Warning  FailedSync             4m (x202 over 51m)   kubelet, ip-10-0-0-227.us-east-2.compute.internal  Error syncing pod
ubuntu@ip-10-0-0-169:~$ kubectl describe pod onap-policydb-c8f56cd86-g6j7t   -n onap

  Type     Reason                 Age                 From                                               Message
  ----     ------                 ----                ----                                               -------
  Normal   Scheduled              55m                 default-scheduler                                  Successfully assigned onap-so-db-5665b9bb6d-57hrf to ip-10-0-0-227.us-east-2.compute.internal
  Normal   SuccessfulMountVolume  54m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "onap-so-db"
  Normal   SuccessfulMountVolume  54m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "docker-entrypoint-initdb-d"
  Normal   SuccessfulMountVolume  54m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "localtime"
  Normal   SuccessfulMountVolume  54m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "mariadb-conf"
  Normal   SuccessfulMountVolume  54m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-bcdgr"
  Normal   Pulling                54m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  pulling image "registry.hub.docker.com/oomk8s/ubuntu-init:2.0.0"
  Normal   Started                53m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  Started container
  Normal   Pulled                 53m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  Successfully pulled image "registry.hub.docker.com/oomk8s/ubuntu-init:2.0.0"
  Normal   Created                53m                 kubelet, ip-10-0-0-227.us-east-2.compute.internal  Created container
  Warning  BackOff                51m (x2 over 51m)   kubelet, ip-10-0-0-227.us-east-2.compute.internal  Back-off restarting failed container
  Normal   Pulled                 50m (x3 over 52m)   kubelet, ip-10-0-0-227.us-east-2.compute.internal  Successfully pulled image "nexus3.onap.org:10001/mariadb:10.1.11"
  Normal   Created                50m (x3 over 51m)   kubelet, ip-10-0-0-227.us-east-2.compute.internal  Created container
  Normal   Started                50m (x3 over 51m)   kubelet, ip-10-0-0-227.us-east-2.compute.internal  Started container
  Normal   Pulling                39m (x8 over 53m)   kubelet, ip-10-0-0-227.us-east-2.compute.internal  pulling image "nexus3.onap.org:10001/mariadb:10.1.11"
  Warning  FailedSync             4m (x203 over 51m)  kubelet, ip-10-0-0-227.us-east-2.compute.internal  Error syncing pod
ubuntu@ip-10-0-0-169:~$ kubectl describe pod onap-so-db-5665b9bb6d-57hrf   -n onap

for
00:42:44 9 pending > 0 at the 189th 15 sec interval
00:42:44 
00:43:00 onap          onap-brmsgw-5b6848777c-4k42x                                      0/1       Running            0          52m       10.42.164.154   ip-10-0-0-111.us-east-2.compute.internal
00:43:00 onap          onap-dcae-redis-5                                                 0/1       CrashLoopBackOff   13         43m       10.42.81.13     ip-10-0-0-13.us-east-2.compute.internal
00:43:00 onap          onap-drools-0                                                     0/1       Init:0/1           0          52m       10.42.4.250     ip-10-0-0-80.us-east-2.compute.internal
00:43:00 onap          onap-nexus-54ddfc9497-njft6                                       0/1       CrashLoopBackOff   19         52m       10.42.2.34      ip-10-0-0-80.us-east-2.compute.internal
00:43:00 onap          onap-policydb-c8f56cd86-g6j7t                                     0/1       CrashLoopBackOff   13         52m       10.42.26.2      ip-10-0-0-227.us-east-2.compute.internal
00:43:00 onap          onap-portal-app-558d99bcd4-2878p                                  0/2       Init:0/1           4          52m       10.42.200.76    ip-10-0-0-111.us-east-2.compute.internal
00:43:00 onap          onap-so-5d7f46b4f9-wlzx5                                          0/2       Init:0/1           5          52m       10.42.60.30     ip-10-0-0-13.us-east-2.compute.internal
00:43:00 onap          onap-so-db-5665b9bb6d-57hrf                                       0/1       CrashLoopBackOff   13         52m       10.42.29.48     ip-10-0-0-227.us-east-2.compute.internal
00:43:00 onap          onap-uui-server-78b6bf799b-z4qnx                                  0/1       CrashLoopBackOff   15         52m       10.42.44.215    ip-10-0-0-111.us-east-2.compute.internal
00:43:00 9 pending > 0 at the 190th 15 sec interval


http://jenkins.onap.info/job/oom-cd-master/2907/console