-
Bug
-
Resolution: Not a Bug
-
High
-
Dublin Release
-
Integration-OOM-Staging-Daily
After master branch ONAP is deployed, one of DMAAP dr-db pods enters into CrashLoopBackOff
root@staging-rancher:~# kubectl -n onap get pod |grep dmaap-dr dev-dmaap-dmaap-dr-db-0 1/1 Running 0 9h dev-dmaap-dmaap-dr-db-1 0/1 CrashLoopBackOff 141 9h dev-dmaap-dmaap-dr-node-7c4dcbf477-2z6nw 0/1 Init:0/1 58 9h dev-dmaap-dmaap-dr-prov-b75f4b9b5-x82v9 0/1 Init:0/1 58 9h
Pod describe shows:
root@staging-rancher:~# kubectl -n onap describe pod dev-dmaap-dmaap-dr-db-1 Name: dev-dmaap-dmaap-dr-db-1 Namespace: onap Node: staging-k8s-08/10.0.0.20 Start Time: Wed, 27 Feb 2019 16:57:48 +0000 Labels: app=dev-dmaap-dmaap-dr-db chart=mariadb-4.0.0 controller-revision-hash=dev-dmaap-dmaap-dr-db-7b76bf85c heritage=Tiller release=dev-dmaap statefulset.kubernetes.io/pod-name=dev-dmaap-dmaap-dr-db-1 Annotations: pod.alpha.kubernetes.io/initialized=true Status: Running IP: 10.42.247.97 Controlled By: StatefulSet/dev-dmaap-dmaap-dr-db Init Containers: mariadb-galera-prepare: Container ID: docker://3cc447535e18b0291a0648e292de660d47f26427f1981a404f53e0d1afc1b23b Image: 10.12.5.2:5000/busybox Image ID: docker-pullable://10.12.5.2:5000/busybox@sha256:4415a904b1aca178c2450fd54928ab362825e863c0ad5452fd020e92f7a6a47e Port: <none> Host Port: <none> Command: sh -c chown -R 27:27 /var/lib/mysql State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 27 Feb 2019 16:58:21 +0000 Finished: Wed, 27 Feb 2019 16:58:22 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/lib/mysql from dev-dmaap-dmaap-dr-db-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-zs2cl (ro) Containers: dmaap-dr-db: Container ID: docker://ff1b256f9d918a2e48c3642ef8115f2777c668bdb69a2d51cb5bd5bfa3314789 Image: 10.12.5.2:5000/adfinissygroup/k8s-mariadb-galera-centos:v002 Image ID: docker-pullable://10.12.5.2:5000/adfinissygroup/k8s-mariadb-galera-centos@sha256:fbcb842f30065ae94532cb1af9bb03cc6e2acaaf896d87d0ec38da7dd09a3dde Ports: 3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP State: Running Started: Wed, 27 Feb 2019 16:59:48 +0000 Last State: Terminated Reason: Error Exit Code: 137 Started: Wed, 27 Feb 2019 16:58:23 +0000 Finished: Wed, 27 Feb 2019 16:59:47 +0000 Ready: False Restart Count: 1 Liveness: exec [mysqladmin ping] delay=30s timeout=5s period=10s #success=1 #failure=3 Readiness: exec [/usr/share/container-scripts/mysql/readiness-probe.sh] delay=15s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAMESPACE: onap (v1:metadata.namespace) MYSQL_USER: datarouter MYSQL_PASSWORD: <set to the key 'user-password' in secret 'dev-dmaap-dmaap-dr-db'> Optional: false MYSQL_DATABASE: datarouter MYSQL_ROOT_PASSWORD: <set to the key 'db-root-password' in secret 'dev-dmaap-dmaap-dr-db'> Optional: false Mounts: /etc/localtime from localtime (ro) /var/lib/mysql from dev-dmaap-dmaap-dr-db-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-zs2cl (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: dev-dmaap-dmaap-dr-db-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: dev-dmaap-dmaap-dr-db-data-dev-dmaap-dmaap-dr-db-1 ReadOnly: false localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: default-token-zs2cl: Type: Secret (a volume populated by a Secret) SecretName: default-token-zs2cl Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m default-scheduler Successfully assigned onap/dev-dmaap-dmaap-dr-db-1 to staging-k8s-08 Normal Pulling 2m kubelet, staging-k8s-08 pulling image "10.12.5.2:5000/busybox" Normal Pulled 2m kubelet, staging-k8s-08 Successfully pulled image "10.12.5.2:5000/busybox" Normal Created 2m kubelet, staging-k8s-08 Created container Normal Started 2m kubelet, staging-k8s-08 Started container Normal Pulled 55s (x2 over 2m) kubelet, staging-k8s-08 Container image "10.12.5.2:5000/adfinissygroup/k8s-mariadb-galera-centos:v002" already present on machine Normal Created 55s (x2 over 2m) kubelet, staging-k8s-08 Created container Normal Killing 55s kubelet, staging-k8s-08 Killing container with id docker://dmaap-dr-db:Container failed liveness probe.. Container will be killed and recreated. Normal Started 54s (x2 over 2m) kubelet, staging-k8s-08 Started container Warning Unhealthy 16s (x4 over 1m) kubelet, staging-k8s-08 Liveness probe failed: mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")' Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists! Warning Unhealthy 13s (x10 over 2m) kubelet, staging-k8s-08 Readiness probe failed: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Pod console log file is attached