Uploaded image for project: 'Data Movement as a Platform'
  1. Data Movement as a Platform
  2. DMAAP-1066

[DR] One of DMAAP dr-db pods in CrashLoopBackOff

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: High High
    • Frankfurt Release
    • Dublin Release
    • Integration-OOM-Staging-Daily

      After master branch ONAP is deployed, one of DMAAP dr-db pods enters into CrashLoopBackOff

       

      root@staging-rancher:~# kubectl -n onap get pod |grep dmaap-dr
      dev-dmaap-dmaap-dr-db-0 1/1 Running 0 9h
      dev-dmaap-dmaap-dr-db-1 0/1 CrashLoopBackOff 141 9h
      dev-dmaap-dmaap-dr-node-7c4dcbf477-2z6nw 0/1 Init:0/1 58 9h
      dev-dmaap-dmaap-dr-prov-b75f4b9b5-x82v9 0/1 Init:0/1 58 9h
      

       

      Pod describe shows:

      root@staging-rancher:~# kubectl -n onap describe pod dev-dmaap-dmaap-dr-db-1
      Name:           dev-dmaap-dmaap-dr-db-1
      Namespace:      onap
      Node:           staging-k8s-08/10.0.0.20
      Start Time:     Wed, 27 Feb 2019 16:57:48 +0000
      Labels:         app=dev-dmaap-dmaap-dr-db
                      chart=mariadb-4.0.0
                      controller-revision-hash=dev-dmaap-dmaap-dr-db-7b76bf85c
                      heritage=Tiller
                      release=dev-dmaap
                      statefulset.kubernetes.io/pod-name=dev-dmaap-dmaap-dr-db-1
      Annotations:    pod.alpha.kubernetes.io/initialized=true
      Status:         Running
      IP:             10.42.247.97
      Controlled By:  StatefulSet/dev-dmaap-dmaap-dr-db
      Init Containers:
        mariadb-galera-prepare:
          Container ID:  docker://3cc447535e18b0291a0648e292de660d47f26427f1981a404f53e0d1afc1b23b
          Image:         10.12.5.2:5000/busybox
          Image ID:      docker-pullable://10.12.5.2:5000/busybox@sha256:4415a904b1aca178c2450fd54928ab362825e863c0ad5452fd020e92f7a6a47e
          Port:          <none>
          Host Port:     <none>
          Command:
            sh
            -c
            chown -R 27:27 /var/lib/mysql
          State:          Terminated
            Reason:       Completed
            Exit Code:    0
            Started:      Wed, 27 Feb 2019 16:58:21 +0000
            Finished:     Wed, 27 Feb 2019 16:58:22 +0000
          Ready:          True
          Restart Count:  0
          Environment:    <none>
          Mounts:
            /var/lib/mysql from dev-dmaap-dmaap-dr-db-data (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from default-token-zs2cl (ro)
      Containers:
        dmaap-dr-db:
          Container ID:   docker://ff1b256f9d918a2e48c3642ef8115f2777c668bdb69a2d51cb5bd5bfa3314789
          Image:          10.12.5.2:5000/adfinissygroup/k8s-mariadb-galera-centos:v002
          Image ID:       docker-pullable://10.12.5.2:5000/adfinissygroup/k8s-mariadb-galera-centos@sha256:fbcb842f30065ae94532cb1af9bb03cc6e2acaaf896d87d0ec38da7dd09a3dde
          Ports:          3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP
          Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
          State:          Running
            Started:      Wed, 27 Feb 2019 16:59:48 +0000
          Last State:     Terminated
            Reason:       Error
            Exit Code:    137
            Started:      Wed, 27 Feb 2019 16:58:23 +0000
            Finished:     Wed, 27 Feb 2019 16:59:47 +0000
          Ready:          False
          Restart Count:  1
          Liveness:       exec [mysqladmin ping] delay=30s timeout=5s period=10s #success=1 #failure=3
          Readiness:      exec [/usr/share/container-scripts/mysql/readiness-probe.sh] delay=15s timeout=1s period=10s #success=1 #failure=3
          Environment:
            POD_NAMESPACE:        onap (v1:metadata.namespace)
            MYSQL_USER:           datarouter
            MYSQL_PASSWORD:       <set to the key 'user-password' in secret 'dev-dmaap-dmaap-dr-db'>  Optional: false
            MYSQL_DATABASE:       datarouter
            MYSQL_ROOT_PASSWORD:  <set to the key 'db-root-password' in secret 'dev-dmaap-dmaap-dr-db'>  Optional: false
          Mounts:
            /etc/localtime from localtime (ro)
            /var/lib/mysql from dev-dmaap-dmaap-dr-db-data (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from default-token-zs2cl (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        dev-dmaap-dmaap-dr-db-data:
          Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
          ClaimName:  dev-dmaap-dmaap-dr-db-data-dev-dmaap-dmaap-dr-db-1
          ReadOnly:   false
        localtime:
          Type:          HostPath (bare host directory volume)
          Path:          /etc/localtime
          HostPathType:  
        default-token-zs2cl:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  default-token-zs2cl
          Optional:    false
      QoS Class:       BestEffort
      Node-Selectors:  <none>
      Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                       node.kubernetes.io/unreachable:NoExecute for 300s
      Events:
        Type     Reason     Age               From                     Message
        ----     ------     ----              ----                     -------
        Normal   Scheduled  2m                default-scheduler        Successfully assigned onap/dev-dmaap-dmaap-dr-db-1 to staging-k8s-08
        Normal   Pulling    2m                kubelet, staging-k8s-08  pulling image "10.12.5.2:5000/busybox"
        Normal   Pulled     2m                kubelet, staging-k8s-08  Successfully pulled image "10.12.5.2:5000/busybox"
        Normal   Created    2m                kubelet, staging-k8s-08  Created container
        Normal   Started    2m                kubelet, staging-k8s-08  Started container
        Normal   Pulled     55s (x2 over 2m)  kubelet, staging-k8s-08  Container image "10.12.5.2:5000/adfinissygroup/k8s-mariadb-galera-centos:v002" already present on machine
        Normal   Created    55s (x2 over 2m)  kubelet, staging-k8s-08  Created container
        Normal   Killing    55s               kubelet, staging-k8s-08  Killing container with id docker://dmaap-dr-db:Container failed liveness probe.. Container will be killed and recreated.
        Normal   Started    54s (x2 over 2m)  kubelet, staging-k8s-08  Started container
        Warning  Unhealthy  16s (x4 over 1m)  kubelet, staging-k8s-08  Liveness probe failed: mysqladmin: connect to server at 'localhost' failed
      error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")'
      Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
        Warning  Unhealthy  13s (x10 over 2m)  kubelet, staging-k8s-08  Readiness probe failed: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory"

       

      Pod console log file is attached

       

       

       

        1. dr-db-pod.log
          16 kB
        2. image.png
          image.png
          114 kB
        3. image(1).png
          image(1).png
          103 kB
        4. image(2).png
          image(2).png
          210 kB

            Unassigned Unassigned
            xuyang11 xuyang11
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: