Uploaded image for project: 'Network Controller'
  1. Network Controller
  2. SDNC-649

SDNC pod fails readiness probe

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Highest Highest
    • Dublin Release
    • Dublin Release
    • None
    • Integration-OOM-Staging-Daily tenant of Windriver lab

    • SDNC Dublin Spr 2 2/18-3/8

      After ONAP master branch is deployed, SDNC pods sdnc-sdnc-0/1/2 fails readiness probe

       

      root@staging-rancher:~# kubectl -n onap get pod |grep sdnc-sdnc
      dev-sdnc-sdnc-0 1/2 Running 0 27m
      dev-sdnc-sdnc-1 1/2 Running 0 27m
      dev-sdnc-sdnc-2 1/2 Running 0 26m
      dev-sdnc-sdnc-ansible-server-7c585b9d99-rq9vh 0/1 Init:0/1 43 7h
      dev-sdnc-sdnc-db-0 2/2 Running 0 7h
      dev-sdnc-sdnc-dgbuilder-57c447cd6b-xdkhb 1/1 Running 0 7h
      dev-sdnc-sdnc-dmaap-listener-6c58f4bc5f-7wq8j 0/1 Init:0/1 0 7h
      dev-sdnc-sdnc-portal-7c6dddcf57-54rr6 0/1 Init:0/1 0 7h
      dev-sdnc-sdnc-ueb-listener-7767955d5f-nrrsp 0/1 Init:0/1 0 7h
      

       

      Karaf log is attached

       

      Pod describe shows:

       

      root@staging-rancher:~# kubectl -n onap describe pod dev-sdnc-sdnc-0
      Name: dev-sdnc-sdnc-0
      Namespace: onap
      Node: staging-k8s-10/10.0.0.29
      Start Time: Mon, 18 Feb 2019 13:54:27 +0000
      Labels: app=sdnc
       controller-revision-hash=dev-sdnc-sdnc-5f5fd9649b
       release=dev-sdnc
       statefulset.kubernetes.io/pod-name=dev-sdnc-sdnc-0
      Annotations: <none>
      Status: Running
      IP: 10.42.41.119
      Controlled By: StatefulSet/dev-sdnc-sdnc
      Init Containers:
       sdnc-readiness:
       Container ID: docker://40010e885b6f321295be078cb416a04c7058c6772051474d1d8b0a6ed207ff47
       Image: oomk8s/readiness-check:2.0.0
       Image ID: docker-pullable://oomk8s/readiness-check@sha256:7daa08b81954360a1111d03364febcb3dcfeb723bcc12ce3eb3ed3e53f2323ed
       Port: <none>
       Host Port: <none>
       Command:
       /root/ready.py
       Args:
       --container-name
       sdnc-db
       State: Terminated
       Reason: Completed
       Exit Code: 0
       Started: Mon, 18 Feb 2019 13:54:43 +0000
       Finished: Mon, 18 Feb 2019 13:54:46 +0000
       Ready: True
       Restart Count: 0
       Environment:
       NAMESPACE: onap (v1:metadata.namespace)
       Mounts:
       /var/run/secrets/kubernetes.io/serviceaccount from default-token-vhmdw (ro)
      Containers:
       sdnc:
       Container ID: docker://b9cf7486bdf4fc6f585c9c9c71178949de5f504436bba9411c3d0b6e405e4459
       Image: 10.12.5.2:5000/onap/sdnc-image:1.5-STAGING-latest
       Image ID: docker-pullable://10.12.5.2:5000/onap/sdnc-image@sha256:041f5a806e1b0c9f189ef62f1d956f49cb3b70b3f31769b2b5e174f11f1e73a9
       Ports: 8181/TCP, 8101/TCP, 8080/TCP, 2550/TCP
       Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
       Command:
       /bin/bash
       Args:
       -c
       /opt/sdnc/bin/startODL.sh
       State: Running
       Started: Mon, 18 Feb 2019 13:56:42 +0000
       Ready: False
       Restart Count: 0
       Readiness: tcp-socket :8181 delay=60s timeout=1s period=10s #success=1 #failure=3
       Environment:
       MYSQL_ROOT_PASSWORD: <set to the key 'db-root-password' in secret 'dev-sdnc-sdnc'> Optional: false
       ODL_ADMIN_PASSWORD: <set to the key 'odl-password' in secret 'dev-sdnc-sdnc-odl'> Optional: false
       SDNC_DB_PASSWORD: <set to the key 'db-sdnctl-password' in secret 'dev-sdnc-sdnc-sdnctl'> Optional: false
       SDNC_CONFIG_DIR: /opt/onap/sdnc/data/properties
       ENABLE_ODL_CLUSTER: true
       MY_ODL_CLUSTER: 127.0.0.1
       PEER_ODL_CLUSTER: 127.0.0.1
       IS_PRIMARY_CLUSTER: true
       GEO_ENABLED: false
       SDNC_REPLICAS: 3
       MYSQL_HOST: dev-sdnc-sdnc-db-0.sdnc-dbhost.onap
       Mounts:
       /etc/localtime from localtime (ro)
       /opt/onap/sdnc/bin/installSdncDb.sh from bin (rw)
       /opt/onap/sdnc/bin/startODL.sh from bin (rw)
       /opt/onap/sdnc/data/properties/aaiclient.properties from properties (rw)
       /opt/onap/sdnc/data/properties/blueprints-processor-adaptor.properties from properties (rw)
       /opt/onap/sdnc/data/properties/dblib.properties from properties (rw)
       /opt/onap/sdnc/data/properties/lcm-dg.properties from properties (rw)
       /opt/onap/sdnc/data/properties/netbox.properties from properties (rw)
       /opt/onap/sdnc/data/properties/svclogic.properties from properties (rw)
       /opt/onap/sdnc/svclogic/config/svclogic.properties from properties (rw)
       /opt/opendaylight/current/daexim from dev-sdnc-sdnc-mdsal (rw)
       /var/log/onap from logs (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from default-token-vhmdw (ro)
       filebeat-onap:
       Container ID: docker://7dff20ce870e930c79b09868327f33620a273a72e7edb764495f899396062f5e
       Image: docker.elastic.co/beats/filebeat:5.5.0
       Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942
       Port: <none>
       Host Port: <none>
       State: Running
       Started: Mon, 18 Feb 2019 13:56:44 +0000
       Ready: True
       Restart Count: 0
       Environment: <none>
       Mounts:
       /usr/share/filebeat/data from data-filebeat (rw)
       /usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
       /var/log/onap from logs (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from default-token-vhmdw (ro)
      Conditions:
       Type Status
       Initialized True 
       Ready False 
       ContainersReady False 
       PodScheduled True 
      Volumes:
       dev-sdnc-sdnc-mdsal:
       Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
       ClaimName: dev-sdnc-sdnc-mdsal-dev-sdnc-sdnc-0
       ReadOnly: false
       localtime:
       Type: HostPath (bare host directory volume)
       Path: /etc/localtime
       HostPathType: 
       logs:
       Type: EmptyDir (a temporary directory that shares a pod's lifetime)
       Medium: 
       data-filebeat:
       Type: EmptyDir (a temporary directory that shares a pod's lifetime)
       Medium: 
       filebeat-conf:
       Type: ConfigMap (a volume populated by a ConfigMap)
       Name: dev-sdnc-sdnc-filebeat-configmap
       Optional: false
       sdnc-logging-cfg-config:
       Type: ConfigMap (a volume populated by a ConfigMap)
       Name: dev-sdnc-sdnc-log-configmap
       Optional: false
       bin:
       Type: ConfigMap (a volume populated by a ConfigMap)
       Name: dev-sdnc-sdnc-bin
       Optional: false
       properties:
       Type: ConfigMap (a volume populated by a ConfigMap)
       Name: dev-sdnc-sdnc-properties
       Optional: false
       default-token-vhmdw:
       Type: Secret (a volume populated by a Secret)
       SecretName: default-token-vhmdw
       Optional: false
      QoS Class: BestEffort
      Node-Selectors: <none>
      Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
       node.kubernetes.io/unreachable:NoExecute for 300s
      Events:
       Type Reason Age From Message
       ---- ------ ---- ---- -------
       Normal Scheduled 7m default-scheduler Successfully assigned onap/dev-sdnc-sdnc-0 to staging-k8s-10
       Normal Pulled 7m kubelet, staging-k8s-10 Container image "oomk8s/readiness-check:2.0.0" already present on machine
       Normal Created 7m kubelet, staging-k8s-10 Created container
       Normal Started 7m kubelet, staging-k8s-10 Started container
       Normal Pulling 7m kubelet, staging-k8s-10 pulling image "10.12.5.2:5000/onap/sdnc-image:1.5-STAGING-latest"
       Normal Pulled 5m kubelet, staging-k8s-10 Successfully pulled image "10.12.5.2:5000/onap/sdnc-image:1.5-STAGING-latest"
       Normal Created 5m kubelet, staging-k8s-10 Created container
       Normal Started 5m kubelet, staging-k8s-10 Started container
       Normal Pulled 5m kubelet, staging-k8s-10 Container image "docker.elastic.co/beats/filebeat:5.5.0" already present on machine
       Normal Created 5m kubelet, staging-k8s-10 Created container
       Normal Started 5m kubelet, staging-k8s-10 Started container
       Warning Unhealthy 1m (x16 over 4m) kubelet, staging-k8s-10 Readiness probe failed: dial tcp 10.42.41.119:8181: connect: connection refused
      

       

            xuyang11 xuyang11
            xuyang11 xuyang11
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: