-
Bug
-
Resolution: Done
-
Highest
-
Dublin Release
-
None
-
Integration-OOM-Staging-Daily tenant of Windriver lab
-
SDNC Dublin Spr 2 2/18-3/8
After ONAP master branch is deployed, SDNC pods sdnc-sdnc-0/1/2 fails readiness probe
root@staging-rancher:~# kubectl -n onap get pod |grep sdnc-sdnc dev-sdnc-sdnc-0 1/2 Running 0 27m dev-sdnc-sdnc-1 1/2 Running 0 27m dev-sdnc-sdnc-2 1/2 Running 0 26m dev-sdnc-sdnc-ansible-server-7c585b9d99-rq9vh 0/1 Init:0/1 43 7h dev-sdnc-sdnc-db-0 2/2 Running 0 7h dev-sdnc-sdnc-dgbuilder-57c447cd6b-xdkhb 1/1 Running 0 7h dev-sdnc-sdnc-dmaap-listener-6c58f4bc5f-7wq8j 0/1 Init:0/1 0 7h dev-sdnc-sdnc-portal-7c6dddcf57-54rr6 0/1 Init:0/1 0 7h dev-sdnc-sdnc-ueb-listener-7767955d5f-nrrsp 0/1 Init:0/1 0 7h
Karaf log is attached
Pod describe shows:
root@staging-rancher:~# kubectl -n onap describe pod dev-sdnc-sdnc-0 Name: dev-sdnc-sdnc-0 Namespace: onap Node: staging-k8s-10/10.0.0.29 Start Time: Mon, 18 Feb 2019 13:54:27 +0000 Labels: app=sdnc controller-revision-hash=dev-sdnc-sdnc-5f5fd9649b release=dev-sdnc statefulset.kubernetes.io/pod-name=dev-sdnc-sdnc-0 Annotations: <none> Status: Running IP: 10.42.41.119 Controlled By: StatefulSet/dev-sdnc-sdnc Init Containers: sdnc-readiness: Container ID: docker://40010e885b6f321295be078cb416a04c7058c6772051474d1d8b0a6ed207ff47 Image: oomk8s/readiness-check:2.0.0 Image ID: docker-pullable://oomk8s/readiness-check@sha256:7daa08b81954360a1111d03364febcb3dcfeb723bcc12ce3eb3ed3e53f2323ed Port: <none> Host Port: <none> Command: /root/ready.py Args: --container-name sdnc-db State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 18 Feb 2019 13:54:43 +0000 Finished: Mon, 18 Feb 2019 13:54:46 +0000 Ready: True Restart Count: 0 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-vhmdw (ro) Containers: sdnc: Container ID: docker://b9cf7486bdf4fc6f585c9c9c71178949de5f504436bba9411c3d0b6e405e4459 Image: 10.12.5.2:5000/onap/sdnc-image:1.5-STAGING-latest Image ID: docker-pullable://10.12.5.2:5000/onap/sdnc-image@sha256:041f5a806e1b0c9f189ef62f1d956f49cb3b70b3f31769b2b5e174f11f1e73a9 Ports: 8181/TCP, 8101/TCP, 8080/TCP, 2550/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: /bin/bash Args: -c /opt/sdnc/bin/startODL.sh State: Running Started: Mon, 18 Feb 2019 13:56:42 +0000 Ready: False Restart Count: 0 Readiness: tcp-socket :8181 delay=60s timeout=1s period=10s #success=1 #failure=3 Environment: MYSQL_ROOT_PASSWORD: <set to the key 'db-root-password' in secret 'dev-sdnc-sdnc'> Optional: false ODL_ADMIN_PASSWORD: <set to the key 'odl-password' in secret 'dev-sdnc-sdnc-odl'> Optional: false SDNC_DB_PASSWORD: <set to the key 'db-sdnctl-password' in secret 'dev-sdnc-sdnc-sdnctl'> Optional: false SDNC_CONFIG_DIR: /opt/onap/sdnc/data/properties ENABLE_ODL_CLUSTER: true MY_ODL_CLUSTER: 127.0.0.1 PEER_ODL_CLUSTER: 127.0.0.1 IS_PRIMARY_CLUSTER: true GEO_ENABLED: false SDNC_REPLICAS: 3 MYSQL_HOST: dev-sdnc-sdnc-db-0.sdnc-dbhost.onap Mounts: /etc/localtime from localtime (ro) /opt/onap/sdnc/bin/installSdncDb.sh from bin (rw) /opt/onap/sdnc/bin/startODL.sh from bin (rw) /opt/onap/sdnc/data/properties/aaiclient.properties from properties (rw) /opt/onap/sdnc/data/properties/blueprints-processor-adaptor.properties from properties (rw) /opt/onap/sdnc/data/properties/dblib.properties from properties (rw) /opt/onap/sdnc/data/properties/lcm-dg.properties from properties (rw) /opt/onap/sdnc/data/properties/netbox.properties from properties (rw) /opt/onap/sdnc/data/properties/svclogic.properties from properties (rw) /opt/onap/sdnc/svclogic/config/svclogic.properties from properties (rw) /opt/opendaylight/current/daexim from dev-sdnc-sdnc-mdsal (rw) /var/log/onap from logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-vhmdw (ro) filebeat-onap: Container ID: docker://7dff20ce870e930c79b09868327f33620a273a72e7edb764495f899396062f5e Image: docker.elastic.co/beats/filebeat:5.5.0 Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 Port: <none> Host Port: <none> State: Running Started: Mon, 18 Feb 2019 13:56:44 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/filebeat/data from data-filebeat (rw) /usr/share/filebeat/filebeat.yml from filebeat-conf (rw) /var/log/onap from logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-vhmdw (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: dev-sdnc-sdnc-mdsal: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: dev-sdnc-sdnc-mdsal-dev-sdnc-sdnc-0 ReadOnly: false localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: data-filebeat: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: filebeat-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-sdnc-sdnc-filebeat-configmap Optional: false sdnc-logging-cfg-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-sdnc-sdnc-log-configmap Optional: false bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-sdnc-sdnc-bin Optional: false properties: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-sdnc-sdnc-properties Optional: false default-token-vhmdw: Type: Secret (a volume populated by a Secret) SecretName: default-token-vhmdw Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 7m default-scheduler Successfully assigned onap/dev-sdnc-sdnc-0 to staging-k8s-10 Normal Pulled 7m kubelet, staging-k8s-10 Container image "oomk8s/readiness-check:2.0.0" already present on machine Normal Created 7m kubelet, staging-k8s-10 Created container Normal Started 7m kubelet, staging-k8s-10 Started container Normal Pulling 7m kubelet, staging-k8s-10 pulling image "10.12.5.2:5000/onap/sdnc-image:1.5-STAGING-latest" Normal Pulled 5m kubelet, staging-k8s-10 Successfully pulled image "10.12.5.2:5000/onap/sdnc-image:1.5-STAGING-latest" Normal Created 5m kubelet, staging-k8s-10 Created container Normal Started 5m kubelet, staging-k8s-10 Started container Normal Pulled 5m kubelet, staging-k8s-10 Container image "docker.elastic.co/beats/filebeat:5.5.0" already present on machine Normal Created 5m kubelet, staging-k8s-10 Created container Normal Started 5m kubelet, staging-k8s-10 Started container Warning Unhealthy 1m (x16 over 4m) kubelet, staging-k8s-10 Readiness probe failed: dial tcp 10.42.41.119:8181: connect: connection refused