Details
-
Bug
-
Status: Closed
-
Medium
-
Resolution: Done
-
Dublin Release
-
Windriver lab Integration-OOM-Daily tenant
Description
After ONAP is deployed, POMBA Network-discovery-context-builder healthcheck times out, and the pod is in CrashLoopBackOff.
------------------------------------------------------------------------------ Basic Pomba Network-discovery-context-builder Health Check | FAIL | Test timeout 10 seconds exceeded. ------------------------------------------------------------------------------
root@release-rancher:~# kubectl -n onap get pod |grep pomba dev-pomba-pomba-aaictxbuilder-656d96bb6b-lmxqm 2/2 Running 6 13h dev-pomba-pomba-contextaggregator-57f47c8564-qtx6x 1/1 Running 0 13h dev-pomba-pomba-data-router-769fcc666-x5mfx 1/1 Running 0 13h dev-pomba-pomba-elasticsearch-d6589d578-4qqhw 1/1 Running 0 13h dev-pomba-pomba-kibana-5d7f684f5b-zxjqv 1/1 Running 0 13h dev-pomba-pomba-networkdiscovery-6f4d44fc94-844n5 2/2 Running 0 13h dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc-lwfzb 1/2 CrashLoopBackOff 225 13h dev-pomba-pomba-sdcctxbuilder-7889df5f7-wnglf 1/1 Running 0 13h dev-pomba-pomba-search-data-dc47bbcb8-6fxcb 2/2 Running 0 13h dev-pomba-pomba-servicedecomposition-567c57859-rk5sl 2/2 Running 5 13h dev-pomba-pomba-validation-service-785fdd58dc-6n6tb 1/1 Running 0 13h
The output of describe the pod:
root@release-rancher:~# kubectl -n onap describe pod dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc-lwfzb Name: dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc-lwfzb Namespace: onap Node: release-k8s-08/10.0.0.10 Start Time: Mon, 28 Jan 2019 12:35:03 +0000 Labels: app=pomba-networkdiscoveryctxbuilder pod-template-hash=2114668097 release=dev-pomba Annotations: <none> Status: Running IP: 10.42.229.209 Controlled By: ReplicaSet/dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc Containers: pomba-networkdiscoveryctxbuilder: Container ID: docker://c5f3f6aac76c275136217dd2160057157fce180bee76c73e643b8410304aa243 Image: 10.12.5.2:5000/onap/pomba-network-discovery-context-builder:1.3.1 Image ID: docker-pullable://10.12.5.2:5000/onap/pomba-network-discovery-context-builder@sha256:3488e02cc06441616b971f9ade079aad49c874828b97ce106ad325053b7da955 Port: 8080/TCP Host Port: 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 137 Started: Tue, 29 Jan 2019 02:31:42 +0000 Finished: Tue, 29 Jan 2019 02:32:54 +0000 Ready: False Restart Count: 225 Liveness: tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /etc/localtime from localtime (ro) /opt/app/config/application.properties from dev-pomba-pomba-networkdiscoveryctxbuilder-config (ro) /opt/app/config/logback.xml from dev-pomba-pomba-networkdiscoveryctxbuilder-log-conf (rw) /var/log/onap from dev-pomba-pomba-networkdiscoveryctxbuilder-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-pf8lf (ro) pomba-networkdiscoveryctxbuilder-filebeat-onap: Container ID: docker://5b9de52de8f0d0190fd2bd31953a114a9432032f6e87941828f5e580dd7b8d12 Image: docker.elastic.co/beats/filebeat:5.5.0 Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 Port: <none> Host Port: <none> State: Running Started: Mon, 28 Jan 2019 12:49:50 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/filebeat/data from dev-pomba-pomba-networkdiscoveryctxbuilder-data-filebeat (rw) /usr/share/filebeat/filebeat.yml from dev-pomba-pomba-networkdiscoveryctxbuilder-filebeat-conf (rw) /var/log/onap from dev-pomba-pomba-networkdiscoveryctxbuilder-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-pf8lf (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: dev-pomba-pomba-networkdiscoveryctxbuilder-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-pomba-pomba-networkdiscoveryctxbuilder-configmap Optional: false dev-pomba-pomba-networkdiscoveryctxbuilder-log-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-pomba-pomba-networkdiscoveryctxbuilder-log Optional: false dev-pomba-pomba-networkdiscoveryctxbuilder-filebeat-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-pomba-pomba-filebeat-configmap Optional: false dev-pomba-pomba-networkdiscoveryctxbuilder-data-filebeat: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: dev-pomba-pomba-networkdiscoveryctxbuilder-logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-pf8lf: Type: Secret (a volume populated by a Secret) SecretName: default-token-pf8lf Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 9m (x1271 over 13h) kubelet, release-k8s-08 Readiness probe failed: dial tcp 10.42.229.209:8080: connect: connection refused Warning BackOff 4m (x2559 over 13h) kubelet, release-k8s-08 Back-off restarting failed container