-
Bug
-
Resolution: Done
-
High
-
Dublin Release
-
OOM-Staging-Daily
-
A&AI Sprint 23
After ONAP is deployed, aai-champ pod is unhealthy
root@staging-rancher:~# kubectl -n onap get pod |grep aai-champ dev-aai-aai-champ-7b4c574896-x6fjp 1/2 Running 0 10h root@staging-rancher:~# kubectl -n onap describe pod dev-aai-aai-champ-7b4c574896-x6fjp Name: dev-aai-aai-champ-7b4c574896-x6fjp Namespace: onap Priority: 0 PriorityClassName: <none> Node: staging-k8s-08/10.0.0.17 Start Time: Tue, 23 Apr 2019 06:13:21 +0000 Labels: app=aai-champ pod-template-hash=7b4c574896 release=dev-aai Annotations: cni.projectcalico.org/podIP: 10.42.9.5/32 Status: Running IP: 10.42.9.5 Controlled By: ReplicaSet/dev-aai-aai-champ-7b4c574896 Init Containers: aai-champ-readiness: Container ID: docker://05461d7b6b1883545c77f609ae2c256a80b453d68ca7669172f2f092b621be2e Image: oomk8s/readiness-check:2.0.2 Image ID: docker-pullable://oomk8s/readiness-check@sha256:875814cc853d7d9b3227165740584c347884bc8c0d831b4148cb5f7952e4cbff Port: <none> Host Port: <none> Command: /root/ready.py Args: --container-name cassandra State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 23 Apr 2019 06:34:59 +0000 Finished: Tue, 23 Apr 2019 06:43:16 +0000 Ready: True Restart Count: 2 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-dxj4z (ro) Containers: aai-champ: Container ID: docker://b22d7f21c60ea3a9f3d70e3555e9dbedb9766b28181c505f6071ec5f5bf69296 Image: 10.12.5.2:5000/onap/champ:1.4-STAGING-latest Image ID: docker-pullable://10.12.5.2:5000/onap/champ@sha256:d3842f6dab1f7c04c0f746a947adcbdf88364636fdfe68b076acc6761139b276 Port: 9522/TCP Host Port: 0/TCP State: Running Started: Tue, 23 Apr 2019 06:44:04 +0000 Ready: False Restart Count: 0 Readiness: tcp-socket :9522 delay=120s timeout=1s period=10s #success=1 #failure=3 Environment: CONFIG_HOME: /opt/app/champ-service/appconfig GRAPHIMPL: janus-deps KEY_STORE_PASSWORD: <set to the key 'KEY_STORE_PASSWORD' in secret 'dev-aai-aai-champ-pass'> Optional: false KEY_MANAGER_PASSWORD: <set to the key 'KEY_MANAGER_PASSWORD' in secret 'dev-aai-aai-champ-pass'> Optional: false SERVICE_BEANS: /opt/app/champ-service/dynamic/conf Mounts: /etc/localtime from localtime (ro) /opt/app/champ-service/appconfig/auth from dev-aai-aai-champ-secrets (rw) /opt/app/champ-service/appconfig/champ-api.properties from dev-aai-aai-champ-config (rw) /opt/app/champ-service/bundleconfig/etc/logback.xml from dev-aai-aai-champ-logback-config (rw) /opt/app/champ-service/dynamic/conf/champ-beans.xml from dev-aai-aai-champ-dynamic-config (rw) /var/log/onap from dev-aai-aai-champ-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-dxj4z (ro) filebeat-onap: Container ID: docker://f01783081910cd0883291b71b5183f1b2e8ec6060deaee124ef865cf60e54642 Image: docker.elastic.co/beats/filebeat:5.5.0 Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 Port: <none> Host Port: <none> State: Running Started: Tue, 23 Apr 2019 06:44:45 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/filebeat/data from aai-filebeat (rw) /usr/share/filebeat/filebeat.yml from filebeat-conf (rw) /var/log/onap from dev-aai-aai-champ-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-dxj4z (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: dev-aai-aai-champ-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-aai-champ Optional: false dev-aai-aai-champ-secrets: Type: Secret (a volume populated by a Secret) SecretName: dev-aai-aai-champ-champ Optional: false dev-aai-aai-champ-dynamic-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-aai-champ-dynamic Optional: false dev-aai-aai-champ-logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: dev-aai-aai-champ-logback-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-aai-champ-log-configmap Optional: false filebeat-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: aai-filebeat Optional: false aai-filebeat: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-dxj4z: Type: Secret (a volume populated by a Secret) SecretName: default-token-dxj4z Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 4m1s (x3616 over 10h) kubelet, staging-k8s-08 Readiness probe failed: dial tcp 10.42.9.5:9522: connect: connection refused