Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-950

periodic pomba-pomba-networkdiscoveryctxbuilder pod in CrashLoopBackoff due to timing

XMLWordPrintable

      After ONAP is deployed, POMBA Network-discovery-context-builder healthcheck times out, and the pod is in CrashLoopBackOff. 

       

      ------------------------------------------------------------------------------
      Basic Pomba Network-discovery-context-builder Health Check | FAIL |
      Test timeout 10 seconds exceeded.
      ------------------------------------------------------------------------------ 
      

       

       

      root@release-rancher:~# kubectl -n onap get pod |grep pomba
      dev-pomba-pomba-aaictxbuilder-656d96bb6b-lmxqm 2/2 Running 6 13h
      dev-pomba-pomba-contextaggregator-57f47c8564-qtx6x 1/1 Running 0 13h
      dev-pomba-pomba-data-router-769fcc666-x5mfx 1/1 Running 0 13h
      dev-pomba-pomba-elasticsearch-d6589d578-4qqhw 1/1 Running 0 13h
      dev-pomba-pomba-kibana-5d7f684f5b-zxjqv 1/1 Running 0 13h
      dev-pomba-pomba-networkdiscovery-6f4d44fc94-844n5 2/2 Running 0 13h
      dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc-lwfzb 1/2 CrashLoopBackOff 225 13h
      dev-pomba-pomba-sdcctxbuilder-7889df5f7-wnglf 1/1 Running 0 13h
      dev-pomba-pomba-search-data-dc47bbcb8-6fxcb 2/2 Running 0 13h
      dev-pomba-pomba-servicedecomposition-567c57859-rk5sl 2/2 Running 5 13h
      dev-pomba-pomba-validation-service-785fdd58dc-6n6tb 1/1 Running 0 13h
      

       

      The output of describe the pod:

       

      root@release-rancher:~# kubectl -n onap describe pod dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc-lwfzb 
      Name: dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc-lwfzb
      Namespace: onap
      Node: release-k8s-08/10.0.0.10
      Start Time: Mon, 28 Jan 2019 12:35:03 +0000
      Labels: app=pomba-networkdiscoveryctxbuilder
       pod-template-hash=2114668097
       release=dev-pomba
      Annotations: <none>
      Status: Running
      IP: 10.42.229.209
      Controlled By: ReplicaSet/dev-pomba-pomba-networkdiscoveryctxbuilder-6558bbd4fc
      Containers:
       pomba-networkdiscoveryctxbuilder:
       Container ID: docker://c5f3f6aac76c275136217dd2160057157fce180bee76c73e643b8410304aa243
       Image: 10.12.5.2:5000/onap/pomba-network-discovery-context-builder:1.3.1
       Image ID: docker-pullable://10.12.5.2:5000/onap/pomba-network-discovery-context-builder@sha256:3488e02cc06441616b971f9ade079aad49c874828b97ce106ad325053b7da955
       Port: 8080/TCP
       Host Port: 0/TCP
       State: Waiting
       Reason: CrashLoopBackOff
       Last State: Terminated
       Reason: Error
       Exit Code: 137
       Started: Tue, 29 Jan 2019 02:31:42 +0000
       Finished: Tue, 29 Jan 2019 02:32:54 +0000
       Ready: False
       Restart Count: 225
       Liveness: tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3
       Readiness: tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3
       Environment: <none>
       Mounts:
       /etc/localtime from localtime (ro)
       /opt/app/config/application.properties from dev-pomba-pomba-networkdiscoveryctxbuilder-config (ro)
       /opt/app/config/logback.xml from dev-pomba-pomba-networkdiscoveryctxbuilder-log-conf (rw)
       /var/log/onap from dev-pomba-pomba-networkdiscoveryctxbuilder-logs (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from default-token-pf8lf (ro)
       pomba-networkdiscoveryctxbuilder-filebeat-onap:
       Container ID: docker://5b9de52de8f0d0190fd2bd31953a114a9432032f6e87941828f5e580dd7b8d12
       Image: docker.elastic.co/beats/filebeat:5.5.0
       Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942
       Port: <none>
       Host Port: <none>
       State: Running
       Started: Mon, 28 Jan 2019 12:49:50 +0000
       Ready: True
       Restart Count: 0
       Environment: <none>
       Mounts:
       /usr/share/filebeat/data from dev-pomba-pomba-networkdiscoveryctxbuilder-data-filebeat (rw)
       /usr/share/filebeat/filebeat.yml from dev-pomba-pomba-networkdiscoveryctxbuilder-filebeat-conf (rw)
       /var/log/onap from dev-pomba-pomba-networkdiscoveryctxbuilder-logs (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from default-token-pf8lf (ro)
      Conditions:
       Type Status
       Initialized True 
       Ready False 
       ContainersReady False 
       PodScheduled True 
      Volumes:
       localtime:
       Type: HostPath (bare host directory volume)
       Path: /etc/localtime
       HostPathType: 
       dev-pomba-pomba-networkdiscoveryctxbuilder-config:
       Type: ConfigMap (a volume populated by a ConfigMap)
       Name: dev-pomba-pomba-networkdiscoveryctxbuilder-configmap
       Optional: false
       dev-pomba-pomba-networkdiscoveryctxbuilder-log-conf:
       Type: ConfigMap (a volume populated by a ConfigMap)
       Name: dev-pomba-pomba-networkdiscoveryctxbuilder-log
       Optional: false
       dev-pomba-pomba-networkdiscoveryctxbuilder-filebeat-conf:
       Type: ConfigMap (a volume populated by a ConfigMap)
       Name: dev-pomba-pomba-filebeat-configmap
       Optional: false
       dev-pomba-pomba-networkdiscoveryctxbuilder-data-filebeat:
       Type: EmptyDir (a temporary directory that shares a pod's lifetime)
       Medium: 
       dev-pomba-pomba-networkdiscoveryctxbuilder-logs:
       Type: EmptyDir (a temporary directory that shares a pod's lifetime)
       Medium: 
       default-token-pf8lf:
       Type: Secret (a volume populated by a Secret)
       SecretName: default-token-pf8lf
       Optional: false
      QoS Class: BestEffort
      Node-Selectors: <none>
      Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
       node.kubernetes.io/unreachable:NoExecute for 300s
      Events:
       Type Reason Age From Message
       ---- ------ ---- ---- -------
       Warning Unhealthy 9m (x1271 over 13h) kubelet, release-k8s-08 Readiness probe failed: dial tcp 10.42.229.209:8080: connect: connection refused
       Warning BackOff 4m (x2559 over 13h) kubelet, release-k8s-08 Back-off restarting failed container 
      

       

       

            yong_chen yong_chen
            xuyang11 xuyang11
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: