Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-913

POMBA: 1 of 11 pods failing on sequenced startup on 3.0.0-ONAP - pomba is 22 on the order - looks timing related

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: High High
    • None
    • Casablanca Release

      Need to triage why pomba pods are failing to startup properly - aai and pomba are the remaining problematic pods
      we have until the 10th - move to dublin after that

      env: 3.0.0-ONAP casablanca on 13+1 standard cluster on openlab
      docker images prepulled on all nodes

      POMBA is nearly the last in the cd.sh sequence for LOG-898

           sudo helm deploy onap local/onap --namespace $ENVIRON -f onap/resources/environments/disable-allcharts.yaml -f ~/dev0.yaml --set pomba.enabled=true --set cli.enabled=true --set dcaegen2.enabled=true --set clamp.enabled=true --set appc.enabled=true --set vnfsdk.enabled=true --set uui.enabled=true --set vfc.enabled=true --set log.enabled=true --set portal.enabled=true --set policy.enabled=true --set vid.enabled=true --set sdnc.enabled=true --set sdc.enabled=true --set so.enabled=true --set oof.enabled=true --set multicloud.enabled=true --set esr.enabled=true --set aai.enabled=true --set msb.enabled=true --set dmaap.enabled=true --set aaf.enabled=true --set consul.enabled=true --set robot.enabled=true --verbose
      
      
      ubuntu@onap-oom-obrien-rancher-e0:~$ kubectl get pods --all-namespaces | grep pomba
      onap          onap-pomba-pomba-aaictxbuilder-8565f7d75c-grzb6                2/2       Running            1          2h
      onap          onap-pomba-pomba-contextaggregator-d9f888c4-l9bb9              1/1       Running            47         2h
      onap          onap-pomba-pomba-data-router-66c5544446-ndjcw                  0/1       CrashLoopBackOff   51         2h
      onap          onap-pomba-pomba-elasticsearch-5c4f8c6b5b-rfxkx                1/1       Running            0          2h
      onap          onap-pomba-pomba-kibana-849cb74588-g9922                       1/1       Running            0          2h
      onap          onap-pomba-pomba-networkdiscovery-556bb9d7b8-65lpx             2/2       Running            0          2h
      onap          onap-pomba-pomba-networkdiscoveryctxbuilder-6ff7d49464-4fj8p   2/2       Running            0          2h
      onap          onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4                0/1       Running            47         2h
      onap          onap-pomba-pomba-search-data-9896cf798-ktbgp                   2/2       Running            2          2h
      onap          onap-pomba-pomba-servicedecomposition-5f8bb9d6f6-zzjmp         2/2       Running            0          2h
      onap          onap-pomba-pomba-validation-service-79b8d5cb9f-tsjlq           1/1       Running            0          2h
      
      ubuntu@onap-oom-obrien-rancher-e0:~$ kubectl get pods --all-namespaces | grep pomba
      onap          onap-pomba-pomba-aaictxbuilder-8565f7d75c-grzb6                2/2       Running            1          2h
      onap          onap-pomba-pomba-contextaggregator-d9f888c4-l9bb9              0/1       CrashLoopBackOff   47         2h
      onap          onap-pomba-pomba-data-router-66c5544446-ndjcw                  0/1       CrashLoopBackOff   51         2h
      onap          onap-pomba-pomba-elasticsearch-5c4f8c6b5b-rfxkx                1/1       Running            0          2h
      onap          onap-pomba-pomba-kibana-849cb74588-g9922                       1/1       Running            0          2h
      onap          onap-pomba-pomba-networkdiscovery-556bb9d7b8-65lpx             2/2       Running            0          2h
      onap          onap-pomba-pomba-networkdiscoveryctxbuilder-6ff7d49464-4fj8p   2/2       Running            0          2h
      onap          onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4                0/1       CrashLoopBackOff   47         2h
      onap          onap-pomba-pomba-search-data-9896cf798-ktbgp                   2/2       Running            2          2h
      onap          onap-pomba-pomba-servicedecomposition-5f8bb9d6f6-zzjmp         2/2       Running            0          2h
      onap          onap-pomba-pomba-validation-service-79b8d5cb9f-tsjlq           1/1       Running            0          2h
      ubuntu@onap-oom-obrien-rancher-e0:~$ kubectl describe pod onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4  -n onap
      Name:           onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4
      Namespace:      onap
      Node:           onap-oom-obrien-rancher-e2/10.0.0.23
      Start Time:     Mon, 07 Jan 2019 12:45:29 +0000
      Labels:         app=pomba-sdcctxbuilder
                      pod-template-hash=3355997775
                      release=onap-pomba
      Annotations:    <none>
      Status:         Running
      IP:             10.42.200.42
      Controlled By:  ReplicaSet/onap-pomba-pomba-sdcctxbuilder-7799ffccc9
      Containers:
        pomba-sdcctxbuilder:
          Container ID:   docker://af03be83bc033124c660a46ffd49840c9a878c6e09cac45cf0b293b0b071bc5b
          Image:          nexus3.onap.org:10001/onap/pomba-sdc-context-builder:1.3.2
          Image ID:       docker-pullable://nexus3.onap.org:10001/onap/pomba-sdc-context-builder@sha256:ba5e3c0de81bdb3805a54c30cbd53cb2886b692a7f5fcf3ebd255b5fd9d0a2f9
          Port:           8080/TCP
          Host Port:      0/TCP
          State:          Running
            Started:      Mon, 07 Jan 2019 15:33:34 +0000
          Last State:     Terminated
            Reason:       Error
            Exit Code:    137
            Started:      Mon, 07 Jan 2019 15:27:15 +0000
            Finished:     Mon, 07 Jan 2019 15:28:23 +0000
          Ready:          False
          Restart Count:  48
          Liveness:       tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3
          Readiness:      tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3
          Environment:    <none>
          Mounts:
            /etc/localtime from localtime (ro)
            /opt/app/config/application.properties from onap-pomba-pomba-sdcctxbuilder-config (ro)
            /var/run/secrets/kubernetes.io/serviceaccount from default-token-7zjzw (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        localtime:
          Type:          HostPath (bare host directory volume)
          Path:          /etc/localtime
          HostPathType:  
        onap-pomba-pomba-sdcctxbuilder-config:
          Type:      ConfigMap (a volume populated by a ConfigMap)
          Name:      onap-pomba-pomba-sdcctxbuilder-configmap
          Optional:  false
        default-token-7zjzw:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  default-token-7zjzw
          Optional:    false
      QoS Class:       BestEffort
      Node-Selectors:  <none>
      Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                       node.kubernetes.io/unreachable:NoExecute for 300s
      Events:
        Type     Reason     Age                 From                                 Message
        ----     ------     ----                ----                                 -------
        Warning  Unhealthy  42m (x190 over 2h)  kubelet, onap-oom-obrien-rancher-e2  Readiness probe failed: dial tcp 10.42.200.42:8080: connect: connection refused
        Normal   Pulled     37m (x38 over 2h)   kubelet, onap-oom-obrien-rancher-e2  Container image "nexus3.onap.org:10001/onap/pomba-sdc-context-builder:1.3.2" already present on machine
        Warning  Unhealthy  7m (x138 over 2h)   kubelet, onap-oom-obrien-rancher-e2  Liveness probe failed: dial tcp 10.42.200.42:8080: connect: connection refused
        Warning  BackOff    2m (x515 over 2h)   kubelet, onap-oom-obrien-rancher-e2  Back-off restarting failed container
      
      restarting pomba
      
      
      

            michaelobrien michaelobrien
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: