POMBA: 1 of 11 pods failing on sequenced startup on 3.0.0-ONAP - pomba is 22 on the order - looks timing related

     sudo helm deploy onap local/onap --namespace $ENVIRON -f onap/resources/environments/disable-allcharts.yaml -f ~/dev0.yaml --set pomba.enabled=true --set cli.enabled=true --set dcaegen2.enabled=true --set clamp.enabled=true --set appc.enabled=true --set vnfsdk.enabled=true --set uui.enabled=true --set vfc.enabled=true --set log.enabled=true --set portal.enabled=true --set policy.enabled=true --set vid.enabled=true --set sdnc.enabled=true --set sdc.enabled=true --set so.enabled=true --set oof.enabled=true --set multicloud.enabled=true --set esr.enabled=true --set aai.enabled=true --set msb.enabled=true --set dmaap.enabled=true --set aaf.enabled=true --set consul.enabled=true --set robot.enabled=true --verbose


ubuntu@onap-oom-obrien-rancher-e0:~$ kubectl get pods --all-namespaces | grep pomba
onap          onap-pomba-pomba-aaictxbuilder-8565f7d75c-grzb6                2/2       Running            1          2h
onap          onap-pomba-pomba-contextaggregator-d9f888c4-l9bb9              1/1       Running            47         2h
onap          onap-pomba-pomba-data-router-66c5544446-ndjcw                  0/1       CrashLoopBackOff   51         2h
onap          onap-pomba-pomba-elasticsearch-5c4f8c6b5b-rfxkx                1/1       Running            0          2h
onap          onap-pomba-pomba-kibana-849cb74588-g9922                       1/1       Running            0          2h
onap          onap-pomba-pomba-networkdiscovery-556bb9d7b8-65lpx             2/2       Running            0          2h
onap          onap-pomba-pomba-networkdiscoveryctxbuilder-6ff7d49464-4fj8p   2/2       Running            0          2h
onap          onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4                0/1       Running            47         2h
onap          onap-pomba-pomba-search-data-9896cf798-ktbgp                   2/2       Running            2          2h
onap          onap-pomba-pomba-servicedecomposition-5f8bb9d6f6-zzjmp         2/2       Running            0          2h
onap          onap-pomba-pomba-validation-service-79b8d5cb9f-tsjlq           1/1       Running            0          2h

ubuntu@onap-oom-obrien-rancher-e0:~$ kubectl get pods --all-namespaces | grep pomba
onap          onap-pomba-pomba-aaictxbuilder-8565f7d75c-grzb6                2/2       Running            1          2h
onap          onap-pomba-pomba-contextaggregator-d9f888c4-l9bb9              0/1       CrashLoopBackOff   47         2h
onap          onap-pomba-pomba-data-router-66c5544446-ndjcw                  0/1       CrashLoopBackOff   51         2h
onap          onap-pomba-pomba-elasticsearch-5c4f8c6b5b-rfxkx                1/1       Running            0          2h
onap          onap-pomba-pomba-kibana-849cb74588-g9922                       1/1       Running            0          2h
onap          onap-pomba-pomba-networkdiscovery-556bb9d7b8-65lpx             2/2       Running            0          2h
onap          onap-pomba-pomba-networkdiscoveryctxbuilder-6ff7d49464-4fj8p   2/2       Running            0          2h
onap          onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4                0/1       CrashLoopBackOff   47         2h
onap          onap-pomba-pomba-search-data-9896cf798-ktbgp                   2/2       Running            2          2h
onap          onap-pomba-pomba-servicedecomposition-5f8bb9d6f6-zzjmp         2/2       Running            0          2h
onap          onap-pomba-pomba-validation-service-79b8d5cb9f-tsjlq           1/1       Running            0          2h
ubuntu@onap-oom-obrien-rancher-e0:~$ kubectl describe pod onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4  -n onap
Name:           onap-pomba-pomba-sdcctxbuilder-7799ffccc9-2jjg4
Namespace:      onap
Node:           onap-oom-obrien-rancher-e2/10.0.0.23
Start Time:     Mon, 07 Jan 2019 12:45:29 +0000
Labels:         app=pomba-sdcctxbuilder
                pod-template-hash=3355997775
                release=onap-pomba
Annotations:    <none>
Status:         Running
IP:             10.42.200.42
Controlled By:  ReplicaSet/onap-pomba-pomba-sdcctxbuilder-7799ffccc9
Containers:
  pomba-sdcctxbuilder:
    Container ID:   docker://af03be83bc033124c660a46ffd49840c9a878c6e09cac45cf0b293b0b071bc5b
    Image:          nexus3.onap.org:10001/onap/pomba-sdc-context-builder:1.3.2
    Image ID:       docker-pullable://nexus3.onap.org:10001/onap/pomba-sdc-context-builder@sha256:ba5e3c0de81bdb3805a54c30cbd53cb2886b692a7f5fcf3ebd255b5fd9d0a2f9
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 07 Jan 2019 15:33:34 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 07 Jan 2019 15:27:15 +0000
      Finished:     Mon, 07 Jan 2019 15:28:23 +0000
    Ready:          False
    Restart Count:  48
    Liveness:       tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      tcp-socket :8080 delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /etc/localtime from localtime (ro)
      /opt/app/config/application.properties from onap-pomba-pomba-sdcctxbuilder-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7zjzw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  localtime:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:  
  onap-pomba-pomba-sdcctxbuilder-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      onap-pomba-pomba-sdcctxbuilder-configmap
    Optional:  false
  default-token-7zjzw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7zjzw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From                                 Message
  ----     ------     ----                ----                                 -------
  Warning  Unhealthy  42m (x190 over 2h)  kubelet, onap-oom-obrien-rancher-e2  Readiness probe failed: dial tcp 10.42.200.42:8080: connect: connection refused
  Normal   Pulled     37m (x38 over 2h)   kubelet, onap-oom-obrien-rancher-e2  Container image "nexus3.onap.org:10001/onap/pomba-sdc-context-builder:1.3.2" already present on machine
  Warning  Unhealthy  7m (x138 over 2h)   kubelet, onap-oom-obrien-rancher-e2  Liveness probe failed: dial tcp 10.42.200.42:8080: connect: connection refused
  Warning  BackOff    2m (x515 over 2h)   kubelet, onap-oom-obrien-rancher-e2  Back-off restarting failed container

restarting pomba