When deploying ONAP Casablanca using OOM, only 1 docker is in CrashLoopBackoff at the end of the instalation

onap-aai-aai-data-router-865c68f454-jl7w9 1/2 CrashLoopBackOff 112 9h

logs are not very verbose...

kubectl logs onap-aai-aai-data-router-865c68f454-jl7w9 -c aai-data-router -n onap
BOOT-INF/lib/ : no such file or directory*

. ____ _ __ _ _
/
/ __' __ _ ()_ __ __ _ \ \ \ \
( ( )___ | '_ | '| | ' \/ _` | \ \ \ \
/ __)| |)| | | | | || (_| | ) ) ) )
' |___| .|| ||| |_, | / / / /
=========||==============|_/=////
:: Spring Boot :: (v1.5.18.RELEASE)

kubectl describe pod onap-aai-aai-data-router-865c68f454-jl7w9 -n onap
Name: onap-aai-aai-data-router-865c68f454-jl7w9
Namespace: onap
Node: compute06-onap-casablanca.novalocal/10.253.0.9
Start Time: Fri, 08 Feb 2019 00:16:26 +0000
Labels: app=aai-data-router
pod-template-hash=4217249010
release=onap-aai
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"onap","name":"onap-aai-aai-data-router-865c68f454","uid":"c6101cfb-2b36-11e9-84c3...
Status: Running
IP: 10.42.114.200
Created By: ReplicaSet/onap-aai-aai-data-router-865c68f454
Controlled By: ReplicaSet/onap-aai-aai-data-router-865c68f454
Init Containers:
init-sysctl:
Container ID: docker://9f4527f7961c2508ba0abb3a302d282b1a8fc501efd3817c01f6aad216008950
Image: docker.io/busybox
Image ID: docker-pullable://busybox@sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0
Port: <none>
Command:
/bin/sh
-c
mkdir -p /logroot/data-router/logs
chmod -R 777 /logroot/data-router/logs
chown -R root:root /logroot

State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 08 Feb 2019 00:19:25 +0000
Finished: Fri, 08 Feb 2019 00:19:25 +0000
Ready: True
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/logroot/ from onap-aai-aai-data-router-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ncq8d (ro)
Containers:
aai-data-router:
Container ID: docker://79a24fb963cbfc0facf205d90ff7ed292a331a0059ab085f5e02ab0e308ebd1c
Image: nexus3.onap.org:10001/onap/data-router:1.3.2
Image ID: docker-pullable://nexus3.onap.org:10001/onap/data-router@sha256:19da04c8ed67e0e82ea813f42141231d7ea8e4dd8598a8b96c4ba556feee3c7a
Port: 9502/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 08 Feb 2019 09:50:59 +0000
Finished: Fri, 08 Feb 2019 09:51:04 +0000
Ready: False
Restart Count: 113
Liveness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
SERVICE_BEANS: /opt/app/data-router/dynamic/conf
CONFIG_HOME: /opt/app/data-router/config/
KEY_STORE_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
DYNAMIC_ROUTES: /opt/app/data-router/dynamic/routes
KEY_MANAGER_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JAVA_HOME: usr/lib/jvm/java-8-openjdk-amd64
Mounts:
/etc/localtime from localtime (ro)
/logs from onap-aai-aai-data-router-logs (rw)
/opt/app/data-router/bundleconfig/etc/logback.xml from onap-aai-aai-data-router-logback-config (rw)
/opt/app/data-router/config/auth from onap-aai-aai-data-router-auth (rw)
/opt/app/data-router/config/data-router.properties from onap-aai-aai-data-router-properties (rw)
/opt/app/data-router/config/schemaIngest.properties from onap-aai-aai-data-router-properties (rw)
/opt/app/data-router/dynamic/conf/data-router-oxm.xml from onap-aai-aai-data-router-dynamic-oxm (rw)
/opt/app/data-router/dynamic/conf/entity-event-policy.xml from onap-aai-aai-data-router-dynamic-policy (rw)
/opt/app/data-router/dynamic/routes/entity-event.route from onap-aai-aai-data-router-dynamic-route (rw)
/var/log/onap from onap-aai-aai-data-router-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ncq8d (ro)
filebeat-onap:
Container ID: docker://20f30a3d9d08e966615df87ce5ef6d2ff67e3a2c0e43165c6b98018898abe07c
Image: docker.elastic.co/beats/filebeat:5.5.0
Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942
Port: <none>
State: Running
Started: Fri, 08 Feb 2019 00:25:33 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/logs from onap-aai-aai-data-router-logs (rw)
/usr/share/filebeat/data from aai-filebeat (rw)
/usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
/var/log/onap from onap-aai-aai-data-router-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ncq8d (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
filebeat-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: aai-filebeat
Optional: false
aai-filebeat:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
onap-aai-aai-data-router-auth:
Type: Secret (a volume populated by a Secret)
SecretName: onap-aai-aai-data-router
Optional: false
onap-aai-aai-data-router-properties:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-prop
Optional: false
onap-aai-aai-data-router-dynamic-route:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-dynamic
Optional: false
onap-aai-aai-data-router-dynamic-policy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-dynamic
Optional: false
onap-aai-aai-data-router-dynamic-oxm:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-dynamic
Optional: false
onap-aai-aai-data-router-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
onap-aai-aai-data-router-logback-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-log-configmap
Optional: false
default-token-ncq8d:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ncq8d
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedSync 7m (x2496 over 9h) kubelet, compute06-onap-casablanca.novalocal Error syncing pod
Warning BackOff 2m (x2517 over 9h) kubelet, compute06-onap-casablanca.novalocal Back-off restarting failed container

Defaulting container name to aai-data-router.
Use 'kubectl describe pod/onap-aai-aai-data-router-865c68f454-jl7w9' to see all of the containers in this pod.
error: unable to upgrade connection: container not found ("aai-data-router")

An attempt of restart does not fix the problem

onap-aai-aai-data-router-865c68f454-xr549 1/2 Error 2 1m

kubectl describe pod onap-aai-aai-data-router-865c68f454-xr549 -n onap
Name: onap-aai-aai-data-router-865c68f454-xr549
Namespace: onap
Node: compute01-onap-casablanca.novalocal/10.253.0.3
Start Time: Fri, 08 Feb 2019 09:56:02 +0000
Labels: app=aai-data-router
pod-template-hash=4217249010
release=onap-aai
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"onap","name":"onap-aai-aai-data-router-865c68f454","uid":"c6101cfb-2b36-11e9-84c3...
Status: Running
IP: 10.42.250.123
Created By: ReplicaSet/onap-aai-aai-data-router-865c68f454
Controlled By: ReplicaSet/onap-aai-aai-data-router-865c68f454
Init Containers:
init-sysctl:
Container ID: docker://c5d0b81e92183f54fd0c3f1b06dd7e59bbf81c51805c5a67ab3c53a8701d4f21
Image: docker.io/busybox
Image ID: docker-pullable://busybox@sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0
Port: <none>
Command:
/bin/sh
-c
mkdir -p /logroot/data-router/logs
chmod -R 777 /logroot/data-router/logs
chown -R root:root /logroot

State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 08 Feb 2019 09:56:21 +0000
Finished: Fri, 08 Feb 2019 09:56:21 +0000
Ready: True
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/logroot/ from onap-aai-aai-data-router-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ncq8d (ro)
Containers:
aai-data-router:
Container ID: docker://fa2943b49ee87c5cc918f41352a3bc36af5fa62c8eecf640a45f39c593150a3c
Image: nexus3.onap.org:10001/onap/data-router:1.3.2
Image ID: docker-pullable://nexus3.onap.org:10001/onap/data-router@sha256:19da04c8ed67e0e82ea813f42141231d7ea8e4dd8598a8b96c4ba556feee3c7a
Port: 9502/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 08 Feb 2019 09:59:21 +0000
Finished: Fri, 08 Feb 2019 09:59:27 +0000
Ready: False
Restart Count: 4
Liveness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
SERVICE_BEANS: /opt/app/data-router/dynamic/conf
CONFIG_HOME: /opt/app/data-router/config/
KEY_STORE_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
DYNAMIC_ROUTES: /opt/app/data-router/dynamic/routes
KEY_MANAGER_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JAVA_HOME: usr/lib/jvm/java-8-openjdk-amd64
Mounts:
/etc/localtime from localtime (ro)
/logs from onap-aai-aai-data-router-logs (rw)
/opt/app/data-router/bundleconfig/etc/logback.xml from onap-aai-aai-data-router-logback-config (rw)
/opt/app/data-router/config/auth from onap-aai-aai-data-router-auth (rw)
/opt/app/data-router/config/data-router.properties from onap-aai-aai-data-router-properties (rw)
/opt/app/data-router/config/schemaIngest.properties from onap-aai-aai-data-router-properties (rw)
/opt/app/data-router/dynamic/conf/data-router-oxm.xml from onap-aai-aai-data-router-dynamic-oxm (rw)
/opt/app/data-router/dynamic/conf/entity-event-policy.xml from onap-aai-aai-data-router-dynamic-policy (rw)
/opt/app/data-router/dynamic/routes/entity-event.route from onap-aai-aai-data-router-dynamic-route (rw)
/var/log/onap from onap-aai-aai-data-router-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ncq8d (ro)
filebeat-onap:
Container ID: docker://82d728aecf06cf6afb315a2d094c8c1d66c1db1fdc821b5e912a39fb087a735e
Image: docker.elastic.co/beats/filebeat:5.5.0
Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942
Port: <none>
State: Running
Started: Fri, 08 Feb 2019 09:57:21 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/logs from onap-aai-aai-data-router-logs (rw)
/usr/share/filebeat/data from aai-filebeat (rw)
/usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
/var/log/onap from onap-aai-aai-data-router-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ncq8d (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
filebeat-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: aai-filebeat
Optional: false
aai-filebeat:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
onap-aai-aai-data-router-auth:
Type: Secret (a volume populated by a Secret)
SecretName: onap-aai-aai-data-router
Optional: false
onap-aai-aai-data-router-properties:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-prop
Optional: false
onap-aai-aai-data-router-dynamic-route:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-dynamic
Optional: false
onap-aai-aai-data-router-dynamic-policy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-dynamic
Optional: false
onap-aai-aai-data-router-dynamic-oxm:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-dynamic
Optional: false
onap-aai-aai-data-router-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
onap-aai-aai-data-router-logback-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-data-router-log-configmap
Optional: false
default-token-ncq8d:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ncq8d
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m default-scheduler Successfully assigned onap-aai-aai-data-router-865c68f454-xr549 to compute01-onap-casablanca.novalocal
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "localtime"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-data-router-logback-config"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-data-router-logs"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "aai-filebeat"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-data-router-properties"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-data-router-dynamic-policy"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "filebeat-conf"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-data-router-dynamic-route"
Normal SuccessfulMountVolume 3m kubelet, compute01-onap-casablanca.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-data-router-dynamic-oxm"
Normal SuccessfulMountVolume 3m (x2 over 3m) kubelet, compute01-onap-casablanca.novalocal (combined from similar events): MountVolume.SetUp succeeded for volume "default-token-ncq8d"
Normal Pulling 3m kubelet, compute01-onap-casablanca.novalocal pulling image "docker.io/busybox"
Normal Pulled 3m kubelet, compute01-onap-casablanca.novalocal Successfully pulled image "docker.io/busybox"
Normal Created 3m kubelet, compute01-onap-casablanca.novalocal Created container
Normal Started 3m kubelet, compute01-onap-casablanca.novalocal Started container
Normal Created 2m kubelet, compute01-onap-casablanca.novalocal Created container
Normal Started 2m kubelet, compute01-onap-casablanca.novalocal Started container
Normal Pulling 2m kubelet, compute01-onap-casablanca.novalocal pulling image "docker.elastic.co/beats/filebeat:5.5.0"
Normal Pulled 2m kubelet, compute01-onap-casablanca.novalocal Successfully pulled image "docker.elastic.co/beats/filebeat:5.5.0"
Normal Created 2m kubelet, compute01-onap-casablanca.novalocal Created container
Normal Started 2m kubelet, compute01-onap-casablanca.novalocal Started container
Normal Pulling 2m (x2 over 3m) kubelet, compute01-onap-casablanca.novalocal pulling image "nexus3.onap.org:10001/onap/data-router:1.3.2"
Normal Pulled 2m (x2 over 2m) kubelet, compute01-onap-casablanca.novalocal Successfully pulled image "nexus3.onap.org:10001/onap/data-router:1.3.2"

kubectl logs onap-aai-aai-data-router-865c68f454-xr549 -c aai-data-router -n onap
BOOT-INF/lib/ : no such file or directory*