When performing E2E tests, there are errors during onboarding (even if Onboarding OK is displayed in test traces).

There is a distribution error for the A&AI in the SDC monitoring page.

=>following AAI logs

During distribution, there is an error reported by model loader

The model loader logs redirect to the aai-babel microservice

we can see in th aaai-babel logs

Exception CsarConverterException: An error occurred trying to generate XML files from a collection of YAML files : org.onap.aai.babel.xml.generator.XmlArtifactGenerationException: Error occurred during artifact generation: {*AAI=[Cannot generate artifacts. Artifact Generator Configuration file not found at /opt/app/babel/config/filter-types.properties*]}
at org.onap.aai.babel.csar.CsarToXmlConverter.generateXmlFromCsar(CsarToXmlConverter.java:79)
at org.onap.aai.babel.service.GenerateArtifactsServiceImpl.generateArtifacts(GenerateArtifactsServiceImpl.java:150)
at org.onap.aai.babel.service.GenerateArtifactsServiceImpl.generateArtifacts(GenerateArtifactsServiceImpl.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
... 67 more frames

And effectively the file does not exist

babel/configi-aai-babel-9dd8bd6dc-5qhhw:/var/log/onap/AAI-BAS# ls -lah /opt/app/
total 36K
drwxr-xr-x 3 root root 130 Feb 12 04:16 .
drwxr-xr-x 1 root root 20 Feb 12 04:16 ..
rw-rr- 1 root root 16K Feb 12 04:14 artifact-generator.properties
drwxrwxrwt 3 root root 120 Feb 12 04:14 auth
rw-rr- 1 root root 668 Feb 12 04:14 babel-auth.properties
rw-rr- 1 root root 8.3K Feb 12 04:14 logback.xml
rw-rr- 1 root root 1.1K Feb 12 04:14 tosca-mappings.json

side effect?

another aai module (aai-gizmo) is in CrashloopBackOff

kubectl get pods -n onap |grep -v Running
NAME READY STATUS RESTARTS AGE
onap-aai-aai-gizmo-644fdfdc85-klt97 1/2 CrashLoopBackOff 11 34m
onap-dcaegen2-dcae-inventory-api-6c9b856894-g7lfj 0/1 InvalidImageName 0 6h
onap-dcaegen2-dcae-servicechange-handler-7d9d7c546-2gd9b 0/1 Init:0/1 8 6h
onap-multicloud-multicloud-vio-b7679f7d7-mk2cn 1/2 CrashLoopBackOff 82 6h
onap-pomba-pomba-networkdiscovery-5fdb88f4cc-2vmvd 1/2 CrashLoopBackOff 74 6h
onap-pomba-pomba-validation-service-6cb545d89c-cg8x6 0/1 CrashLoopBackOff 74 6h
onap-sdnc-controller-blueprints-6547954976-f24kw 0/1 CrashLoopBackOff 80 6h
onap-sdnc-sdnc-ansible-server-7759c997d7-sd7lx 0/1 CrashLoopBackOff 71 6h

debian@control01-onap-master:~$ kubectl describe pods -n onap onap-aai-aai-gizmo-644fdfdc85-klt97
Name: onap-aai-aai-gizmo-644fdfdc85-klt97
Namespace: onap
Node: compute09-onap-master.novalocal/10.253.0.9
Start Time: Tue, 12 Feb 2019 10:23:54 +0000
Labels: app=aai-gizmo
pod-template-hash=2009898741
release=onap-aai
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference"{"kind":"ReplicaSet","namespace""onap","name":"onap-aai-aai-gizmo-644fdfdc85","uid":"c1244d55-2e7c-11e9-bbd3-0238c...
Status: Running
IP: 10.42.22.70
Created By: ReplicaSet/onap-aai-aai-gizmo-644fdfdc85
Controlled By: ReplicaSet/onap-aai-aai-gizmo-644fdfdc85
Containers:
aai-gizmo:
Container ID: docker://edb6c47f9dd8b97febfbd8c73afcf218e9427a17dec326102e14ce00f9461b5e
Image: nexus3.onap.org:10001/onap/gizmo:1.3-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/gizmo@sha256:62c5d92cb74669f3c6d8e0012d29ca8b793ace694792c7ee398fdb3c31c70ef1
Port: 9520/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 12 Feb 2019 10:57:14 +0000
Finished: Tue, 12 Feb 2019 10:57:19 +0000
Ready: False
Restart Count: 11
Liveness: tcp-socket :9520 delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: tcp-socket :9520 delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
CONFIG_HOME: /opt/app/crud-service/config/
KEY_STORE_PASSWORD: <set to the key 'KEY_STORE_PASSWORD' in secret 'onap-aai-aai-gizmo-pass'> Optional: false
KEY_MANAGER_PASSWORD: <set to the key 'KEY_MANAGER_PASSWORD' in secret 'onap-aai-aai-gizmo-pass'> Optional: false
SERVICE_BEANS: /opt/app/crud-service/dynamic/conf
Mounts:
/etc/localtime from localtime (ro)
/opt/app/crud-api/bundleconfig/etc/logback.xml from onap-aai-aai-gizmo-logback-config (rw)
/opt/app/crud-service/config/auth from onap-aai-aai-gizmo-auth-secret (rw)
/opt/app/crud-service/config/crud-api.properties from onap-aai-aai-gizmo-config (rw)
/opt/app/crud-service/config/edgeprops-ingest.properties from onap-aai-aai-gizmo-config (rw)
/opt/app/crud-service/config/model/ from onap-aai-aai-gizmo-model-config (rw)
/opt/app/crud-service/config/schema-ingest.properties from onap-aai-aai-gizmo-config (rw)
/opt/app/crud-service/dynamic/conf/crud-beans.xml from onap-aai-aai-gizmo-config (rw)
/var/log/onap from onap-aai-aai-gizmo-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ldtmj (ro)
filebeat-onap:
Container ID: docker://f636d9fb677b8d13dd3d3a1bcced3d71bf5bfa2cf8ae0c0effe480670d6c9818
Image: docker.elastic.co/beats/filebeat:5.5.0
Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942
Port: <none>
State: Running
Started: Tue, 12 Feb 2019 10:24:25 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/filebeat/data from onap-aai-aai-gizmo-data-filebeat (rw)
/usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
/var/log/onap from onap-aai-aai-gizmo-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ldtmj (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
onap-aai-aai-gizmo-data-filebeat:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
filebeat-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-gizmo-filebeat-configmap
Optional: false
onap-aai-aai-gizmo-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
onap-aai-aai-gizmo-auth-secret:
Type: Secret (a volume populated by a Secret)
SecretName: onap-aai-aai-gizmo-auth
Optional: false
onap-aai-aai-gizmo-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-gizmo-configmap
Optional: false
onap-aai-aai-gizmo-logback-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-gizmo-log-configmap
Optional: false
onap-aai-aai-gizmo-model-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-aai-aai-gizmo-model-configmap
Optional: false
default-token-ldtmj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ldtmj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 35m default-scheduler Successfully assigned onap-aai-aai-gizmo-644fdfdc85-klt97 to compute09-onap-master.novalocal
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "localtime"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-gizmo-logs"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-gizmo-data-filebeat"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "filebeat-conf"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-gizmo-config"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-gizmo-model-config"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-gizmo-auth-secret"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "onap-aai-aai-gizmo-logback-config"
Normal SuccessfulMountVolume 35m kubelet, compute09-onap-master.novalocal MountVolume.SetUp succeeded for volume "default-token-ldtmj"
Normal Pulling 35m kubelet, compute09-onap-master.novalocal pulling image "docker.elastic.co/beats/filebeat:5.5.0"
Normal Started 35m kubelet, compute09-onap-master.novalocal Started container
Normal Created 35m kubelet, compute09-onap-master.novalocal Created container
Normal Pulled 35m kubelet, compute09-onap-master.novalocal Successfully pulled image "docker.elastic.co/beats/filebeat:5.5.0"
Normal Pulling 35m (x2 over 35m) kubelet, compute09-onap-master.novalocal pulling image "nexus3.onap.org:10001/onap/gizmo:1.3-STAGING-latest"
Normal Started 35m (x2 over 35m) kubelet, compute09-onap-master.novalocal Started container
Normal Created 35m (x2 over 35m) kubelet, compute09-onap-master.novalocal Created container
Normal Pulled 35m (x2 over 35m) kubelet, compute09-onap-master.novalocal Successfully pulled image "nexus3.onap.org:10001/onap/gizmo:1.3-STAGING-latest"
Warning FailedSync 5m (x135 over 35m) kubelet, compute09-onap-master.novalocal Error syncing pod
Warning BackOff 35s (x158 over 35m) kubelet, compute09-onap-master.novalocal Back-off restarting failed container