-
Bug
-
Resolution: Cannot Reproduce
-
Medium
-
None
-
Guilin Release
-
None
-
None
-
Hide
SUMMARY
We have deployed ONAP on RKE cluster based on OOM Quick Start Guide as per below documentation guide, where we have configured RKE cluster on AWS vm instances (3 control plane + 12 worker nodes)
https://docs.onap.org/projects/onap-oom/en/latest/oom_quickstart_guide.html#oom-quickstart-guideWhen we are bringing down/up the control plan & worker node AWS instances after they come up, we noticed that the below applications health checks are getting
failed (Before bringing down the VMs, they are all fine) and some of the pods went into "Init:CrashLoopBackOff" state with "Back-off restarting failed container" as per kubectl describe.Looks like it could be an issue while bringing down the nodes & after comes back up, so filing this bug under OOM project
OS / ENVIRONMENT
- Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} - Helm version:
Client: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
OOM VERSION
seen on Master (G)
CONFIGURATION
~/oom/kubernetes/onap/resources/overrides/onap-all.yaml
# Copyright © 2019 Amdocs, Bell Canada # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ################################################################### # This override file enables helm charts for all ONAP applications. ################################################################### global: addTestingComponents: &testing true centralizedLoggingEnabled: ¢ralizedLogging false cassandra: enabled: true mariadb-galera: enabled: true aaf: enabled: true aai: enabled: true appc: enabled: true cds: enabled: true clamp: enabled: true cli: enabled: true # Today, "contrib" chart that hosting these components must also be enabled # in order to make it work. So `contrib.enabled` must have the same value than # addTestingComponents contrib: enabled: *testing consul: enabled: true dcaegen2: enabled: true dcaemod: enabled: true dmaap: enabled: true esr: enabled: true oof: enabled: true msb: enabled: true multicloud: enabled: true nbi: enabled: true policy: enabled: true portal: enabled: true robot: enabled: true sdc: enabled: true sdnc: enabled: true so: enabled: true uui: enabled: true vfc: enabled: true vid: enabled: true vnfsdk: enabled: true modeling: enabled: true
STEPS TO REPRODUCE
- make oom with master build
- deploy ONAP on RKE cluster (3 control plane & 12 worker AWS instance )as below:
nohup helm deploy dev local/onap --namespace onap --set global.masterPassword=onap -f onap/resources/overrides/onap-all.yaml
-f onap/resources/overrides/environment.yaml -f onap/resources/overrides/openstack.yaml --timeout 900 > /home/ubuntu/full-onap-helm-deploy_new.log & - After successful deployment, bring donw/up control plane and worker instances
- Check for ./ete-k8s.sh onap health and kubctl get pods -n onap
EXPECTED RESULTS
All pods should be up & running and robot health tests should PASS without any failures
ACTUAL RESULTS
Currently we noticed there below application health check failures & pods failures
./ete-k8s.sh onap health
++ export NAMESPACE=onap ++ NAMESPACE=onap +++ kubectl --namespace onap get pods +++ sed 's/ .*//' +++ grep robot ++ POD=dev-robot-578b965f4d-w7p9t ++ TAGS='-i health' +++ dirname ./ete-k8s.sh ++ DIR=. ++ SCRIPTDIR=scripts/etescript ++ ETEHOME=/var/opt/ONAP ++ [[ health == \e\x\e\c\s\c\r\i\p\t ]] +++ kubectl --namespace onap exec dev-robot-578b965f4d-w7p9t -- bash -c 'ls -1q /share/logs/ | wc -l' ++ export GLOBAL_BUILD_NUMBER=34 ++ GLOBAL_BUILD_NUMBER=34 +++ printf %04d 34 ++ OUTPUT_FOLDER=0034_ete_health ++ DISPLAY_NUM=124 ++ VARIABLEFILES='-V /share/config/robot_properties.py' ++ VARIABLES='-v GLOBAL_BUILD_NUMBER:1035' ++ case $2 in ++ kubectl --namespace onap exec dev-robot-578b965f4d-w7p9t -- /var/opt/ONAP/runTags.sh -V /share/config/robot_properties.py -v GLOBAL_BUILD_NUMBER:1035 -d /share/logs/0034_ete_health -i health --display 124 Starting Xvfb on display :124 with res 1280x1024x24 Executing robot tests at log level TRACE ============================================================================== Testsuites ============================================================================== Testsuites.Health-Check :: Test that ONAP components are available via basi... ============================================================================== Basic A&AI Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF SMS Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic APPC Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic CLI Health Check | PASS | ------------------------------------------------------------------------------ Basic CLAMP Health Check | PASS | ------------------------------------------------------------------------------ Basic DCAE Health Check | FAIL | 500 != 200 ------------------------------------------------------------------------------ Basic DMAAP Data Router Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf25718d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf256b310>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf2decdd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv | FAIL | ConnectionError: HTTPSConnectionPool(host='dmaap-dr-node.onap', port=8443): Max retries exceeded with url: /internal/fetchProv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf4c18650>: Failed to establish a new connection: [Errno 111] Connection refused',)) ------------------------------------------------------------------------------ Basic DMAAP Message Router Health Check | PASS | ------------------------------------------------------------------------------ Basic DMAAP Bus Controller Health Check With Basic Auth | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic External API NBI Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic Microservice Bus Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-pike API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-starlingx API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-titanium_cloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-vio API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-k8s API Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-Homing Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic OOF-SNIRO Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic OOF-CMSO Health Check | FAIL | 401 != 200 ------------------------------------------------------------------------------ Basic Policy Health Check | PASS | ------------------------------------------------------------------------------ Basic Policy New Healthcheck | PASS | ------------------------------------------------------------------------------ Basic Portal Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic SDC Health Check (DMaaP:UP) | PASS | ------------------------------------------------------------------------------ Basic SDNC Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic SO Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf24cce50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf24bb790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf2466690>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health | FAIL | ConnectionError: HTTPConnectionPool(host='so.onap', port=8080): Max retries exceeded with url: /manage/health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf2466050>: Failed to establish a new connection: [Errno 111] Connection refused',)) ------------------------------------------------------------------------------ Basic UseCaseUI API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC emsdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC gvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC huaweivnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC jujuvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC multivimproxy API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nokiav2driver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nslcm API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC resmgr API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnflcm API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC vnfmgr API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC vnfres API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC workflow API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztesdncdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztevnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VID Health Check | FAIL | 500 != 200 ------------------------------------------------------------------------------ Basic VNFSDK Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-fcaps API Health Check | PASS | ------------------------------------------------------------------------------ Basic Modeling genericparser API Health Check | FAIL | 502 != 200 ------------------------------------------------------------------------------ Basic CDS Health Check | PASS | ------------------------------------------------------------------------------ Testsuites.Health-Check :: Test that ONAP components are available... | FAIL | 47 critical tests, 29 passed, 18 failed 47 tests total, 29 passed, 18 failed ============================================================================== Testsuites | FAIL | 47 critical tests, 29 passed, 18 failed 47 tests total, 29 passed, 18 failed ============================================================================== Output: /share/logs/0034_ete_health/output.xml Log: /share/logs/0034_ete_health/log.html Report: /share/logs/0034_ete_health/report.html command terminated with exit code 18
ShowSUMMARY We have deployed ONAP on RKE cluster based on OOM Quick Start Guide as per below documentation guide, where we have configured RKE cluster on AWS vm instances (3 control plane + 12 worker nodes) https://docs.onap.org/projects/onap-oom/en/latest/oom_quickstart_guide.html#oom-quickstart-guide When we are bringing down/up the control plan & worker node AWS instances after they come up, we noticed that the below applications health checks are getting failed (Before bringing down the VMs, they are all fine) and some of the pods went into "Init:CrashLoopBackOff" state with "Back-off restarting failed container" as per kubectl describe. Looks like it could be an issue while bringing down the nodes & after comes back up, so filing this bug under OOM project OS / ENVIRONMENT Kubernetes version: Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Helm version: Client: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"} OOM VERSION seen on Master (G) CONFIGURATION ~/oom/kubernetes/onap/resources/overrides/onap-all.yaml # Copyright © 2019 Amdocs, Bell Canada # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ################################################################### # This override file enables helm charts for all ONAP applications. ################################################################### global: addTestingComponents: &testing true centralizedLoggingEnabled: ¢ralizedLogging false cassandra: enabled: true mariadb-galera: enabled: true aaf: enabled: true aai: enabled: true appc: enabled: true cds: enabled: true clamp: enabled: true cli: enabled: true # Today, "contrib" chart that hosting these components must also be enabled # in order to make it work. So `contrib.enabled` must have the same value than # addTestingComponents contrib: enabled: *testing consul: enabled: true dcaegen2: enabled: true dcaemod: enabled: true dmaap: enabled: true esr: enabled: true oof: enabled: true msb: enabled: true multicloud: enabled: true nbi: enabled: true policy: enabled: true portal: enabled: true robot: enabled: true sdc: enabled: true sdnc: enabled: true so: enabled: true uui: enabled: true vfc: enabled: true vid: enabled: true vnfsdk: enabled: true modeling: enabled: true STEPS TO REPRODUCE make oom with master build deploy ONAP on RKE cluster (3 control plane & 12 worker AWS instance )as below: nohup helm deploy dev local/onap --namespace onap --set global.masterPassword=onap -f onap/resources/overrides/onap-all.yaml -f onap/resources/overrides/environment.yaml -f onap/resources/overrides/openstack.yaml --timeout 900 > /home/ubuntu/full-onap-helm-deploy_new.log & After successful deployment, bring donw/up control plane and worker instances Check for ./ete-k8s.sh onap health and kubctl get pods -n onap EXPECTED RESULTS All pods should be up & running and robot health tests should PASS without any failures ACTUAL RESULTS Currently we noticed there below application health check failures & pods failures ./ete-k8s.sh onap health ++ export NAMESPACE=onap ++ NAMESPACE=onap +++ kubectl --namespace onap get pods +++ sed 's/ .*//' +++ grep robot ++ POD=dev-robot-578b965f4d-w7p9t ++ TAGS='-i health' +++ dirname ./ete-k8s.sh ++ DIR=. ++ SCRIPTDIR=scripts/etescript ++ ETEHOME=/var/opt/ONAP ++ [[ health == \e\x\e\c\s\c\r\i\p\t ]] +++ kubectl --namespace onap exec dev-robot-578b965f4d-w7p9t -- bash -c 'ls -1q /share/logs/ | wc -l' ++ export GLOBAL_BUILD_NUMBER=34 ++ GLOBAL_BUILD_NUMBER=34 +++ printf %04d 34 ++ OUTPUT_FOLDER=0034_ete_health ++ DISPLAY_NUM=124 ++ VARIABLEFILES='-V /share/config/robot_properties.py' ++ VARIABLES='-v GLOBAL_BUILD_NUMBER:1035' ++ case $2 in ++ kubectl --namespace onap exec dev-robot-578b965f4d-w7p9t -- /var/opt/ONAP/runTags.sh -V /share/config/robot_properties.py -v GLOBAL_BUILD_NUMBER:1035 -d /share/logs/0034_ete_health -i health --display 124 Starting Xvfb on display :124 with res 1280x1024x24 Executing robot tests at log level TRACE ============================================================================== Testsuites ============================================================================== Testsuites.Health-Check :: Test that ONAP components are available via basi... ============================================================================== Basic A&AI Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF SMS Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic APPC Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic CLI Health Check | PASS | ------------------------------------------------------------------------------ Basic CLAMP Health Check | PASS | ------------------------------------------------------------------------------ Basic DCAE Health Check | FAIL | 500 != 200 ------------------------------------------------------------------------------ Basic DMAAP Data Router Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf25718d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf256b310>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf2decdd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv | FAIL | ConnectionError: HTTPSConnectionPool(host='dmaap-dr-node.onap', port=8443): Max retries exceeded with url: /internal/fetchProv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bf4c18650>: Failed to establish a new connection: [Errno 111] Connection refused',)) ------------------------------------------------------------------------------ Basic DMAAP Message Router Health Check | PASS | ------------------------------------------------------------------------------ Basic DMAAP Bus Controller Health Check With Basic Auth | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic External API NBI Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic Microservice Bus Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-pike API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-starlingx API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-titanium_cloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-vio API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-k8s API Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-Homing Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic OOF-SNIRO Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic OOF-CMSO Health Check | FAIL | 401 != 200 ------------------------------------------------------------------------------ Basic Policy Health Check | PASS | ------------------------------------------------------------------------------ Basic Policy New Healthcheck | PASS | ------------------------------------------------------------------------------ Basic Portal Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic SDC Health Check (DMaaP:UP) | PASS | ------------------------------------------------------------------------------ Basic SDNC Health Check | FAIL | Test timeout 20 seconds exceeded. ------------------------------------------------------------------------------ Basic SO Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf24cce50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf24bb790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf2466690>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health | FAIL | ConnectionError: HTTPConnectionPool(host='so.onap', port=8080): Max retries exceeded with url: /manage/health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bf2466050>: Failed to establish a new connection: [Errno 111] Connection refused',)) ------------------------------------------------------------------------------ Basic UseCaseUI API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC emsdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC gvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC huaweivnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC jujuvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC multivimproxy API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nokiav2driver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nslcm API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC resmgr API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnflcm API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC vnfmgr API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC vnfres API Health Check | FAIL | 504 != 200 ------------------------------------------------------------------------------ Basic VFC workflow API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztesdncdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztevnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VID Health Check | FAIL | 500 != 200 ------------------------------------------------------------------------------ Basic VNFSDK Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-fcaps API Health Check | PASS | ------------------------------------------------------------------------------ Basic Modeling genericparser API Health Check | FAIL | 502 != 200 ------------------------------------------------------------------------------ Basic CDS Health Check | PASS | ------------------------------------------------------------------------------ Testsuites.Health-Check :: Test that ONAP components are available... | FAIL | 47 critical tests, 29 passed, 18 failed 47 tests total, 29 passed, 18 failed ============================================================================== Testsuites | FAIL | 47 critical tests, 29 passed, 18 failed 47 tests total, 29 passed, 18 failed ============================================================================== Output: /share/logs/0034_ete_health/output.xml Log: /share/logs/0034_ete_health/log.html Report: /share/logs/0034_ete_health/report.html command terminated with exit code 18 - Kubernetes version: