-
Story
-
Resolution: Done
-
Medium
-
None
-
None
as of 20180418 our healthcheck peaked at 27/34 (20180421 32/39)
There are several types of issues occurring in the 27 passes and the 7 failures that need to be fixed
see
https://jira.onap.org/browse/OOM-963
Missing Healthchecks
- log for now in
OOM-914
Failed Heathcheck for failed containers - some of the checks can't be run because the containers are down
- 17 containers down for aaf, aai, drools, esr, policy, portal, sdc, sdnc, sms)
- the pdp-1 container down is handled under
OOM-667- the other one is up
07:14:54 onap dev-aaf-84dbb784f-xg6s5 0/1 CrashLoopBackOff 11 1h 07:14:54 onap dev-aai-64d5df65c8-5np9r 0/1 Init:0/1 6 1h 07:14:54 onap dev-aai-champ-7484559d98-mj89k 0/1 CrashLoopBackOff 4 1h 07:14:54 onap dev-aai-traversal-update-query-data-dw2wp 0/1 Init:0/1 6 1h 07:14:54 onap dev-drools-1 0/1 Init:0/1 0 17m 07:14:54 onap dev-esr-57bc5b6b6d-mgrtx 1/2 CrashLoopBackOff 18 1h 07:14:55 onap dev-pdp-1 0/2 Pending 0 16m 07:14:55 onap dev-portal-cassandra-85cc65d5dc-f6vlw 0/1 CrashLoopBackOff 16 1h 07:14:55 onap dev-sdc-be-config-backend-5gh2t 0/1 Pending 0 1m 07:14:55 onap dev-sdc-kb-64884d65f4-t44l7 0/1 CrashLoopBackOff 11 1h 07:14:55 onap dev-sdnc-0 0/2 Init:0/1 5 1h 07:14:55 onap dev-sdnc-db-0 0/2 Pending 0 1h 07:14:55 onap dev-sdnc-dgbuilder-c85bdcd-sb8f5 0/1 Init:0/1 5 1h 07:14:55 onap dev-sdnc-dmaap-listener-6f4c55fc7d-hnqtt 0/1 Init:0/1 6 1h 07:14:55 onap dev-sdnc-portal-9f969bfb-zpjgq 0/1 Init:Error 5 1h 07:14:55 onap dev-sdnc-ueb-listener-846fc46c9c-s9p7k 0/1 Init:0/1 6 1h 07:14:55 onap dev-sms-857f6dbd87-rkl6h 0/1 Running 20 1h 07:14:55 onap dev-smsdb-0 0/2 ImagePullBackOff 0 1h
Healthcheck passes but it should fail because some containers are down in the pod
AAI
07:20:28 Basic A&AI Health Check | PASS |
07:14:54 onap dev-aai-64d5df65c8-5np9r 0/1 Init:0/1 6 1h 07:14:54 onap dev-aai-champ-7484559d98-mj89k 0/1 CrashLoopBackOff 4 1h 07:14:54 onap dev-aai-traversal-update-query-data-dw2wp 0/1 Init:0/1 6 1h 0
Healthcheck fails for containers that are fully up
results
07:20:28 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp components are...
07:20:28 ==============================================================================
07:20:28 Basic A&AI Health Check | PASS |
07:20:28 ------------------------------------------------------------------------------
07:20:28 Basic APPC Health Check | PASS |
07:20:28 ------------------------------------------------------------------------------
07:20:28 Basic CLAMP Health Check | PASS |
07:20:28 ------------------------------------------------------------------------------
07:20:28 Basic DCAE Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc5ed650>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
07:20:28 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc66dd90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
07:20:29 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc702f50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
07:20:29 | FAIL |
07:20:29 ConnectionError: HTTPConnectionPool(host='dev-dcae-controller.onap', port=8000): Max retries exceeded with url: /healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94c01e6890>: Failed to establish a new connection: [Errno -2] Name or service not known',))
07:20:29 ------------------------------------------------------------------------------
07:20:29 Basic DMAAP Message Router Health Check | PASS |
07:20:29 ------------------------------------------------------------------------------
07:20:29 Basic Microservice Bus Health Check | PASS |
07:20:29 ------------------------------------------------------------------------------
07:20:29 Basic Multicloud API Health Check | FAIL |
07:20:29 502 != 200
07:20:29 ------------------------------------------------------------------------------
07:20:29 Basic Multicloud-ocata API Health Check | PASS |
07:20:29 ------------------------------------------------------------------------------
07:20:29 Basic Multicloud-titanium_cloud API Health Check | PASS |
07:20:29 ------------------------------------------------------------------------------
07:20:29 Basic Multicloud-vio API Health Check | PASS |
07:20:29 ------------------------------------------------------------------------------
07:20:29 Basic MUSIC Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc660a10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version
07:20:29 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc6663d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version
07:20:30 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc666690>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version
07:20:30 | FAIL |
07:20:30 ConnectionError: HTTPConnectionPool(host='music.onap', port=8080): Max retries exceeded with url: /MUSIC/rest/version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc6c2a10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
07:20:30 ------------------------------------------------------------------------------
07:20:30 Basic Policy Health Check | PASS |
07:20:30 ------------------------------------------------------------------------------
07:21:30 Basic Portal Health Check | FAIL |
07:21:30 Test timeout 1 minute exceeded.
07:21:30 ------------------------------------------------------------------------------
07:21:30 Basic SDC Health Check | FAIL |
07:21:30 500 != 200
07:21:30 ------------------------------------------------------------------------------
07:22:30 Basic SDNC Health Check | FAIL |
07:22:30 Test timeout 1 minute exceeded.
07:22:30 ------------------------------------------------------------------------------
07:22:30 Basic SO Health Check | PASS |
07:22:30 ------------------------------------------------------------------------------
07:22:30 Basic UseCaseUI API Health Check | FAIL |
07:22:30 502 != 200
07:22:30 ------------------------------------------------------------------------------
07:22:31 Basic VFC catalog API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:31 Basic VFC emsdriver API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:31 Basic VFC gvnfmdriver API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:31 Basic VFC jujuvnfmdriver API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:31 Basic VFC multivimproxy API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:31 Basic VFC huaweivnfmdriver API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:31 Basic VFC nokiavnfmdriver API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:31 Basic VFC nokiav2driver API Health Check | PASS |
07:22:31 ------------------------------------------------------------------------------
07:22:33 Basic VFC nslcm API Health Check | PASS |
07:22:33 ------------------------------------------------------------------------------
07:22:33 Basic VFC resmgr API Health Check | PASS |
07:22:33 ------------------------------------------------------------------------------
07:22:33 Basic VFC vnflcm API Health Check | PASS |
07:22:33 ------------------------------------------------------------------------------
07:22:34 Basic VFC vnfmgr API Health Check | PASS |
07:22:34 ------------------------------------------------------------------------------
07:22:34 Basic VFC vnfres API Health Check | PASS |
07:22:34 ------------------------------------------------------------------------------
07:22:34 Basic VFC workflow API Health Check | PASS |
07:22:34 ------------------------------------------------------------------------------
07:22:34 Basic VFC ztesdncdriver API Health Check | PASS |
07:22:34 ------------------------------------------------------------------------------
07:22:35 Basic VFC ztevnfmdriver API Health Check | PASS |
07:22:35 ------------------------------------------------------------------------------
07:22:35 Basic VID Health Check | PASS |
07:22:35 ------------------------------------------------------------------------------
07:22:35 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp compo... | FAIL |
07:22:35 34 critical tests, 27 passed, 7 failed
07:22:35 34 tests total, 27 passed, 7 failed
07:22:35 ==============================================================================
07:22:35 OpenECOMP ETE.Robot.Testsuites | FAIL |
07:22:35 34 critical tests, 27 passed, 7 failed
07:22:35 34 tests total, 27 passed, 7 failed
07:22:35 ==============================================================================
07:22:35 OpenECOMP ETE.Robot | FAIL |
07:22:35 34 critical tests, 27 passed, 7 failed
07:22:35 34 tests total, 27 passed, 7 failed
07:22:35 ==============================================================================
07:22:35 OpenECOMP ETE | FAIL |
07:22:35 34 critical tests, 27 passed, 7 failed
07:22:35 34 tests total, 27 passed, 7 failed
07:22:35 ==============================================================================
07:22:35 Output: /share/logs/ETE_101134/output.xml
07:22:35 Log: /share/logs/ETE_101134/log.html
07:22:35 Report: /share/logs/ETE_101134/report.html
07:22:35 /var/opt/OpenECOMP_ETE/runTags.sh: line 102: 46 Killed Xvfb ${DISPLAY} -ac -screen 0 ${RES} +extension RANDR (wd: /)
07:22:35 command terminated with exit code 7
- is blocked by
-
MUSIC-70 portal music-cassandra container fails in OOM
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
OOM-914
Add LOG component robot healthcheck
-
- Closed
-
-
OOM-934
Consul health checks for SDNC
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- mentioned in
-
Page Loading...
(11 is blocked by, 1 mentioned in)
There are no Sub-Tasks for this issue.