-
Story
-
Resolution: Done
-
Medium
-
None
-
None
as of 20180418 our healthcheck peaked at 27/34 (20180421 32/39)
There are several types of issues occurring in the 27 passes and the 7 failures that need to be fixed
see
https://jira.onap.org/browse/OOM-963
Missing Healthchecks
- log for now in
OOM-914
Failed Heathcheck for failed containers - some of the checks can't be run because the containers are down
- 17 containers down for aaf, aai, drools, esr, policy, portal, sdc, sdnc, sms)
- the pdp-1 container down is handled under
OOM-667- the other one is up
07:14:54 onap dev-aaf-84dbb784f-xg6s5 0/1 CrashLoopBackOff 11 1h 07:14:54 onap dev-aai-64d5df65c8-5np9r 0/1 Init:0/1 6 1h 07:14:54 onap dev-aai-champ-7484559d98-mj89k 0/1 CrashLoopBackOff 4 1h 07:14:54 onap dev-aai-traversal-update-query-data-dw2wp 0/1 Init:0/1 6 1h 07:14:54 onap dev-drools-1 0/1 Init:0/1 0 17m 07:14:54 onap dev-esr-57bc5b6b6d-mgrtx 1/2 CrashLoopBackOff 18 1h 07:14:55 onap dev-pdp-1 0/2 Pending 0 16m 07:14:55 onap dev-portal-cassandra-85cc65d5dc-f6vlw 0/1 CrashLoopBackOff 16 1h 07:14:55 onap dev-sdc-be-config-backend-5gh2t 0/1 Pending 0 1m 07:14:55 onap dev-sdc-kb-64884d65f4-t44l7 0/1 CrashLoopBackOff 11 1h 07:14:55 onap dev-sdnc-0 0/2 Init:0/1 5 1h 07:14:55 onap dev-sdnc-db-0 0/2 Pending 0 1h 07:14:55 onap dev-sdnc-dgbuilder-c85bdcd-sb8f5 0/1 Init:0/1 5 1h 07:14:55 onap dev-sdnc-dmaap-listener-6f4c55fc7d-hnqtt 0/1 Init:0/1 6 1h 07:14:55 onap dev-sdnc-portal-9f969bfb-zpjgq 0/1 Init:Error 5 1h 07:14:55 onap dev-sdnc-ueb-listener-846fc46c9c-s9p7k 0/1 Init:0/1 6 1h 07:14:55 onap dev-sms-857f6dbd87-rkl6h 0/1 Running 20 1h 07:14:55 onap dev-smsdb-0 0/2 ImagePullBackOff 0 1h
Healthcheck passes but it should fail because some containers are down in the pod
AAI
07:20:28 Basic A&AI Health Check | PASS |
07:14:54 onap dev-aai-64d5df65c8-5np9r 0/1 Init:0/1 6 1h 07:14:54 onap dev-aai-champ-7484559d98-mj89k 0/1 CrashLoopBackOff 4 1h 07:14:54 onap dev-aai-traversal-update-query-data-dw2wp 0/1 Init:0/1 6 1h 0
Healthcheck fails for containers that are fully up
results
07:20:28 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp components are... 07:20:28 ============================================================================== 07:20:28 Basic A&AI Health Check | PASS | 07:20:28 ------------------------------------------------------------------------------ 07:20:28 Basic APPC Health Check | PASS | 07:20:28 ------------------------------------------------------------------------------ 07:20:28 Basic CLAMP Health Check | PASS | 07:20:28 ------------------------------------------------------------------------------ 07:20:28 Basic DCAE Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc5ed650>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 07:20:28 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc66dd90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 07:20:29 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc702f50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 07:20:29 | FAIL | 07:20:29 ConnectionError: HTTPConnectionPool(host='dev-dcae-controller.onap', port=8000): Max retries exceeded with url: /healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94c01e6890>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 07:20:29 ------------------------------------------------------------------------------ 07:20:29 Basic DMAAP Message Router Health Check | PASS | 07:20:29 ------------------------------------------------------------------------------ 07:20:29 Basic Microservice Bus Health Check | PASS | 07:20:29 ------------------------------------------------------------------------------ 07:20:29 Basic Multicloud API Health Check | FAIL | 07:20:29 502 != 200 07:20:29 ------------------------------------------------------------------------------ 07:20:29 Basic Multicloud-ocata API Health Check | PASS | 07:20:29 ------------------------------------------------------------------------------ 07:20:29 Basic Multicloud-titanium_cloud API Health Check | PASS | 07:20:29 ------------------------------------------------------------------------------ 07:20:29 Basic Multicloud-vio API Health Check | PASS | 07:20:29 ------------------------------------------------------------------------------ 07:20:29 Basic MUSIC Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc660a10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version 07:20:29 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc6663d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version 07:20:30 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc666690>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version 07:20:30 | FAIL | 07:20:30 ConnectionError: HTTPConnectionPool(host='music.onap', port=8080): Max retries exceeded with url: /MUSIC/rest/version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc6c2a10>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 07:20:30 ------------------------------------------------------------------------------ 07:20:30 Basic Policy Health Check | PASS | 07:20:30 ------------------------------------------------------------------------------ 07:21:30 Basic Portal Health Check | FAIL | 07:21:30 Test timeout 1 minute exceeded. 07:21:30 ------------------------------------------------------------------------------ 07:21:30 Basic SDC Health Check | FAIL | 07:21:30 500 != 200 07:21:30 ------------------------------------------------------------------------------ 07:22:30 Basic SDNC Health Check | FAIL | 07:22:30 Test timeout 1 minute exceeded. 07:22:30 ------------------------------------------------------------------------------ 07:22:30 Basic SO Health Check | PASS | 07:22:30 ------------------------------------------------------------------------------ 07:22:30 Basic UseCaseUI API Health Check | FAIL | 07:22:30 502 != 200 07:22:30 ------------------------------------------------------------------------------ 07:22:31 Basic VFC catalog API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:31 Basic VFC emsdriver API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:31 Basic VFC gvnfmdriver API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:31 Basic VFC jujuvnfmdriver API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:31 Basic VFC multivimproxy API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:31 Basic VFC huaweivnfmdriver API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:31 Basic VFC nokiavnfmdriver API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:31 Basic VFC nokiav2driver API Health Check | PASS | 07:22:31 ------------------------------------------------------------------------------ 07:22:33 Basic VFC nslcm API Health Check | PASS | 07:22:33 ------------------------------------------------------------------------------ 07:22:33 Basic VFC resmgr API Health Check | PASS | 07:22:33 ------------------------------------------------------------------------------ 07:22:33 Basic VFC vnflcm API Health Check | PASS | 07:22:33 ------------------------------------------------------------------------------ 07:22:34 Basic VFC vnfmgr API Health Check | PASS | 07:22:34 ------------------------------------------------------------------------------ 07:22:34 Basic VFC vnfres API Health Check | PASS | 07:22:34 ------------------------------------------------------------------------------ 07:22:34 Basic VFC workflow API Health Check | PASS | 07:22:34 ------------------------------------------------------------------------------ 07:22:34 Basic VFC ztesdncdriver API Health Check | PASS | 07:22:34 ------------------------------------------------------------------------------ 07:22:35 Basic VFC ztevnfmdriver API Health Check | PASS | 07:22:35 ------------------------------------------------------------------------------ 07:22:35 Basic VID Health Check | PASS | 07:22:35 ------------------------------------------------------------------------------ 07:22:35 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp compo... | FAIL | 07:22:35 34 critical tests, 27 passed, 7 failed 07:22:35 34 tests total, 27 passed, 7 failed 07:22:35 ============================================================================== 07:22:35 OpenECOMP ETE.Robot.Testsuites | FAIL | 07:22:35 34 critical tests, 27 passed, 7 failed 07:22:35 34 tests total, 27 passed, 7 failed 07:22:35 ============================================================================== 07:22:35 OpenECOMP ETE.Robot | FAIL | 07:22:35 34 critical tests, 27 passed, 7 failed 07:22:35 34 tests total, 27 passed, 7 failed 07:22:35 ============================================================================== 07:22:35 OpenECOMP ETE | FAIL | 07:22:35 34 critical tests, 27 passed, 7 failed 07:22:35 34 tests total, 27 passed, 7 failed 07:22:35 ============================================================================== 07:22:35 Output: /share/logs/ETE_101134/output.xml 07:22:35 Log: /share/logs/ETE_101134/log.html 07:22:35 Report: /share/logs/ETE_101134/report.html 07:22:35 /var/opt/OpenECOMP_ETE/runTags.sh: line 102: 46 Killed Xvfb ${DISPLAY} -ac -screen 0 ${RES} +extension RANDR (wd: /) 07:22:35 command terminated with exit code 7
- is blocked by
-
MUSIC-70 portal music-cassandra container fails in OOM
- Closed
-
MUSIC-71 MUSIC healthcheck failing 100% of the time
- Closed
-
APPC-856 appc-dgbuilder container image error with onap/ccsdk-dgbuilder-image:0.2.1-SNAPSHOT deleted from nexus3
- Closed
-
MUSIC-69 Music OOM healthcheck error 20180413
- Closed
-
OOM-914 Add LOG component robot healthcheck
- Closed
-
OOM-934 Consul health checks for SDNC
- Closed
-
SDNC-284 sdnc-dgbuilder container image error with onap/ccsdk-dgbuilder-image:0.2.1-SNAPSHOT deleted from nexus3
- Closed
-
SDNC-285 sdnc-dgbuilder container image error with onap/ccsdk-dgbuilder-image:0.2.1-SNAPSHOT
- Closed
-
USECASEUI-106 UUI Health Check fails in OOM
- Closed
-
APPC-857 ccsdk-dgbuilder container fails to come up
- Closed
-
MULTICLOUD-213 Multicloud Healthcheck failure - pods up
- Closed
-
OOM-964 SDC Healthcheck failure on sdc-be and sdc-kb containers down
- Closed
-
OOM-1002 SDC-ES no longer starting in OOM
- Closed
-
POLICY-753 Policy Health Check failed with multi-node cluster
- Closed
-
PORTAL-258 Portal Healthcheck failure timeout on cassandra container falure
- Closed
-
AAI-1085 AAI Healthcheck passes even though aai-champ is crashed
- Closed
- mentioned in
-
Page Loading...
1.
|
empty | Closed | david.sauvageau | |
2.
|
Adjust SDC-BE init job timing from 10 to 30s to avoid restarts on single node systems | Closed | michaelobrien |