-
Task
-
Resolution: Duplicate
-
Medium
-
None
-
None
the last 12 hours show regressions on SDC(90% failure) and SDNC (100% failure) - looking into it
see CD Healthcheck and top right - select 7 days
http://jenkins.onap.info/job/oom-cd/1414/console
14:35:10 sleep 4 min - to allow rest frameworks to finish 14:39:10 run healthcheck 3 times to warm caches and frameworks so rest endpoints report properly - see OOM-447 14:39:10 run healthcheck prep 1 14:39:15 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9d086210>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:39:16 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9b2b9bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:39:16 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9b2c1450>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:39:16 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9ac15bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:16 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9ac15b90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:17 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9d086550>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:24 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9abf5710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:24 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9abf5910>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:25 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9abf5b10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:34 command terminated with exit code 6 14:39:34 run healthcheck prep 2 14:39:37 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9f15b1c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:39:37 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef81dbd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:39:38 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef88d090>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:39:38 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef177d50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:38 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef177c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:39 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9f15e4710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:39 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef158750>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:39 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef158950>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:40 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef158b50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:39:41 command terminated with exit code 6 14:39:41 run healthcheck for real - wait a further 6 min 14:45:42 Starting Xvfb on display :88 with res 1280x1024x24 14:45:42 Executing robot tests at log level TRACE 14:45:44 ============================================================================== 14:45:44 OpenECOMP ETE 14:45:44 ============================================================================== 14:45:44 OpenECOMP ETE.Robot 14:45:44 ============================================================================== 14:45:44 OpenECOMP ETE.Robot.Testsuites 14:45:44 ============================================================================== 14:45:45 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp components are... 14:45:45 ============================================================================== 14:45:45 Basic DCAE Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f6bb16c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:45:45 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69d82bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:45:45 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69df2090>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck 14:45:45 | FAIL | 14:45:45 ConnectionError: HTTPConnectionPool(host='dcae-controller.onap-dcae', port=8080): Max retries exceeded with url: /healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69df2b90>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 14:45:45 ------------------------------------------------------------------------------ 14:45:45 Basic SDNGC Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696dcf90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:45:45 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696dce50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:45:46 | FAIL | 14:45:46 ConnectionError: HTTPConnectionPool(host='sdnhost.onap-sdnc', port=8282): Max retries exceeded with url: /restconf/operations/SLI-API:healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696dc650>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 14:45:46 ------------------------------------------------------------------------------ 14:45:46 Basic A&AI Health Check [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69d82910>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:45:46 | PASS | 14:45:46 ------------------------------------------------------------------------------ 14:45:46 Basic Policy Health Check | PASS | 14:45:46 ------------------------------------------------------------------------------ 14:45:46 Basic MSO Health Check | PASS | 14:45:46 ------------------------------------------------------------------------------ 14:45:46 Basic ASDC Health Check | FAIL | 14:45:46 500 != 200 14:45:46 ------------------------------------------------------------------------------ 14:45:46 Basic APPC Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696bf710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:45:46 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696bf910>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:45:47 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696bfb10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck 14:45:47 | FAIL | 14:45:47 ConnectionError: HTTPConnectionPool(host='sdnhost.onap-appc', port=8282): Max retries exceeded with url: /restconf/operations/SLI-API:healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696a4690>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 14:45:47 ------------------------------------------------------------------------------ 14:45:47 Basic Portal Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 Basic Message Router Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 Basic VID Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 Basic Microservice Bus Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 Basic CLAMP Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 catalog API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 emsdriver API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 gvnfmdriver API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 huaweivnfmdriver API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 multicloud API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 multicloud-ocata API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 multicloud-titanium_cloud API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 multicloud-vio API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 nokiavnfmdriver API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 nslcm API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 resmgr API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 usecaseui-gui API Health Check | FAIL | 14:45:47 502 != 200 14:45:47 ------------------------------------------------------------------------------ 14:45:47 vnflcm API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:47 vnfmgr API Health Check | PASS | 14:45:47 ------------------------------------------------------------------------------ 14:45:48 vnfres API Health Check | PASS | 14:45:48 ------------------------------------------------------------------------------ 14:45:48 workflow API Health Check | PASS | 14:45:48 ------------------------------------------------------------------------------ 14:45:48 ztesdncdriver API Health Check | PASS | 14:45:48 ------------------------------------------------------------------------------ 14:45:48 ztevmanagerdriver API Health Check | FAIL | 14:45:48 502 != 200 14:45:48 ------------------------------------------------------------------------------ 14:45:48 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp compo... | FAIL | 14:45:48 30 critical tests, 24 passed, 6 failed 14:45:48 30 tests total, 24 passed, 6 failed 14:45:48 ============================================================================== 14:45:48 OpenECOMP ETE.Robot.Testsuites | FAIL | 14:45:48 30 critical tests, 24 passed, 6 failed 14:45:48 30 tests total, 24 passed, 6 failed 14:45:48 ============================================================================== 14:45:48 OpenECOMP ETE.Robot | FAIL | 14:45:48 30 critical tests, 24 passed, 6 failed 14:45:48 30 tests total, 24 passed, 6 failed 14:45:48 ============================================================================== 14:45:48 OpenECOMP ETE | FAIL | 14:45:48 30 critical tests, 24 passed, 6 failed 14:45:48 30 tests total, 24 passed, 6 failed
- is blocked by
-
OOM-486 HELM upgrade from 2.3 to 2.8.0
- Closed