Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-614

SDC, SDNC, AAI Healthcheck failures last 12 hours 20180124:1100EST

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Duplicate
    • Icon: Medium Medium
    • Beijing Release
    • None
    • None

      the last 12 hours show regressions on SDC(90% failure) and SDNC (100% failure) - looking into it

      see CD Healthcheck and top right - select 7 days

      http://kibana.onap.info:5601/app/kibana#/dashboards?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-12h,mode:quick,to:now))

       

      http://jenkins.onap.info/job/oom-cd/1414/console 

      14:35:10 sleep 4 min - to allow rest frameworks to finish
      14:39:10 run healthcheck 3 times to warm caches and frameworks so rest endpoints report properly - see OOM-447
      14:39:10 run healthcheck prep 1
      14:39:15 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9d086210>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:39:16 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9b2b9bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:39:16 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9b2c1450>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:39:16 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9ac15bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:16 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9ac15b90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:17 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9d086550>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:24 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9abf5710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:24 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9abf5910>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:25 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a9abf5b10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:34 command terminated with exit code 6
      14:39:34 run healthcheck prep 2
      14:39:37 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9f15b1c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:39:37 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef81dbd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:39:38 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef88d090>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:39:38 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef177d50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:38 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef177c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:39 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9f15e4710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:39 [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef158750>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:39 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef158950>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:40 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9ef158b50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:39:41 command terminated with exit code 6
      14:39:41 run healthcheck for real - wait a further 6 min
      14:45:42 Starting Xvfb on display :88 with res 1280x1024x24
      14:45:42 Executing robot tests at log level TRACE
      14:45:44 ==============================================================================
      14:45:44 OpenECOMP ETE                                                                 
      14:45:44 ==============================================================================
      14:45:44 OpenECOMP ETE.Robot                                                           
      14:45:44 ==============================================================================
      14:45:44 OpenECOMP ETE.Robot.Testsuites                                                
      14:45:44 ==============================================================================
      14:45:45 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp components are...
      14:45:45 ==============================================================================
      14:45:45 Basic DCAE Health Check                                               [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f6bb16c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:45:45 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69d82bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:45:45 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69df2090>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      14:45:45 | FAIL |
      14:45:45 ConnectionError: HTTPConnectionPool(host='dcae-controller.onap-dcae', port=8080): Max retries exceeded with url: /healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69df2b90>: Failed to establish a new connection: [Errno -2] Name or service not known',))
      14:45:45 ------------------------------------------------------------------------------
      14:45:45 Basic SDNGC Health Check                                              [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696dcf90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:45:45 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696dce50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:45:46 | FAIL |
      14:45:46 ConnectionError: HTTPConnectionPool(host='sdnhost.onap-sdnc', port=8282): Max retries exceeded with url: /restconf/operations/SLI-API:healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696dc650>: Failed to establish a new connection: [Errno -2] Name or service not known',))
      14:45:46 ------------------------------------------------------------------------------
      14:45:46 Basic A&AI Health Check                                               [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f69d82910>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:45:46 | PASS |
      14:45:46 ------------------------------------------------------------------------------
      14:45:46 Basic Policy Health Check                                             | PASS |
      14:45:46 ------------------------------------------------------------------------------
      14:45:46 Basic MSO Health Check                                                | PASS |
      14:45:46 ------------------------------------------------------------------------------
      14:45:46 Basic ASDC Health Check                                               | FAIL |
      14:45:46 500 != 200
      14:45:46 ------------------------------------------------------------------------------
      14:45:46 Basic APPC Health Check                                               [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696bf710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:45:46 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696bf910>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:45:47 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696bfb10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /restconf/operations/SLI-API:healthcheck
      14:45:47 | FAIL |
      14:45:47 ConnectionError: HTTPConnectionPool(host='sdnhost.onap-appc', port=8282): Max retries exceeded with url: /restconf/operations/SLI-API:healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0f696a4690>: Failed to establish a new connection: [Errno -2] Name or service not known',))
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 Basic Portal Health Check                                             | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 Basic Message Router Health Check                                     | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 Basic VID Health Check                                                | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 Basic Microservice Bus Health Check                                   | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 Basic CLAMP Health Check                                              | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 catalog API Health Check                                              | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 emsdriver API Health Check                                            | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 gvnfmdriver API Health Check                                          | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 huaweivnfmdriver API Health Check                                     | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 multicloud API Health Check                                           | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 multicloud-ocata API Health Check                                     | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 multicloud-titanium_cloud API Health Check                            | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 multicloud-vio API Health Check                                       | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 nokiavnfmdriver API Health Check                                      | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 nslcm API Health Check                                                | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 resmgr API Health Check                                               | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 usecaseui-gui API Health Check                                        | FAIL |
      14:45:47 502 != 200
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 vnflcm API Health Check                                               | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:47 vnfmgr API Health Check                                               | PASS |
      14:45:47 ------------------------------------------------------------------------------
      14:45:48 vnfres API Health Check                                               | PASS |
      14:45:48 ------------------------------------------------------------------------------
      14:45:48 workflow API Health Check                                             | PASS |
      14:45:48 ------------------------------------------------------------------------------
      14:45:48 ztesdncdriver API Health Check                                        | PASS |
      14:45:48 ------------------------------------------------------------------------------
      14:45:48 ztevmanagerdriver API Health Check                                    | FAIL |
      14:45:48 502 != 200
      14:45:48 ------------------------------------------------------------------------------
      14:45:48 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp compo... | FAIL |
      14:45:48 30 critical tests, 24 passed, 6 failed
      14:45:48 30 tests total, 24 passed, 6 failed
      14:45:48 ==============================================================================
      14:45:48 OpenECOMP ETE.Robot.Testsuites                                        | FAIL |
      14:45:48 30 critical tests, 24 passed, 6 failed
      14:45:48 30 tests total, 24 passed, 6 failed
      14:45:48 ==============================================================================
      14:45:48 OpenECOMP ETE.Robot                                                   | FAIL |
      14:45:48 30 critical tests, 24 passed, 6 failed
      14:45:48 30 tests total, 24 passed, 6 failed
      14:45:48 ==============================================================================
      14:45:48 OpenECOMP ETE                                                         | FAIL |
      14:45:48 30 critical tests, 24 passed, 6 failed
      14:45:48 30 tests total, 24 passed, 6 failed

            michaelobrien michaelobrien
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: