Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-960

OOM Healthcheck lockdown - currently 32/39 : 20180421

    Details

      Description

      as of 20180418 our healthcheck peaked at 27/34 (20180421 32/39)
      There are several types of issues occurring in the 27 passes and the 7 failures that need to be fixed
      see
      https://jira.onap.org/browse/OOM-963

      Missing Healthchecks

      • log for now in OOM-914
        Failed Heathcheck for failed containers
      • some of the checks can't be run because the containers are down
      • 17 containers down for aaf, aai, drools, esr, policy, portal, sdc, sdnc, sms)
      • the pdp-1 container down is handled under OOM-667 - the other one is up
      07:14:54 onap          dev-aaf-84dbb784f-xg6s5                        0/1       CrashLoopBackOff   11         1h
      07:14:54 onap          dev-aai-64d5df65c8-5np9r                       0/1       Init:0/1           6          1h
      07:14:54 onap          dev-aai-champ-7484559d98-mj89k                 0/1       CrashLoopBackOff   4          1h
      07:14:54 onap          dev-aai-traversal-update-query-data-dw2wp      0/1       Init:0/1           6          1h
      07:14:54 onap          dev-drools-1                                   0/1       Init:0/1           0          17m
      07:14:54 onap          dev-esr-57bc5b6b6d-mgrtx                       1/2       CrashLoopBackOff   18         1h
      07:14:55 onap          dev-pdp-1                                      0/2       Pending            0          16m
      07:14:55 onap          dev-portal-cassandra-85cc65d5dc-f6vlw          0/1       CrashLoopBackOff   16         1h
      07:14:55 onap          dev-sdc-be-config-backend-5gh2t                0/1       Pending            0          1m
      07:14:55 onap          dev-sdc-kb-64884d65f4-t44l7                    0/1       CrashLoopBackOff   11         1h
      07:14:55 onap          dev-sdnc-0                                     0/2       Init:0/1           5          1h
      07:14:55 onap          dev-sdnc-db-0                                  0/2       Pending            0          1h
      07:14:55 onap          dev-sdnc-dgbuilder-c85bdcd-sb8f5               0/1       Init:0/1           5          1h
      07:14:55 onap          dev-sdnc-dmaap-listener-6f4c55fc7d-hnqtt       0/1       Init:0/1           6          1h
      07:14:55 onap          dev-sdnc-portal-9f969bfb-zpjgq                 0/1       Init:Error         5          1h
      07:14:55 onap          dev-sdnc-ueb-listener-846fc46c9c-s9p7k         0/1       Init:0/1           6          1h
      07:14:55 onap          dev-sms-857f6dbd87-rkl6h                       0/1       Running            20         1h
      07:14:55 onap          dev-smsdb-0                                    0/2       ImagePullBackOff   0          1h
      

      Healthcheck passes but it should fail because some containers are down in the pod
      AAI
      07:20:28 Basic A&AI Health Check | PASS |

      07:14:54 onap          dev-aai-64d5df65c8-5np9r                       0/1       Init:0/1           6          1h
      07:14:54 onap          dev-aai-champ-7484559d98-mj89k                 0/1       CrashLoopBackOff   4          1h
      07:14:54 onap          dev-aai-traversal-update-query-data-dw2wp      0/1       Init:0/1           6          1h 0
      

      Healthcheck fails for containers that are fully up

      results

      07:20:28 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp components are...
      07:20:28 ==============================================================================
      07:20:28 Basic A&AI Health Check                                               | PASS |
      07:20:28 ------------------------------------------------------------------------------
      07:20:28 Basic APPC Health Check                                               | PASS |
      07:20:28 ------------------------------------------------------------------------------
      07:20:28 Basic CLAMP Health Check                                              | PASS |
      07:20:28 ------------------------------------------------------------------------------
      07:20:28 Basic DCAE Health Check                                               [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc5ed650>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      07:20:28 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc66dd90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      07:20:29 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc702f50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /healthcheck
      07:20:29 | FAIL |
      07:20:29 ConnectionError: HTTPConnectionPool(host='dev-dcae-controller.onap', port=8000): Max retries exceeded with url: /healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94c01e6890>: Failed to establish a new connection: [Errno -2] Name or service not known',))
      07:20:29 ------------------------------------------------------------------------------
      07:20:29 Basic DMAAP Message Router Health Check                               | PASS |
      07:20:29 ------------------------------------------------------------------------------
      07:20:29 Basic Microservice Bus Health Check                                   | PASS |
      07:20:29 ------------------------------------------------------------------------------
      07:20:29 Basic Multicloud API Health Check                                     | FAIL |
      07:20:29 502 != 200
      07:20:29 ------------------------------------------------------------------------------
      07:20:29 Basic Multicloud-ocata API Health Check                               | PASS |
      07:20:29 ------------------------------------------------------------------------------
      07:20:29 Basic Multicloud-titanium_cloud API Health Check                      | PASS |
      07:20:29 ------------------------------------------------------------------------------
      07:20:29 Basic Multicloud-vio API Health Check                                 | PASS |
      07:20:29 ------------------------------------------------------------------------------
      07:20:29 Basic MUSIC Health Check                                              [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc660a10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version
      07:20:29 [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc6663d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version
      07:20:30 [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc666690>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /MUSIC/rest/version
      07:20:30 | FAIL |
      07:20:30 ConnectionError: HTTPConnectionPool(host='music.onap', port=8080): Max retries exceeded with url: /MUSIC/rest/version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f94bc6c2a10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
      07:20:30 ------------------------------------------------------------------------------
      07:20:30 Basic Policy Health Check                                             | PASS |
      07:20:30 ------------------------------------------------------------------------------
      07:21:30 Basic Portal Health Check                                             | FAIL |
      07:21:30 Test timeout 1 minute exceeded.
      07:21:30 ------------------------------------------------------------------------------
      07:21:30 Basic SDC Health Check                                                | FAIL |
      07:21:30 500 != 200
      07:21:30 ------------------------------------------------------------------------------
      07:22:30 Basic SDNC Health Check                                               | FAIL |
      07:22:30 Test timeout 1 minute exceeded.
      07:22:30 ------------------------------------------------------------------------------
      07:22:30 Basic SO Health Check                                                 | PASS |
      07:22:30 ------------------------------------------------------------------------------
      07:22:30 Basic UseCaseUI API Health Check                                      | FAIL |
      07:22:30 502 != 200
      07:22:30 ------------------------------------------------------------------------------
      07:22:31 Basic VFC catalog API Health Check                                    | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:31 Basic VFC emsdriver API Health Check                                  | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:31 Basic VFC gvnfmdriver API Health Check                                | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:31 Basic VFC jujuvnfmdriver API Health Check                             | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:31 Basic VFC multivimproxy API Health Check                              | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:31 Basic VFC huaweivnfmdriver API Health Check                           | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:31 Basic VFC nokiavnfmdriver API Health Check                            | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:31 Basic VFC nokiav2driver API Health Check                              | PASS |
      07:22:31 ------------------------------------------------------------------------------
      07:22:33 Basic VFC nslcm API Health Check                                      | PASS |
      07:22:33 ------------------------------------------------------------------------------
      07:22:33 Basic VFC resmgr API Health Check                                     | PASS |
      07:22:33 ------------------------------------------------------------------------------
      07:22:33 Basic VFC vnflcm API Health Check                                     | PASS |
      07:22:33 ------------------------------------------------------------------------------
      07:22:34 Basic VFC vnfmgr API Health Check                                     | PASS |
      07:22:34 ------------------------------------------------------------------------------
      07:22:34 Basic VFC vnfres API Health Check                                     | PASS |
      07:22:34 ------------------------------------------------------------------------------
      07:22:34 Basic VFC workflow API Health Check                                   | PASS |
      07:22:34 ------------------------------------------------------------------------------
      07:22:34 Basic VFC ztesdncdriver API Health Check                              | PASS |
      07:22:34 ------------------------------------------------------------------------------
      07:22:35 Basic VFC ztevnfmdriver API Health Check                              | PASS |
      07:22:35 ------------------------------------------------------------------------------
      07:22:35 Basic VID Health Check                                                | PASS |
      07:22:35 ------------------------------------------------------------------------------
      07:22:35 OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp compo... | FAIL |
      07:22:35 34 critical tests, 27 passed, 7 failed
      07:22:35 34 tests total, 27 passed, 7 failed
      07:22:35 ==============================================================================
      07:22:35 OpenECOMP ETE.Robot.Testsuites                                        | FAIL |
      07:22:35 34 critical tests, 27 passed, 7 failed
      07:22:35 34 tests total, 27 passed, 7 failed
      07:22:35 ==============================================================================
      07:22:35 OpenECOMP ETE.Robot                                                   | FAIL |
      07:22:35 34 critical tests, 27 passed, 7 failed
      07:22:35 34 tests total, 27 passed, 7 failed
      07:22:35 ==============================================================================
      07:22:35 OpenECOMP ETE                                                         | FAIL |
      07:22:35 34 critical tests, 27 passed, 7 failed
      07:22:35 34 tests total, 27 passed, 7 failed
      07:22:35 ==============================================================================
      07:22:35 Output:  /share/logs/ETE_101134/output.xml
      07:22:35 Log:     /share/logs/ETE_101134/log.html
      07:22:35 Report:  /share/logs/ETE_101134/report.html
      07:22:35 /var/opt/OpenECOMP_ETE/runTags.sh: line 102:    46 Killed                  Xvfb ${DISPLAY} -ac -screen 0 ${RES} +extension RANDR  (wd: /)
      07:22:35 command terminated with exit code 7
      

      https://wiki.onap.org/display/DW/Healthcheck

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              • Assignee:
                rogerm Roger Maitland
                Reporter:
                michaelobrien Michael O'Brien
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: