Uploaded image for project: 'Network Controller'
  1. Network Controller
  2. SDNC-482

SDNC health check fails intermittently in clustering mode

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: High High
    • Dublin Release
    • Casablanca Release
    • sdnc-northbound
    • OOM with staging images installed on 10/16/2018 at Windriver lab. 

       

      SDNC is deployed with replica of 3 and clustering as true.

    • SDNC Dublin Spr 2 2/18-3/8

      SDNC fails health check with 404 error after ONAP installation. Robot log is following: 

       

      16:52:34.815 DEBUG http://sdnc.onap:8282 "POST /restconf/operations/SLI-API:healthcheck HTTP/1.1" 404 276  
      16:52:34.816 DEBUG post response: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /restconf/operations/SLI-API:healthcheck. Reason: <pre> Not Found</pre></p> <hr /> </body> </html>

       

      Also, the RESTCONF API page  (http://ip:30202/apidoc/explorer/index.html ) fails on browser with 404. 

      Not clear if the error will impact SDNC function at this point of time. If found not, will lower the priority. 

       

      Reran 5 times of health check, actually now SDNC failed 2 times and succeeded other 3 times. See the output of the commands:

      root@oom-rancher:~# oom/kubernetes/robot/ete-k8s.sh onap health |grep SDNC
      ++ export NAMESPACE=onap
      ++ NAMESPACE=onap
      +++ kubectl --namespace onap get pods
      +++ sed 's/ .*//'
      +++ grep robot
      ++ POD=dev-robot-robot-5f7c96cb9c-4zcfv
      ++ export 'TAGS=-i health'
      ++ TAGS='-i health'
      ++ export ETEHOME=/var/opt/OpenECOMP_ETE
      ++ ETEHOME=/var/opt/OpenECOMP_ETE
      +++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- bash -c 'ls -1q /share/logs/ | wc -l'
      ++ export GLOBAL_BUILD_NUMBER=21
      ++ GLOBAL_BUILD_NUMBER=21
      +++ printf %04d 21
      ++ export OUTPUT_FOLDER=ETE_0021_health
      ++ OUTPUT_FOLDER=ETE_0021_health
      ++ VARIABLEFILES='-V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py'
      ++ VARIABLES='-v GLOBAL_BUILD_NUMBER:22005'
      ++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- /var/opt/OpenECOMP_ETE/runTags.sh -V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py -v GLOBAL_BUILD_NUMBER:22005 -d /share/logs/ETE_0021_health -i health --display 88
      Basic SDNC Health Check                                               | PASS |
      command terminated with exit code 2
      root@oom-rancher:~# oom/kubernetes/robot/ete-k8s.sh onap health |grep SDNC
      ++ export NAMESPACE=onap
      ++ NAMESPACE=onap
      +++ kubectl --namespace onap get pods
      +++ sed 's/ .*//'
      +++ grep robot
      ++ POD=dev-robot-robot-5f7c96cb9c-4zcfv
      ++ export 'TAGS=-i health'
      ++ TAGS='-i health'
      ++ export ETEHOME=/var/opt/OpenECOMP_ETE
      ++ ETEHOME=/var/opt/OpenECOMP_ETE
      +++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- bash -c 'ls -1q /share/logs/ | wc -l'
      ++ export GLOBAL_BUILD_NUMBER=22
      ++ GLOBAL_BUILD_NUMBER=22
      +++ printf %04d 22
      ++ export OUTPUT_FOLDER=ETE_0022_health
      ++ OUTPUT_FOLDER=ETE_0022_health
      ++ VARIABLEFILES='-V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py'
      ++ VARIABLES='-v GLOBAL_BUILD_NUMBER:22053'
      ++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- /var/opt/OpenECOMP_ETE/runTags.sh -V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py -v GLOBAL_BUILD_NUMBER:22053 -d /share/logs/ETE_0022_health -i health --display 88
      Basic SDNC Health Check                                               | FAIL |
      command terminated with exit code 4
      root@oom-rancher:~# 
      root@oom-rancher:~# oom/kubernetes/robot/ete-k8s.sh onap health |grep SDNC
      ++ export NAMESPACE=onap
      ++ NAMESPACE=onap
      +++ kubectl --namespace onap get pods
      +++ grep robot
      +++ sed 's/ .*//'
      ++ POD=dev-robot-robot-5f7c96cb9c-4zcfv
      ++ export 'TAGS=-i health'
      ++ TAGS='-i health'
      ++ export ETEHOME=/var/opt/OpenECOMP_ETE
      ++ ETEHOME=/var/opt/OpenECOMP_ETE
      +++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- bash -c 'ls -1q /share/logs/ | wc -l'
      ++ export GLOBAL_BUILD_NUMBER=23
      ++ GLOBAL_BUILD_NUMBER=23
      +++ printf %04d 23
      ++ export OUTPUT_FOLDER=ETE_0023_health
      ++ OUTPUT_FOLDER=ETE_0023_health
      ++ VARIABLEFILES='-V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py'
      ++ VARIABLES='-v GLOBAL_BUILD_NUMBER:22101'
      ++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- /var/opt/OpenECOMP_ETE/runTags.sh -V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py -v GLOBAL_BUILD_NUMBER:22101 -d /share/logs/ETE_0023_health -i health --display 88
      Basic SDNC Health Check                                               | PASS |
      command terminated with exit code 2
      root@oom-rancher:~# oom/kubernetes/robot/ete-k8s.sh onap health |grep SDNC
      ++ export NAMESPACE=onap
      ++ NAMESPACE=onap
      +++ kubectl --namespace onap get pods
      +++ sed 's/ .*//'
      +++ grep robot
      ++ POD=dev-robot-robot-5f7c96cb9c-4zcfv
      ++ export 'TAGS=-i health'
      ++ TAGS='-i health'
      ++ export ETEHOME=/var/opt/OpenECOMP_ETE
      ++ ETEHOME=/var/opt/OpenECOMP_ETE
      +++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- bash -c 'ls -1q /share/logs/ | wc -l'
      ++ export GLOBAL_BUILD_NUMBER=24
      ++ GLOBAL_BUILD_NUMBER=24
      +++ printf %04d 24
      ++ export OUTPUT_FOLDER=ETE_0024_health
      ++ OUTPUT_FOLDER=ETE_0024_health
      ++ VARIABLEFILES='-V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py'
      ++ VARIABLES='-v GLOBAL_BUILD_NUMBER:22147'
      ++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- /var/opt/OpenECOMP_ETE/runTags.sh -V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py -v GLOBAL_BUILD_NUMBER:22147 -d /share/logs/ETE_0024_health -i health --display 88
      Basic SDNC Health Check                                               | PASS |
      command terminated with exit code 3
      root@oom-rancher:~# oom/kubernetes/robot/ete-k8s.sh onap health |grep SDNC
      ++ export NAMESPACE=onap
      ++ NAMESPACE=onap
      +++ kubectl --namespace onap get pods
      +++ grep robot
      +++ sed 's/ .*//'
      ++ POD=dev-robot-robot-5f7c96cb9c-4zcfv
      ++ export 'TAGS=-i health'
      ++ TAGS='-i health'
      ++ export ETEHOME=/var/opt/OpenECOMP_ETE
      ++ ETEHOME=/var/opt/OpenECOMP_ETE
      +++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- bash -c 'ls -1q /share/logs/ | wc -l'
      ++ export GLOBAL_BUILD_NUMBER=25
      ++ GLOBAL_BUILD_NUMBER=25
      +++ printf %04d 25
      ++ export OUTPUT_FOLDER=ETE_0025_health
      ++ OUTPUT_FOLDER=ETE_0025_health
      ++ VARIABLEFILES='-V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py'
      ++ VARIABLES='-v GLOBAL_BUILD_NUMBER:22195'
      ++ kubectl --namespace onap exec dev-robot-robot-5f7c96cb9c-4zcfv -- /var/opt/OpenECOMP_ETE/runTags.sh -V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py -v GLOBAL_BUILD_NUMBER:22195 -d /share/logs/ETE_0025_health -i health --display 88
      Basic SDNC Health Check                                               | FAIL |
      command terminated with exit code 3
      

       

            xuyang11 xuyang11
            xuyang11 xuyang11
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: