Uploaded image for project: 'Integration'
  1. Integration
  2. INT-437

AAI-vm1 HEAT deploy failing to start all 6 containers

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Medium Medium
    • Beijing Release
    • Beijing Release
    • None

      20180203: actually this JIRA should be in INT project - it has nothing really to do with AAI - just with the HEAT bootstrap cripts

       

      Getting the following failed AAI deployment on the 20171125 HEAT deployment - vm2 is ok - Tried restarting aai1 - to simulate the reverse order startup issue

      oam_onap_awsf
      10.0.1.1
      10.12.5.118
      m1.large
      onap_key_awsf
      Active
      nova
      None
      Running
      1 hour
      
      root@onap-aai-inst1:~# docker ps
      The program 'docker' is currently not installed. You can install it by typing:
      apt-get install docker
      root@onap-aai-inst1:~# ls
      root@onap-aai-inst1:~# cd /opt
      root@onap-aai-inst1:/opt# ls
      aai  aai_install.sh  aai_vm_init.sh  config  docker  test-config
      
      aai install not run
      
      root@onap-aai-inst1:/opt# ./aai_install.sh
      Configuration file '/etc/default/docker'
      ==> File on system created by you or by a script.
      ==> File also in package provided by package maintainer.
         What would you like to do about it ?  Your options are:
          Y or I  : install the package maintainer's version
          N or O  : keep your currently-installed version
            D     : show the differences between the versions
            Z     : start a shell to examine the situation
      The default action is to keep your current version.
      *** docker (Y/I/N/O/D/Z) [default=N] ?
      docker start/running, process 19430
      Setting up cgroup-lite (1.9) ...
      cgroup-lite start/running
      Processing triggers for libc-bin (2.19-0ubuntu6.11) ...
      Processing triggers for ureadahead (0.100.0-16) ...
      mkdir: cannot create directory ‘/opt/docker’: File exists
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100   617    0   617    0     0   3488      0 --:--:-- --:--:-- --:--:--  3505
      100 7857k  100 7857k    0     0  4481k      0  0:00:01  0:00:01 --:--:-- 6874k
      docker stop/waiting
      docker start/running, process 19645
      fatal: destination path 'test-config' already exists and is not an empty directory.
      
      root@onap-aai-inst1:/opt# docker ps
      CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
      root@onap-aai-inst1:/opt# ls
      aai  aai_install.sh  aai_vm_init.sh  config  docker  test-config
      
      
      root@onap-aai-inst1:/opt# ./aai_vm_init.sh
      Already up-to-date.
      Login Succeeded
      v1.1.0: Pulling from openecomp/aai-resources
      c62795f78da9: Pull complete
      d4fceeeb758e: Pull complete
      5c9125a401ae: Pull complete
      0062f774e994: Pull complete
      6b33fd031fac: Pull complete
      9a115de3c55d: Pull complete
      55300a0e3724: Pull complete
      371557fd7d66: Pull complete
      d2bdda3e7e77: Pull complete
      9f576608b8d3: Pull complete
      5cd856c2dc16: Pull complete
      bc16c71b0b67: Pull complete
      9cc2b456d372: Pull complete
      Digest: sha256:e91f978a90d96329806b45d4401a6bc99df02cb1e50eb8a7e4228786d28ddf98
      Status: Downloaded newer image for nexus3.onap.org:10001/openecomp/aai-resources:v1.1.0
      v1.1.0: Pulling from openecomp/aai-traversal
      c62795f78da9: Already exists
      d4fceeeb758e: Already exists
      5c9125a401ae: Already exists
      0062f774e994: Already exists
      6b33fd031fac: Already exists
      9a115de3c55d: Already exists
      68c3d464854d: Pull complete
      faa3b2d2c320: Pull complete
      2c6b2af8254f: Pull complete
      5dd96440c821: Pull complete
      b08079c86d7c: Pull complete
      dcd84be09eca: Pull complete
      4d79a5ad749d: Pull complete
      Digest: sha256:fafe5333ba2385397a03ce5b1cc1753be3e05ea27ae198509c2e8eb6d7e5db3b
      Status: Downloaded newer image for nexus3.onap.org:10001/openecomp/aai-traversal:v1.1.0
      v1.1.0: Pulling from onap/search-data-service
      b2375db445eb: Pull complete
      83db2251c697: Pull complete
      ec87cd47d907: Pull complete
      5872a8ac3284: Pull complete
      6f59f41904eb: Pull complete
      e8fd751676de: Pull complete
      86393c59ac24: Pull complete
      c837f3b521d1: Pull complete
      986e4a8739d3: Pull complete
      45f3e629ef9d: Pull complete
      80b70b36c332: Pull complete
      2c835edf6d48: Pull complete
      19cb19ebc9be: Pull complete
      e91762c1b6f3: Pull complete
      Digest: sha256:054a6309515b8dc1f0577332ee845af98e53c1fd84a1448ffcb74ad86e758238
      Status: Downloaded newer image for nexus3.onap.org:10001/onap/search-data-service:v1.1.0
      v1.1.0: Pulling from onap/data-router
      b2375db445eb: Already exists
      83db2251c697: Already exists
      ec87cd47d907: Already exists
      5872a8ac3284: Already exists
      6f59f41904eb: Already exists
      3960630aeb3e: Pull complete
      204851a4c6bb: Pull complete
      216bc6e5c631: Pull complete
      2a21058f7444: Pull complete
      9de840837cba: Pull complete
      04a5a4718d0d: Pull complete
      9cf59472bcb1: Pull complete
      680d14a1d355: Pull complete
      a48e250250b3: Pull complete
      Digest: sha256:10b920dde66a0c61b153a569b7ca87ba531afeafd3f07c28a591f903d87c3ef1
      Status: Downloaded newer image for nexus3.onap.org:10001/onap/data-router:v1.1.0
      v1.1.0: Pulling from onap/model-loader
      b2375db445eb: Already exists
      83db2251c697: Already exists
      ec87cd47d907: Already exists
      5872a8ac3284: Already exists
      6f59f41904eb: Already exists
      18dbfb8cc2f3: Pull complete
      b74f3b3c26f4: Pull complete
      304f3fa31aab: Pull complete
      bfd8929fc07e: Pull complete
      3018e7cb928e: Pull complete
      f38d0a3e58ff: Pull complete
      29bef1ba1105: Pull complete
      5609b8da62d2: Pull complete
      20298b6b4454: Pull complete
      Digest: sha256:1a1b29dda2beff5bcfeee77f8a7ed3eb4119db15214c242c27d60fa1cc47408a
      Status: Downloaded newer image for nexus3.onap.org:10001/onap/model-loader:v1.1.0
      v1.1.0: Pulling from onap/sparky-be
      b2375db445eb: Already exists
      83db2251c697: Already exists
      ec87cd47d907: Already exists
      5872a8ac3284: Already exists
      6f59f41904eb: Already exists
      69aa3508d957: Pull complete
      b3fdf477559b: Pull complete
      0039f513d8a3: Pull complete
      1618b5487e73: Pull complete
      55261b56faf4: Pull complete
      4f23d6fd6171: Pull complete
      032366839203: Pull complete
      Digest: sha256:8c850a34decf38c02f1d0e0976659cab0bc39a65275b0714bf9f207cdceff8c2
      Status: Downloaded newer image for nexus3.onap.org:10001/onap/sparky-be:v1.1.0
      WARNING: The USER_ID variable is not set. Defaulting to a blank string.
      WARNING: The USER_ID variable is not set. Defaulting to a blank string.
      No stopped containers
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      
      
      root@onap-aai-inst1:/opt# docker ps
      CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS                    NAMES
      853afc3cecd1        nexus3.onap.org:10001/openecomp/aai-resources   "/bin/bash /opt/ap..."   7 minutes ago       Up 7 minutes        0.0.0.0:8447->8447/tcp   testconfig_aai-resources.api.simpledemo.openecomp.org_1
      
      
      checking other server
      root@onap-aai-inst2:~# docker ps
      CONTAINER ID        IMAGE                 COMMAND                   CREATED             STATUS              PORTS                                                                                                                                                                                               NAMES
      7c4289cb1431        elasticsearch:2.4.1   "/docker-entrypoin..."    About an hour ago   Up About an hour    0.0.0.0:9200->9200/tcp, 9300/tcp                                                                                                                                                                    elasticsearch
      886206c78364        aaionap/hbase:1.2.0   "/bin/sh -c \"/entr..."   About an hour ago   Up About an hour    0.0.0.0:2181->2181/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8085->8085/tcp, 0.0.0.0:9090->9090/tcp, 0.0.0.0:16000->16000/tcp, 0.0.0.0:16010->16010/tcp, 9095/tcp, 0.0.0.0:16201->16201/tcp, 16301/tcp   testconfig_aai.hbase.simpledemo.openecomp.org_1
      
      
      rebooting aai-1
      
      see
      root@onap-aai-inst1:/opt/test-config# cat deploy_vm1.sh
      RESOURCES_CONTAINER_NAME=$($DOCKER_COMPOSE_CMD up -d aai-resources.api.simpledemo.openecomp.org 2>&1 | grep 'Creating' | grep -v 'volume' | grep -v 'network' | awk '{ print $2; }' | head -1);
      wait_for_container $RESOURCES_CONTAINER_NAME '0.0.0.0:8447';
      
      intermittent in open-lab
      Started also failing on contacting gerrit
      
      root@onap-aai-inst1:/opt# ./aai_vm_init.sh
      fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not resolve host: gerrit.onap.org
      
      root@onap-aai-inst1:/opt# ./aai_vm_init.sh
      fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not resolve host: gerrit.onap.org
      Error response from daemon: Get https://nexus3.onap.org:10001/v1/users/: dial tcp: lookup nexus3.onap.org on 10.0.100.1:53: read udp 10.0.1.1:34203->10.0.100.1:53: read: connection refused
      
      
      repos working
      https://gerrit.onap.org/r/#/admin/projects/aai/test-config
      
      restack 30 min later OK - but still no resources container
      root@onap-aai-inst1:/opt/test-config# ./deploy_vm1.sh
      Login Succeeded
      v1.1.0: Pulling from openecomp/aai-resources
      c62795f78da9: Already exists
      d4fceeeb758e: Already exists
      5c9125a401ae: Already exists
      0062f774e994: Already exists
      6b33fd031fac: Already exists
      9a115de3c55d: Already exists
      55300a0e3724: Already exists
      371557fd7d66: Downloading [=========================================>         ]  137.4MB/165.8MB
      
      
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
      ERROR: testconfig_aai-resources.api.simpledemo.openecomp.org_1 deployment failed.
      root@onap-aai-inst1:/opt/test-config# docker ps -a
      CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS                   PORTS               NAMES
      cb4345647f0a        nexus3.onap.org:10001/openecomp/aai-resources   "/bin/bash /opt/ap..."   4 hours ago         Exited (1) 3 hours ago                       testconfig_aai-resources.api.simpledemo.openecomp.org_1

            Unassigned Unassigned
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: