Uploaded image for project: 'Active and Available Inventory'
  1. Active and Available Inventory
  2. AAI-140

AAI1 rackspace fails to start 8443 API model-loader and 8446 graph/traversal container - resources container timeout

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Highest Highest
    • None
    • None
    • None

       Reproduction: deploy master/1.1 to Rackspace - ----------AAI-1---------- VM fails to start fully (8443 api endpoint and 8446 traversal not up)

      Update:20170808: just found root issue in - AAI-134

      emoving testconfig_aai-resources.api.simpledemo.openecomp.org_1 ... done
Removing testconfig_sparky-be_1 ... done
Creating testconfig_sparky-be_1
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
...
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1' deployment to finish ...
ERROR: testconfig_aai-resources.api.simpledemo.openecomp.org_1 deployment failed.
2017-08-08 03:12:04,154 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
2017-08-08 03:12:04,155 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2017-08-08 03:12:04,155 - util.py[WARNING]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 0.7.5 finished at Tue, 08 Aug 2017 03:12:04 +0000. Datasource DataSourceConfigDriveNet [net,ver=2][source=/dev/xvdd].  Up 912.76 seconds

      So the issue may be just timing related on SDC - investigating - it looks however like a jaxb code generation issue with EclipseLink

       

      -- Loading new schema elements into Titan --
03:09:34.350 [main] DEBUG com.jayway.jsonpath.internal.path.CompiledPath - Evaluating path: $['rules'][*]
Making EdgeLabel: [hasClassOfService]
Making EdgeLabel: [hasLAGinterface]
Making EdgeLabel: [relatedTo]
Making EdgeLabel: [hasVlan]
Making EdgeLabel: [owns]
Making EdgeLabel: [usesLogicalLink]
Making EdgeLabel: [has]
Making EdgeLabel: [hostedOn]
Making EdgeLabel: [hasLInterface]
Making EdgeLabel: [supportsServiceCapability]
Making EdgeLabel: [hasLAGInterface]
Making EdgeLabel: [dependsOn]
Making EdgeLabel: [hasInstance]
Making EdgeLabel: [usesPhysicalLink]
Making EdgeLabel: [hosts]
Making EdgeLabel: [hasPinterface]
Making EdgeLabel: [isPartOf]
Making EdgeLabel: [hasCtagAssignment]
Making EdgeLabel: [bridgedTo]
Making EdgeLabel: [hasVNF]
Making EdgeLabel: [contains]
Making EdgeLabel: [locatedIn]
Making EdgeLabel: [usesLAGLink]
Making EdgeLabel: [usesCtagPool]
Making EdgeLabel: [usesVnfImage]
Making EdgeLabel: [hasFlavor]
Making EdgeLabel: [hasSitePairSet]
Making EdgeLabel: [definedFor]
Making EdgeLabel: [hasImage]
Making EdgeLabel: [runsOnPserver]
Making EdgeLabel: [targets]
Making EdgeLabel: [subscribesTo]
Making EdgeLabel: [hasVigServer]
Making EdgeLabel: [hasVolume]
Making EdgeLabel: [hasRoutingInstance]
Making EdgeLabel: [usesL3Network]
Making EdgeLabel: [isA]
Making EdgeLabel: [isMemberOf]
Making EdgeLabel: [hasAvailabilityZone]
Making EdgeLabel: [groupsResourcesIn]
Making EdgeLabel: [hasServiceInstance]
Making EdgeLabel: [existsIn]
Making EdgeLabel: [memberOf]
Making EdgeLabel: [hasIPAGFacingVLAN]
Making EdgeLabel: [hasSubnet]
Making EdgeLabel: [hasCtagPool]
Making EdgeLabel: [connectsTo]
Making EdgeLabel: [supportsAvailabilityZone]
Making EdgeLabel: [hasPortGroup]
Making EdgeLabel: [hasSitePair]
Making EdgeLabel: [hasMetaDatum]
Making EdgeLabel: [usesPInterface]
Making EdgeLabel: [hasCTag]
Making EdgeLabel: [usesVpnBinding]
Making EdgeLabel: [hasNetworkProfile]
Making EdgeLabel: [runsOnVserver]
Making EdgeLabel: [uses]
Making EdgeLabel: [hasMetaData]
Making EdgeLabel: [hasIpAddress]
Making EdgeLabel: [belongsTo]
Making EdgeLabel: [startsWith]
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.openecomp.aai.introspection.ModelInjestor.getInstance(ModelInjestor.java:69)
at org.openecomp.aai.introspection.MoxyLoader.process(MoxyLoader.java:118)
at org.openecomp.aai.introspection.MoxyLoader.<init>(MoxyLoader.java:69)
at org.openecomp.aai.introspection.LoaderFactory.createLoaderForVersion(LoaderFactory.java:36)
at org.openecomp.aai.dbgen.SchemaGenerator.loadSchemaIntoTitan(SchemaGenerator.java:118)
at org.openecomp.aai.dbgen.SchemaGenerator.loadSchemaIntoTitan(SchemaGenerator.java:68)
at org.openecomp.aai.dbgen.GenTester.main(GenTester.java:120)
Caused by: Exception [EclipseLink-50072] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.JAXBException
Exception Description: Duplicate Property named [invStatus] found on class [inventory.aai.onap.org.v11.PInterface]
at org.eclipse.persistence.exceptions.JAXBException.duplicatePropertyName(JAXBException.java:971)
at org.eclipse.persistence.jaxb.compiler.TypeInfo.addProperty(TypeInfo.java:314)
at org.eclipse.persistence.jaxb.compiler.TypeInfo.setProperties(TypeInfo.java:340)
at org.eclipse.persistence.jaxb.compiler.AnnotationsProcessor.buildTypeInfo(AnnotationsProcessor.java:864)
at org.eclipse.persistence.jaxb.compiler.AnnotationsProcessor.postBuildTypeInfo(AnnotationsProcessor.java:773)
at org.eclipse.persistence.jaxb.compiler.XMLProcessor.processXML(XMLProcessor.java:389)
at org.eclipse.persistence.jaxb.compiler.Generator.<init>(Generator.java:106)
at org.eclipse.persistence.jaxb.dynamic.DynamicJAXBContext$MetadataContextInput.createContextState(DynamicJAXBContext.java:236)
at org.eclipse.persistence.jaxb.JAXBContext.<init>(JAXBContext.java:192)
at org.eclipse.persistence.jaxb.dynamic.DynamicJAXBContext.<init>(DynamicJAXBContext.java:71)
at org.eclipse.persistence.jaxb.dynamic.DynamicJAXBContextFactory.createContextFromOXM(DynamicJAXBContextFactory.java:347)
at org.openecomp.aai.introspection.ModelInjestor.injestModel(ModelInjestor.java:97)
at org.openecomp.aai.introspection.ModelInjestor.injestModels(ModelInjestor.java:81)
at org.openecomp.aai.introspection.ModelInjestor.<init>(ModelInjestor.java:53)
at org.openecomp.aai.introspection.ModelInjestor.<init>(ModelInjestor.java:41)
at org.openecomp.aai.introspection.ModelInjestor$Helper.<clinit>(ModelInjestor.java:60)
... 7 more
root@vm1-aai-inst1:/opt/test-config# docker logs testconfig_aai-resources.api.simpledemo.openecomp.org_1

       

       aai1 vm before

       

      root@vm1-aai-inst1:~# docker ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS                                                NAMES
990cafe29c8a        nexus3.onap.org:10001/openecomp/aai-resources   "/bin/sh -c ./dock..."   6 minutes ago       Up 6 minutes        0.0.0.0:8447->8447/tcp                               testconfig_aai-resources.api.simpledemo.openecomp.org_1
903b776aefff        attos/dmaap                                     "/bin/sh -c ./star..."   7 minutes ago       Up 7 minutes        0.0.0.0:3904-3905->3904-3905/tcp                     dockerfiles_dmaap_1
3464d7b32ad0        dockerfiles_kafka                               "start-kafka.sh"         7 minutes ago       Up 7 minutes        0.0.0.0:9092->9092/tcp                               dockerfiles_kafka_1
cd7951896575        wurstmeister/zookeeper                          "/bin/sh -c '/usr/..."   7 minutes ago       Up 7 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   dockerfiles_zookeeper_1

       

       

       aai1 vm after

       

      obrienbiometrics:onap michaelobrien$ openstack stack create -t onap_rackspace.yaml -e onap_rackspace_11_iad.env ONAP0807_11_aai1_fix2
+---------------------+-----------------------------------------+
| Field               | Value                                   |
+---------------------+-----------------------------------------+
| id                  | a8e13d52-f556-4cf7-a423-31f50aea56d8    |
| stack_name          | ONAP0807_11_aai1_fix2                   |
| description         | Heat template to deploy ONAP components |
| creation_time       | 2017-08-08T01:32:09                     |
| updated_time        | None                                    |
| stack_status        | CREATE_IN_PROGRESS                      |
| stack_status_reason |                                         |
+---------------------+-----------------------------------------+

root@vm1-aai-inst1:~# tail -f /var/log/cloud-init-output.log
latest: Pulling from aaidocker/aai-hbase-1.2.3
root@vm1-aai-inst1:~# docker ps
CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS              PORTS               NAMES
44463cfe2a71        nexus3.onap.org:10001/aaidocker/aai-hbase-1.2.3   "/bin/bash /opt/en..."   10 seconds ago      Up 9 seconds                            hbase-1.2.3

root@vm1-aai-inst1:/opt# ls
aai_install.sh  aai_vm_init.sh  config  docker  openecomp

# seems kind of arbitrary to wait 180 sec
# Wait 3 minutes before instantiating the A&AI container
sleep 180
waiting on 5 sdc containers to come up in aai_vm_init.sh after a 180 sec pre-wait
while [[ $RES -lt 5 ]]
do
   RES=$(curl http://$SDC_IP_ADDR:8181/sdc1/rest/healthCheck | grep -c "OK")
done

# At this point, SDC is healthy and Model Loader container can start
docker run --name=model-loader-service -it -d -e DISTR_CLIENT_ASDC_ADDRESS=c2.vm1.sdc.simpledemo.openecomp.org:8443 -e DISTR_CLIENT_ENVIRONMENT_NAME=$DMAAP_TOPIC -e DISTR_CLIENT_USER=aai -e DISTR_CLIENT_PASSWORD=OBF:1ks51l8d1o3i1pcc1r2r1e211r391kls1pyj1z7u1njf1lx51go21hnj1y0k1mli1sop1k8o1j651vu91mxw1vun1mze1vv11j8x1k5i1sp11mjc1y161hlr1gm41m111nkj1z781pw31kku1r4p1e391r571pbm1o741l4x1ksp -e APP_SERVER_BASE_URL=https://c1.vm1.aai.simpledemo.openecomp.org:8443 -e APP_SERVER_KEYSTORE_PASSWORD=OBF:1i9a1u2a1unz1lr61wn51wn11lss1unz1u301i6o -e APP_SERVER_AUTH_USER=ModelLoader -e APP_SERVER_AUTH_PASSWORD=OBF:1qvu1v2h1sov1sar1wfw1j7j1wg21saj1sov1v1x1qxw $NEXUS_DOCKER_REPO/openecomp/model-loader:$DOCKER_IMAGE_VERSION


root@vm1-aai-inst1:/opt# ps -ef | grep aai
root      1740  1735  0 01:40 ?        00:00:00 /bin/bash ./aai_install.sh
root     14372  1740  0 01:43 ?        00:00:00 /bin/bash ./aai_vm_init.sh

root@vm1-aai-inst1:/opt# docker ps
CONTAINER ID        IMAGE                                                         COMMAND                  CREATED             STATUS              PORTS               NAMES
9ded374567af        nexus3.onap.org:10001/openecomp/ajsc-aai:1.1-STAGING-latest   "/bin/sh -c ./star..."   3 seconds ago       Up 2 seconds                            aai-service
44463cfe2a71        nexus3.onap.org:10001/aaidocker/aai-hbase-1.2.3               "/bin/bash /opt/en..."   5 minutes ago       Up 5 minutes                            hbase-1.2.3
containers up

all sdc up
root@vm1-aai-inst1:/opt# curl http://10.0.3.1:8181/sdc1/rest/healthCheck
{
  "sdcVersion": "1.1.0-SNAPSHOT",
  "siteMode": "unknown",
  "componentsInfo": [
    {
      "healthCheckComponent": "BE",
      "healthCheckStatus": "UP",
      "version": "1.1.0-SNAPSHOT",
      "description": "OK"
    },
    {
      "healthCheckComponent": "TITAN",
      "healthCheckStatus": "UP",
      "description": "OK"
    },
    {
      "healthCheckComponent": "DE",
      "healthCheckStatus": "UP",
      "description": "OK"
    },
    {
      "healthCheckComponent": "CASSANDRA",
      "healthCheckStatus": "UP",
      "description": "OK"
    },
    {
      "healthCheckComponent": "ON_BOARDING",
      "healthCheckStatus": "UP",
      "version": "1.1.0-SNAPSHOT",
      "description": "OK",
      "componentsInfo": [
        {
          "healthCheckComponent": "ZU",
          "healthCheckStatus": "UP",
          "version": "0.2.0",
          "description": "OK"
        },
        {
          "healthCheckComponent": "BE",
          "healthCheckStatus": "UP",
          "version": "1.1.0-SNAPSHOT",
          "description": "OK"
        },
        {
          "healthCheckComponent": "CAS",
          "healthCheckStatus": "UP",
          "version": "2.1.17",
          "description": "OK"
        },
        {
          "healthCheckComponent": "FE",
          "healthCheckStatus": "UP",
          "version": "1.1.0-SNAPSHOT",
          "description": "OK"
        }
      ]
    },
    {
      "healthCheckComponent": "FE",
      "healthCheckStatus": "UP",
      "version": "1.1.0-SNAPSHOT",
      "description": "OK"
    }
  ]

container exiting - investigating
root@vm1-aai-inst1:/opt# docker ps -a
CONTAINER ID        IMAGE                                                             COMMAND                  CREATED             STATUS                      PORTS               NAMES
d0e4fb1d3dba        nexus3.onap.org:10001/openecomp/model-loader:1.1-STAGING-latest   "/opt/app/model-lo..."   13 seconds ago      Exited (1) 12 seconds ago                       model-loader-service
c481b338e15c        nexus3.onap.org:10001/aaidocker/aai-hbase-1.2.3                   "/bin/bash /opt/en..."   13 seconds ago      Up 13 seconds                                   hbase-1.2.3
9ded374567af        nexus3.onap.org:10001/openecomp/ajsc-aai:1.1-STAGING-latest       "/bin/sh -c ./star..."   6 minutes ago       Up 6 minutes                                    aai-service


ok just figured out that aai_vm_init.sh is only 1.0.0, and we need aai2_vm_init.sh to call deploy_vm1.sh (the 8447 container at least)
reworking the SDC health timing

       

       

            jimmydot jimmydot
            michaelobrien michaelobrien
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: