Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-1067

Error in aai-resources pod

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Medium Medium
    • None
    • None
    • None
    • None

      Environment : 

      I used Kubeadm to create a single node cluster.

      Helm 1.8.2

      Kubectl:  client v1.8.10  server  v1.8.12

      Docker: 18.03.1-ce

      AAI was deployed by OOM aai helm chart as 5/9. Here is the results.

      [root@kubeadm-test centos]# kubectl get pod -n onap
      NAME READY STATUS RESTARTS AGE
      aai-aai-6cdf679c4-lz6ld 0/1 Init:0/1 2 27m
      aai-aai-babel-6c8db4d7d7-gx2lx 1/1 Running 0 27m
      aai-aai-cassandra-0 1/1 Running 0 27m
      aai-aai-cassandra-1 1/1 Running 0 25m
      aai-aai-cassandra-2 1/1 Running 0 23m
      aai-aai-champ-6cc9d659-zzfn7 0/1 Init:0/1 2 27m
      aai-aai-data-router-67ffdcdc7d-fsl2g 1/1 Running 0 27m
      aai-aai-elasticsearch-7949b94d69-dwvjm 1/1 Running 0 27m
      aai-aai-gizmo-794b4d4ccf-hwxr4 2/2 Running 0 27m
      aai-aai-hbase-67669fd585-9g6ql 1/1 Running 0 27m
      aai-aai-modelloader-5467bd574-s846m 2/2 Running 0 27m
      aai-aai-resources-85c6c5d777-4b4gv 1/2 CrashLoopBackOff 6 27m
      aai-aai-search-data-69f585557f-nldmz 2/2 Running 0 27m
      aai-aai-sparky-be-59664d67d6-7jpth 2/2 Running 0 27m
      aai-aai-traversal-7b4b64d77f-v6r7m 0/2 Init:0/1 0 27m
      aai-aai-traversal-update-query-data-jz6cr 0/1 Init:0/1 2 27m

      1. check the aai-resources-readiness  all aai-cassandra is ready**[root@kubeadm-test centos]# kubectl logs -n onap aai-aai-resources-85c6c5d777-4b4gv aai-resources-readiness...2018-05-09 17:43:04,477 - INFO - Checking if aai-cassandra is ready
        2018-05-09 17:43:04,529 - INFO - aai-cassandra is ready!
        2018-05-09 17:43:04,530 - INFO - aai-cassandra is ready!
        2018-05-09 17:43:04,530 - INFO - aai-cassandra is not ready.
        2018-05-09 17:43:09,535 - INFO - Checking if aai-cassandra is ready
        2018-05-09 17:43:09,583 - INFO - aai-cassandra is ready!
        2018-05-09 17:43:09,583 - INFO - aai-cassandra is ready!
        2018-05-09 17:43:09,583 - INFO - aai-cassandra is not ready.
        2018-05-09 17:43:14,587 - INFO - Checking if aai-cassandra is ready
        2018-05-09 17:43:14,637 - INFO - aai-cassandra is ready!
        2018-05-09 17:43:14,637 - INFO - aai-cassandra is ready!
        2018-05-09 17:43:14,637 - INFO - aai-cassandra is ready!
      2. The I check the log of aai-resource pod, seems the connection between aai-resource and aai-cassandra has issue**

      ---- NOTE — about to open graph (takes a little while)--------;
      Exception in thread "main" java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
      at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:594)
      Caused by: java.lang.ExceptionInInitializerError
      at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:98)
      at org.onap.aai.dbgen.GenTester.main(GenTester.java:118)
      ... 8 more
      Caused by: java.lang.RuntimeException: Failed to instantiate graphs
      at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:83)
      at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:55)
      at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:88)
      ... 10 more
      Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager
      at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
      at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
      at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
      at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1376)
      at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
      at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
      at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
      at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:110)
      at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:80)
      ... 12 more
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
      ... 20 more
      Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
      at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:619)
      at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:314)
      ... 25 more
      Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=aai-aai-cassandra-2(aai-aai-cassandra-2):9160, latency=30001(30001), attempts=3]Ti
      at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231)
      at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198)
      at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84)
      at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117)
      at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
      at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146)
      at com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateKeyspace(ThriftClusterImpl.java:321)
      at com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:294)
      at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:614)
      ... 26 more

              3.I checked the config map of aai-resource

      In aai-resources-db-real-configmap and aai-resources-db-cached-configmap

      storage.hostname=aai-aai-cassandra-0,aai-aai-cassandra-1,aai-aai-cassandra-2,

       

       

      The storage hostname is the pod name instead of cassandra service name. Is this the reason why aai-resource pod got error?

            BorislavG Borislav Glozman
            jh245g jh245g
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: