-
Story
-
Resolution: Done
-
Medium
-
None
-
None
-
None
-
None
Environment :
I used Kubeadm to create a single node cluster.
Helm 1.8.2
Kubectl: client v1.8.10 server v1.8.12
Docker: 18.03.1-ce
AAI was deployed by OOM aai helm chart as 5/9. Here is the results.
[root@kubeadm-test centos]# kubectl get pod -n onap
NAME READY STATUS RESTARTS AGE
aai-aai-6cdf679c4-lz6ld 0/1 Init:0/1 2 27m
aai-aai-babel-6c8db4d7d7-gx2lx 1/1 Running 0 27m
aai-aai-cassandra-0 1/1 Running 0 27m
aai-aai-cassandra-1 1/1 Running 0 25m
aai-aai-cassandra-2 1/1 Running 0 23m
aai-aai-champ-6cc9d659-zzfn7 0/1 Init:0/1 2 27m
aai-aai-data-router-67ffdcdc7d-fsl2g 1/1 Running 0 27m
aai-aai-elasticsearch-7949b94d69-dwvjm 1/1 Running 0 27m
aai-aai-gizmo-794b4d4ccf-hwxr4 2/2 Running 0 27m
aai-aai-hbase-67669fd585-9g6ql 1/1 Running 0 27m
aai-aai-modelloader-5467bd574-s846m 2/2 Running 0 27m
aai-aai-resources-85c6c5d777-4b4gv 1/2 CrashLoopBackOff 6 27m
aai-aai-search-data-69f585557f-nldmz 2/2 Running 0 27m
aai-aai-sparky-be-59664d67d6-7jpth 2/2 Running 0 27m
aai-aai-traversal-7b4b64d77f-v6r7m 0/2 Init:0/1 0 27m
aai-aai-traversal-update-query-data-jz6cr 0/1 Init:0/1 2 27m
- check the aai-resources-readiness all aai-cassandra is ready**[root@kubeadm-test centos]# kubectl logs -n onap aai-aai-resources-85c6c5d777-4b4gv aai-resources-readiness...2018-05-09 17:43:04,477 - INFO - Checking if aai-cassandra is ready
2018-05-09 17:43:04,529 - INFO - aai-cassandra is ready!
2018-05-09 17:43:04,530 - INFO - aai-cassandra is ready!
2018-05-09 17:43:04,530 - INFO - aai-cassandra is not ready.
2018-05-09 17:43:09,535 - INFO - Checking if aai-cassandra is ready
2018-05-09 17:43:09,583 - INFO - aai-cassandra is ready!
2018-05-09 17:43:09,583 - INFO - aai-cassandra is ready!
2018-05-09 17:43:09,583 - INFO - aai-cassandra is not ready.
2018-05-09 17:43:14,587 - INFO - Checking if aai-cassandra is ready
2018-05-09 17:43:14,637 - INFO - aai-cassandra is ready!
2018-05-09 17:43:14,637 - INFO - aai-cassandra is ready!
2018-05-09 17:43:14,637 - INFO - aai-cassandra is ready! - The I check the log of aai-resource pod, seems the connection between aai-resource and aai-cassandra has issue**
---- NOTE — about to open graph (takes a little while)--------;
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:594)
Caused by: java.lang.ExceptionInInitializerError
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:98)
at org.onap.aai.dbgen.GenTester.main(GenTester.java:118)
... 8 more
Caused by: java.lang.RuntimeException: Failed to instantiate graphs
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:83)
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:55)
at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:88)
... 10 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1376)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:110)
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:80)
... 12 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 20 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:619)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:314)
... 25 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=aai-aai-cassandra-2(aai-aai-cassandra-2):9160, latency=30001(30001), attempts=3]Ti
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231)
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198)
at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146)
at com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateKeyspace(ThriftClusterImpl.java:321)
at com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:294)
at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:614)
... 26 more
3.I checked the config map of aai-resource
In aai-resources-db-real-configmap and aai-resources-db-cached-configmap
storage.hostname=aai-aai-cassandra-0,aai-aai-cassandra-1,aai-aai-cassandra-2,
The storage hostname is the pod name instead of cassandra service name. Is this the reason why aai-resource pod got error?
- is duplicated by
-
OOM-1077 AAI resources and traversal deployment failure on non-rancher envs
- Closed