In our Dublin daily CI chain (fresh installation using OOM on top of a k8s installed by kubespray), we got the following status

• All the SDNC pods are up&running
• Robot Healthcheck SDNC is Fail
• java exception in sdnc saying that the mariadb jdbc driver cannot be loaded

we were used to have this exception at the beginning of the trace one or two times before the connection occurs. But here it keeps on displaying this exception (see full traces attached)

2019-07-02T13:33:45,829 | ERROR | qtp575915670-133 | sliapiProvider | 232 - org.onap.ccsdk.sli.core.sliapi-provider - 0.4.4 | - | Caught exception looking for directed graph for sli/healthcheck/sync
org.onap.ccsdk.sli.core.sli.ConfigurationException: SQL query failed
at org.onap.ccsdk.sli.core.sli.SvcLogicDblibStore.hasGraph(SvcLogicDblibStore.java:129) [227:org.onap.ccsdk.sli.core.sli-common:0.4.4]
at org.onap.ccsdk.sli.core.sli.provider.base.SvcLogicServiceImplBase.hasGraph(SvcLogicServiceImplBase.java:155) [477:wrap_file__opt_opendaylight_system_org_onap_ccsdk_sli_core_sli-provider-base_0.4.4_sli-provider-base-0.4.4.jar:0.0.0]
at Proxyf685d651_7971_4187_b3d4_3b04c495d2bc.hasGraph(Unknown Source) [?:?]
at org.onap.ccsdk.sli.core.sliapi.sliapiProvider.healthcheck(sliapiProvider.java:409) [232:org.onap.ccsdk.sli.core.sliapi-provider:0.4.4]
at org.opendaylight.mdsal.binding.dom.adapter.invoke.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:32) [302:org.opendaylight.mdsal.binding-dom-adapter:2.5.2]
at org.opendaylight.mdsal.binding.dom.adapter.invoke.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:57) [302:org.opendaylight.mdsal.binding-dom-adapter:2.5.2]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:86) [273:org.opendaylight.controller.sal-binding-broker-impl:1.8.2]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:73) [273:org.opendaylight.controller.sal-binding-broker-impl:1.8.2]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter$1.invokeRpc(DOMRpcRouter.java:81) [275:org.opendaylight.controller.sal-broker-impl:1.8.2]
at org.opendaylight.mdsal.dom.broker.DOMRpcRouter$OperationInvocation.invokeGlobalRpc(DOMRpcRouter.java:542) [314:org.opendaylight.mdsal.dom-broker:2.5.2]
at org.opendaylight.mdsal.dom.broker.DOMRpcRouter$OperationInvocation.invoke(DOMRpcRouter.java:487) [314:org.opendaylight.mdsal.dom-broker:2.5.2]
at org.opendaylight.mdsal.dom.broker.DOMRpcRouter$RpcServiceFacade.invokeRpc(DOMRpcRouter.java:423) [314:org.opendaylight.mdsal.dom-broker:2.5.2]
at Proxy53473a5c_ccde_4000_9369_1d16bb6f7e14.invokeRpc(Unknown Source) [?:?]
at Proxya9aae8dd_72a7_4b16_a99c_628a4d0d3078.invokeRpc(Unknown Source) [?:?]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:111) [275:org.opendaylight.controller.sal-broker-impl:1.8.2]
at Proxybc016ead_4b52_4e11_8eb7_5a80ccc8dca5.invokeRpc(Unknown Source) [?:?]
at Proxyd4605dd3_39f8_47e9_94c2_c77ddaae667a.invokeRpc(Unknown Source) [?:?]
at org.opendaylight.netconf.sal.restconf.impl.BrokerFacade.invokeRpc(BrokerFacade.java:496) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.invokeRpc(RestconfImpl.java:454) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.invokeRpc(StatisticsRestconfServiceWrapper.java:83) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.invokeRpc(RestconfCompositeWrapper.java:64) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308) [172:org.eclipse.jetty.servlets:9.3.24.v20180605]
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262) [172:org.eclipse.jetty.servlets:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [147:org.apache.shiro.core:1.3.2]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [147:org.apache.shiro.core:1.3.2]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [147:org.apache.shiro.core:1.3.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [148:org.apache.shiro.web:1.3.2]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51) [172:org.eclipse.jetty.servlets:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:205) [181:org.eclipse.jetty.websocket.server:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.opendaylight.aaa.filterchain.filters.CustomFilterAdapter.doFilter(CustomFilterAdapter.java:86) [253:org.opendaylight.aaa.filterchain:0.8.2]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [422:org.ops4j.pax.web.pax-web-jetty:6.0.11]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [168:org.eclipse.jetty.security:9.3.24.v20180605]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:296) [422:org.ops4j.pax.web.pax-web-jetty:6.0.11]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [422:org.ops4j.pax.web.pax-web-jetty:6.0.11]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.Server.handle(Server.java:539) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [162:org.eclipse.jetty.io:9.3.24.v20180605]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [162:org.eclipse.jetty.io:9.3.24.v20180605]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [162:org.eclipse.jetty.io:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: org.onap.ccsdk.sli.core.dblib.DBLibException: No active DB connection pools are available in RequestDataNoRecovery call.
at org.onap.ccsdk.sli.core.dblib.DBResourceManager.requestDataNoRecovery(DBResourceManager.java:535) ~[?:?]
at org.onap.ccsdk.sli.core.dblib.DBResourceManager.getData(DBResourceManager.java:462) ~[?:?]
at Proxy113f6d93_b194_4c0d_b813_126357e5ab1b.getData(Unknown Source) ~[?:?]
at Proxy3322ea50_6c2e_47ee_8813_220db30d1c98.getData(Unknown Source) ~[?:?]
at org.onap.ccsdk.sli.core.sli.SvcLogicDblibStore.hasGraph(SvcLogicDblibStore.java:115) ~[?:?]
... 94 more

One possible cause could be a problem due to the duplication of the mariadb-java-client library and/or a race condition during installation

mariaddb-java-client duplication

in fact there are 2 versions of this the mariadb driver in the container

• ./opt/opendaylight/system/org/mariadb/jdbc/mariadb-java-client/1.5.9/mariadb-java-client-1.5.9.jar
• ./opt/opendaylight/system/org/mariadb/jdbc/mariadb-java-client/2.1.1/mariadb-java-client-2.1.1.jar

race condition

we may notice that we do not have the error on our master-daily branch

the SDNC code/versions are the same but the way we install is a little bit different, on our master daily we start things incrementally. In Dublin we start everything at once.

Anyway it is surprising to see the docker running, helm chart OK and the configuration Completed when we got such looping java. exceptions

A simple possible workaround consists in  deleting the running sdnc pod.

I tried several delete in a row, I was not able to reproduce the issue I got the first time.