-
Bug
-
Resolution: Done
-
High
-
Dublin Release
-
None
-
Orange Openlab daily-dublin CI chain
OpenStack Rocky
Kubernetes
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}ONAP OOM dublin
Orange Openlab daily-dublin CI chain OpenStack Rocky Kubernetes Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} ONAP OOM dublin
-
El Alto Sprint 1, El Alto Sprint 4
In our Dublin daily CI chain (fresh installation using OOM on top of a k8s installed by kubespray), we got the following status
- All the SDNC pods are up&running
- Robot Healthcheck SDNC is Fail
- java exception in sdnc saying that the mariadb jdbc driver cannot be loaded
we were used to have this exception at the beginning of the trace one or two times before the connection occurs. But here it keeps on displaying this exception (see full traces attached)
2019-07-02T13:33:45,829 | ERROR | qtp575915670-133 | sliapiProvider | 232 - org.onap.ccsdk.sli.core.sliapi-provider - 0.4.4 | - | Caught exception looking for directed graph for sli/healthcheck/sync
org.onap.ccsdk.sli.core.sli.ConfigurationException: SQL query failed
at org.onap.ccsdk.sli.core.sli.SvcLogicDblibStore.hasGraph(SvcLogicDblibStore.java:129) [227:org.onap.ccsdk.sli.core.sli-common:0.4.4]
at org.onap.ccsdk.sli.core.sli.provider.base.SvcLogicServiceImplBase.hasGraph(SvcLogicServiceImplBase.java:155) [477:wrap_file__opt_opendaylight_system_org_onap_ccsdk_sli_core_sli-provider-base_0.4.4_sli-provider-base-0.4.4.jar:0.0.0]
at Proxyf685d651_7971_4187_b3d4_3b04c495d2bc.hasGraph(Unknown Source) [?:?]
at org.onap.ccsdk.sli.core.sliapi.sliapiProvider.healthcheck(sliapiProvider.java:409) [232:org.onap.ccsdk.sli.core.sliapi-provider:0.4.4]
at org.opendaylight.mdsal.binding.dom.adapter.invoke.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:32) [302:org.opendaylight.mdsal.binding-dom-adapter:2.5.2]
at org.opendaylight.mdsal.binding.dom.adapter.invoke.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:57) [302:org.opendaylight.mdsal.binding-dom-adapter:2.5.2]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:86) [273:org.opendaylight.controller.sal-binding-broker-impl:1.8.2]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:73) [273:org.opendaylight.controller.sal-binding-broker-impl:1.8.2]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter$1.invokeRpc(DOMRpcRouter.java:81) [275:org.opendaylight.controller.sal-broker-impl:1.8.2]
at org.opendaylight.mdsal.dom.broker.DOMRpcRouter$OperationInvocation.invokeGlobalRpc(DOMRpcRouter.java:542) [314:org.opendaylight.mdsal.dom-broker:2.5.2]
at org.opendaylight.mdsal.dom.broker.DOMRpcRouter$OperationInvocation.invoke(DOMRpcRouter.java:487) [314:org.opendaylight.mdsal.dom-broker:2.5.2]
at org.opendaylight.mdsal.dom.broker.DOMRpcRouter$RpcServiceFacade.invokeRpc(DOMRpcRouter.java:423) [314:org.opendaylight.mdsal.dom-broker:2.5.2]
at Proxy53473a5c_ccde_4000_9369_1d16bb6f7e14.invokeRpc(Unknown Source) [?:?]
at Proxya9aae8dd_72a7_4b16_a99c_628a4d0d3078.invokeRpc(Unknown Source) [?:?]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:111) [275:org.opendaylight.controller.sal-broker-impl:1.8.2]
at Proxybc016ead_4b52_4e11_8eb7_5a80ccc8dca5.invokeRpc(Unknown Source) [?:?]
at Proxyd4605dd3_39f8_47e9_94c2_c77ddaae667a.invokeRpc(Unknown Source) [?:?]
at org.opendaylight.netconf.sal.restconf.impl.BrokerFacade.invokeRpc(BrokerFacade.java:496) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.invokeRpc(RestconfImpl.java:454) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.invokeRpc(StatisticsRestconfServiceWrapper.java:83) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.invokeRpc(RestconfCompositeWrapper.java:64) [353:org.opendaylight.netconf.restconf-nb-bierman02:1.8.2]
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [196:org.glassfish.jersey.core.jersey-common:2.25.1]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [197:org.glassfish.jersey.core.jersey-server:2.25.1]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [194:org.glassfish.jersey.containers.jersey-container-servlet-core:2.25.1]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308) [172:org.eclipse.jetty.servlets:9.3.24.v20180605]
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262) [172:org.eclipse.jetty.servlets:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [147:org.apache.shiro.core:1.3.2]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [147:org.apache.shiro.core:1.3.2]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [147:org.apache.shiro.core:1.3.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [148:org.apache.shiro.web:1.3.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [148:org.apache.shiro.web:1.3.2]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51) [172:org.eclipse.jetty.servlets:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:205) [181:org.eclipse.jetty.websocket.server:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.opendaylight.aaa.filterchain.filters.CustomFilterAdapter.doFilter(CustomFilterAdapter.java:86) [253:org.opendaylight.aaa.filterchain:0.8.2]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [422:org.ops4j.pax.web.pax-web-jetty:6.0.11]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [168:org.eclipse.jetty.security:9.3.24.v20180605]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:296) [422:org.ops4j.pax.web.pax-web-jetty:6.0.11]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [171:org.eclipse.jetty.servlet:9.3.24.v20180605]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [422:org.ops4j.pax.web.pax-web-jetty:6.0.11]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.Server.handle(Server.java:539) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [170:org.eclipse.jetty.server:9.3.24.v20180605]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [162:org.eclipse.jetty.io:9.3.24.v20180605]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [162:org.eclipse.jetty.io:9.3.24.v20180605]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [162:org.eclipse.jetty.io:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [173:org.eclipse.jetty.util:9.3.24.v20180605]
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: org.onap.ccsdk.sli.core.dblib.DBLibException: No active DB connection pools are available in RequestDataNoRecovery call.
at org.onap.ccsdk.sli.core.dblib.DBResourceManager.requestDataNoRecovery(DBResourceManager.java:535) ~[?:?]
at org.onap.ccsdk.sli.core.dblib.DBResourceManager.getData(DBResourceManager.java:462) ~[?:?]
at Proxy113f6d93_b194_4c0d_b813_126357e5ab1b.getData(Unknown Source) ~[?:?]
at Proxy3322ea50_6c2e_47ee_8813_220db30d1c98.getData(Unknown Source) ~[?:?]
at org.onap.ccsdk.sli.core.sli.SvcLogicDblibStore.hasGraph(SvcLogicDblibStore.java:115) ~[?:?]
... 94 more
One possible cause could be a problem due to the duplication of the mariadb-java-client library and/or a race condition during installation
mariaddb-java-client duplication
in fact there are 2 versions of this the mariadb driver in the container
- ./opt/opendaylight/system/org/mariadb/jdbc/mariadb-java-client/1.5.9/mariadb-java-client-1.5.9.jar
- ./opt/opendaylight/system/org/mariadb/jdbc/mariadb-java-client/2.1.1/mariadb-java-client-2.1.1.jar
race condition
we may notice that we do not have the error on our master-daily branch
the SDNC code/versions are the same but the way we install is a little bit different, on our master daily we start things incrementally. In Dublin we start everything at once.
Anyway it is surprising to see the docker running, helm chart OK and the configuration Completed when we got such looping java. exceptions
A simple possible workaround consists in deleting the running sdnc pod.
I tried several delete in a row, I was not able to reproduce the issue I got the first time.
- is duplicated by
-
CCSDK-1683 Dblib throws class not found exception on MariaDB driver
- Closed