12-Sep-2019 01:54:22.694 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.45 12-Sep-2019 01:54:22.695 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Aug 14 2019 22:21:25 UTC 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.45.0 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.15.0-22-generic 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/local/openjdk-8/jre 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_222-b10 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation 12-Sep-2019 01:54:22.696 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/local/tomcat 12-Sep-2019 01:54:22.697 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/local/tomcat 12-Sep-2019 01:54:22.697 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties 12-Sep-2019 01:54:22.697 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 12-Sep-2019 01:54:22.697 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048 12-Sep-2019 01:54:22.697 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources 12-Sep-2019 01:54:22.697 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 12-Sep-2019 01:54:22.698 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs= 12-Sep-2019 01:54:22.698 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat 12-Sep-2019 01:54:22.698 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat 12-Sep-2019 01:54:22.698 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp 12-Sep-2019 01:54:22.698 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.23] using APR version [1.5.2]. 12-Sep-2019 01:54:22.698 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 12-Sep-2019 01:54:22.698 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 12-Sep-2019 01:54:22.702 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.0k 28 May 2019] 12-Sep-2019 01:54:22.770 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"] 12-Sep-2019 01:54:22.778 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read 12-Sep-2019 01:54:22.788 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"] 12-Sep-2019 01:54:22.789 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read 12-Sep-2019 01:54:22.790 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 432 ms 12-Sep-2019 01:54:22.810 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina] 12-Sep-2019 01:54:22.811 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.45 12-Sep-2019 01:54:22.835 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/local/tomcat/webapps/MUSIC.war] 12-Sep-2019 01:54:24.905 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. 12-Sep-2019 01:54:25.738 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file 12-Sep-2019 01:54:25.742 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [/MUSIC] startup failed due to previous errors 12-Sep-2019 01:54:25.752 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.util.regex.Matcher.group(Matcher.java:541) java.util.Formatter$FormatSpecifier.(Formatter.java:2720) java.util.Formatter.parse(Formatter.java:2560) java.util.Formatter.format(Formatter.java:2501) java.util.Formatter.format(Formatter.java:2455) java.lang.String.format(String.java:2940) com.datastax.driver.core.ControlConnection.logError(ControlConnection.java:287) com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:264) com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:107) com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1652) com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1571) com.datastax.driver.core.Cluster.getMetadata(Cluster.java:451) org.onap.music.datastore.MusicDataStore.connectToCassaCluster(MusicDataStore.java:230) org.onap.music.datastore.MusicDataStore.(MusicDataStore.java:119) org.onap.music.main.MusicCore.getDSHandle(MusicCore.java:129) org.onap.music.main.MusicCore.get(MusicCore.java:708) org.onap.music.main.CachingUtil.initializeAafCache(CachingUtil.java:109) org.onap.music.main.CachingUtil.run(CachingUtil.java:141) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) 12-Sep-2019 01:54:25.756 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [cluster1-connection-reaper-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) 12-Sep-2019 01:54:25.757 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [cluster2-connection-reaper-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) 12-Sep-2019 01:54:25.757 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [cluster2-nio-worker-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:741) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:472) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) 12-Sep-2019 01:54:25.758 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [cluster1-nio-worker-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) 12-Sep-2019 01:54:25.758 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [cluster2-timeouter-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Thread.sleep(Native Method) io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:567) io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) 12-Sep-2019 01:54:25.759 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [Shutdown-checker] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:253) io.netty.util.concurrent.DefaultPromise.syncUninterruptibly(DefaultPromise.java:344) io.netty.util.concurrent.DefaultPromise.syncUninterruptibly(DefaultPromise.java:33) com.datastax.driver.core.NettyOptions.onClusterClose(NettyOptions.java:198) com.datastax.driver.core.Connection$Factory.shutdown(Connection.java:1062) com.datastax.driver.core.Cluster$Manager$ClusterCloseFuture$1.run(Cluster.java:2776) 12-Sep-2019 01:54:25.759 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [MUSIC] appears to have started a thread named [cluster2-nio-worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:741) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:472) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) 12-Sep-2019 01:54:25.760 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [MUSIC] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7f2c0112]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@6291d754]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 12-Sep-2019 01:54:25.764 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/usr/local/tomcat/webapps/MUSIC.war] has finished in [2,928] ms 12-Sep-2019 01:54:25.766 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 12-Sep-2019 01:54:25.773 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"] 12-Sep-2019 01:54:25.775 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 2985 ms 12-Sep-2019 01:54:27.945 INFO [cluster2-nio-worker-0] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [io.netty.buffer.PoolArena$1]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [io.netty.buffer.PoolArena$1]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1380) at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1368) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1182) at io.netty.buffer.PoolArena.freeChunk(PoolArena.java:293) at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:460) at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:430) at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:422) at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:279) at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:270) at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:241) at io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.onRemoval(PooledByteBufAllocator.java:450) at io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.onRemoval(PooledByteBufAllocator.java:426) at io.netty.util.concurrent.FastThreadLocal.remove(FastThreadLocal.java:271) at io.netty.util.concurrent.FastThreadLocal.removeAll(FastThreadLocal.java:67) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:32) at java.lang.Thread.run(Thread.java:748)