-
Bug
-
Resolution: Done
-
High
-
Guilin Release
-
None
I have lab installed from master with VES Collector 1.7.6 . I used it to test events handling, where i send around only 20 events.
After that time lab was not used few days and I observed that VES Collector is down due to memory issues.
--------------------------------------------------------------------------------------------------
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.HashMap.resize(Unknown Source)
at java.base/java.util.HashMap.putVal(Unknown Source)
at java.base/java.util.HashMap.put(Unknown Source)
at java.base/sun.security.util.MemoryCache.put(Unknown Source)
at java.base/sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
at java.base/sun.security.ssl.Finished$T12FinishedConsumer.onConsumeFinished(Unknown Source)
at java.base/sun.security.ssl.Finished$T12FinishedConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
---------------------------------------------------------------------------------------------------
What i observed is that VES config is very often updated with usage off CBS client. every time new connection is set-up. It can be hint where to search memory issues.
UPDATED by Bogumil Zebek:
In Java heap dump there is an error reported by Reactor thread. As I know CBS client is using reactor framework.
Tread 'reactor-http-epoll-1' with ID = 29
java.lang.OutOfMemoryError.<init>(Native method)
java.util.HashMap.resize(Native method)
java.util.HashMap.putVal(Native method)
java.util.HashMap.put(Native method)
sun.security.util.MemoryCache.put(Native method)
sun.security.ssl.SSLSessionContextImpl.put(Native method)
sun.security.ssl.Finished$T12FinishedConsumer.onConsumeFinished(Native method)
sun.security.ssl.Finished$T12FinishedConsumer.consume(Native method)
sun.security.ssl.SSLHandshake.consume(Native method)
sun.security.ssl.HandshakeContext.dispatch(Native method)
sun.security.ssl.HandshakeContext.dispatch(Native method)
sun.security.ssl.TransportContext.dispatch(Native method)
sun.security.ssl.SSLTransport.decode(Native method)
sun.security.ssl.SSLEngineImpl.decode(Native method)
sun.security.ssl.SSLEngineImpl.readRecord(Native method)
sun.security.ssl.SSLEngineImpl.unwrap(Native method)
sun.security.ssl.SSLEngineImpl.unwrap(Native method)
javax.net.ssl.SSLEngine.unwrap(Native method)
io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Native method)