Uploaded image for project: 'Data Collection, Analytics, and Events'
  1. Data Collection, Analytics, and Events
  2. DCAEGEN2-2495

Ves Collector is down because of java heap space

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: High High
    • Guilin Release
    • Guilin Release
    • None

      I have lab installed from master  with VES Collector 1.7.6 . I used it to test events handling, where i send around only 20 events.

      After that time lab was not used few days and I observed that VES Collector is down due to memory issues.

      --------------------------------------------------------------------------------------------------

      java.lang.OutOfMemoryError: Java heap space
      at java.base/java.util.HashMap.resize(Unknown Source)
      at java.base/java.util.HashMap.putVal(Unknown Source)
      at java.base/java.util.HashMap.put(Unknown Source)
      at java.base/sun.security.util.MemoryCache.put(Unknown Source)
      at java.base/sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at java.base/sun.security.ssl.Finished$T12FinishedConsumer.onConsumeFinished(Unknown Source)
      at java.base/sun.security.ssl.Finished$T12FinishedConsumer.consume(Unknown Source)
      at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
      at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
      at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
      at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
      at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
      at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
      at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
      at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
      at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
      at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
      at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280)
      at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332)
      at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
      at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
      at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
      at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)

      ---------------------------------------------------------------------------------------------------

      What i observed is that VES config is very often updated with usage off CBS client. every time new connection is set-up. It can be hint where to search memory issues.

       UPDATED by Bogumil Zebek:
      In Java heap dump there is an error reported by Reactor thread. As I know CBS client is using reactor framework.

      Tread 'reactor-http-epoll-1' with ID = 29

      java.lang.OutOfMemoryError.<init>(Native method)

      java.util.HashMap.resize(Native method)

      java.util.HashMap.putVal(Native method)

      java.util.HashMap.put(Native method)

      sun.security.util.MemoryCache.put(Native method)

      sun.security.ssl.SSLSessionContextImpl.put(Native method)

      sun.security.ssl.Finished$T12FinishedConsumer.onConsumeFinished(Native method)

      sun.security.ssl.Finished$T12FinishedConsumer.consume(Native method)

      sun.security.ssl.SSLHandshake.consume(Native method)

      sun.security.ssl.HandshakeContext.dispatch(Native method)

      sun.security.ssl.HandshakeContext.dispatch(Native method)

      sun.security.ssl.TransportContext.dispatch(Native method)

      sun.security.ssl.SSLTransport.decode(Native method)

      sun.security.ssl.SSLEngineImpl.decode(Native method)

      sun.security.ssl.SSLEngineImpl.readRecord(Native method)

      sun.security.ssl.SSLEngineImpl.unwrap(Native method)

      sun.security.ssl.SSLEngineImpl.unwrap(Native method)

      javax.net.ssl.SSLEngine.unwrap(Native method)

      io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280)

      io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332)

      io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)

      io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)

      io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)

      io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)

      io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)

      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)

      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)

      io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)

      io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)

      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)

      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)

      io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)

      io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)

      io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)

      io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)

      io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)

      io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

      io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

      java.lang.Thread.run(Native method)

       

       

            bogumil_zebek bogumil_zebek
            mprzybys mprzybys
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: