Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-334

onap-log elasticsearch container memory config too low to start - fix: sysctl -w vm.max_map_count=262144 under Rancher 1.6.11+

    XMLWordPrintable

    Details

      Description

      Startup is failing as of 20171115 - still fails under Rancher 1.6.11/helm 2.7.2 on client/server (works under Rancher 1.6.10/helm 2.3.0)

      http://jenkins.onap.info/job/oom-cd/103/console
      see fix in
      https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-vm.max_map_count64to256kblimit
      via
      https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n49

      # fix virtual memory for onap-log:elasticsearch under Rancher 1.6.11 - OOM-431
        sudo sysctl -w vm.max_map_count=262144
      

       

      onap-log              elasticsearch-6df4f65775-vklfl          0/1       CrashLoopBackOff   5          7m
      onap-log              kibana-846489d66d-pm2lk                 1/1       Running            0          7m
      onap-log              logstash-68f8d87968-rbvnh               1/1       Running            0          7m
      
      ubuntu@ip-172-31-23-250:~$ kubectl -n onap-log logs -f elasticsearch-6df4f65775-vklfl
      [2017-11-15T05:27:03,430][INFO ][o.e.n.Node               ] [] initializing ...
      [2017-11-15T05:27:03,665][INFO ][o.e.e.NodeEnvironment    ] [DPjUrvL] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvda1)]], net usable_space [62.1gb], net total_space [116.2gb], spins? [possibly], types [ext4]
      [2017-11-15T05:27:03,667][INFO ][o.e.e.NodeEnvironment    ] [DPjUrvL] heap size [1.9gb], compressed ordinary object pointers [true]
      [2017-11-15T05:27:03,684][INFO ][o.e.n.Node               ] node name [DPjUrvL] derived from node ID [DPjUrvLcS-21ZJ69X8l7LQ]; set [node.name] to override
      [2017-11-15T05:27:03,684][INFO ][o.e.n.Node               ] version[5.5.0], pid[1], build[260387d/2017-06-30T23:16:05.735Z], OS[Linux/4.4.0-1022-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b12]
      [2017-11-15T05:27:03,685][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
      [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [aggs-matrix-stats]
      [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [ingest-common]
      [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [lang-expression]
      [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [lang-groovy]
      [2017-11-15T05:27:09,271][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [lang-mustache]
      [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [lang-painless]
      [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [parent-join]
      [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [percolator]
      [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [reindex]
      [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [transport-netty3]
      [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded module [transport-netty4]
      [2017-11-15T05:27:09,275][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded plugin [ingest-geoip]
      [2017-11-15T05:27:09,275][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded plugin [ingest-user-agent]
      [2017-11-15T05:27:09,275][INFO ][o.e.p.PluginsService     ] [DPjUrvL] loaded plugin [x-pack]
      [2017-11-15T05:27:14,754][INFO ][o.e.d.DiscoveryModule    ] [DPjUrvL] using discovery type [zen]
      [2017-11-15T05:27:16,541][INFO ][o.e.n.Node               ] initialized
      [2017-11-15T05:27:16,543][INFO ][o.e.n.Node               ] [DPjUrvL] starting ...
      [2017-11-15T05:27:16,985][INFO ][o.e.t.TransportService   ] [DPjUrvL] publish_address {10.42.7.99:9300}, bound_addresses {[::]:9300}
      [2017-11-15T05:27:17,019][INFO ][o.e.b.BootstrapChecks    ] [DPjUrvL] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
      ERROR: [1] bootstrap checks failed
      [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
      [2017-11-15T05:27:17,041][INFO ][o.e.n.Node               ] [DPjUrvL] stopping ...
      [2017-11-15T05:27:17,191][INFO ][o.e.n.Node               ] [DPjUrvL] stopped
      [2017-11-15T05:27:17,191][INFO ][o.e.n.Node               ] [DPjUrvL] closing ...
      [2017-11-15T05:27:17,223][INFO ][o.e.n.Node               ] [DPjUrvL] closed

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              Assignee:
              michaelobrien Michael O'Brien
              Reporter:
              michaelobrien Michael O'Brien
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: