-
Bug
-
Resolution: Done
-
Highest
-
None
Startup is failing as of 20171115 - still fails under Rancher 1.6.11/helm 2.7.2 on client/server (works under Rancher 1.6.10/helm 2.3.0)
http://jenkins.onap.info/job/oom-cd/103/console
see fix in
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-vm.max_map_count64to256kblimit
via
https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n49
# fix virtual memory for onap-log:elasticsearch under Rancher 1.6.11 - OOM-431 sudo sysctl -w vm.max_map_count=262144
onap-log elasticsearch-6df4f65775-vklfl 0/1 CrashLoopBackOff 5 7m onap-log kibana-846489d66d-pm2lk 1/1 Running 0 7m onap-log logstash-68f8d87968-rbvnh 1/1 Running 0 7m ubuntu@ip-172-31-23-250:~$ kubectl -n onap-log logs -f elasticsearch-6df4f65775-vklfl [2017-11-15T05:27:03,430][INFO ][o.e.n.Node ] [] initializing ... [2017-11-15T05:27:03,665][INFO ][o.e.e.NodeEnvironment ] [DPjUrvL] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvda1)]], net usable_space [62.1gb], net total_space [116.2gb], spins? [possibly], types [ext4] [2017-11-15T05:27:03,667][INFO ][o.e.e.NodeEnvironment ] [DPjUrvL] heap size [1.9gb], compressed ordinary object pointers [true] [2017-11-15T05:27:03,684][INFO ][o.e.n.Node ] node name [DPjUrvL] derived from node ID [DPjUrvLcS-21ZJ69X8l7LQ]; set [node.name] to override [2017-11-15T05:27:03,684][INFO ][o.e.n.Node ] version[5.5.0], pid[1], build[260387d/2017-06-30T23:16:05.735Z], OS[Linux/4.4.0-1022-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b12] [2017-11-15T05:27:03,685][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [aggs-matrix-stats] [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [ingest-common] [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [lang-expression] [2017-11-15T05:27:09,270][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [lang-groovy] [2017-11-15T05:27:09,271][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [lang-mustache] [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [lang-painless] [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [parent-join] [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [percolator] [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [reindex] [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [transport-netty3] [2017-11-15T05:27:09,272][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded module [transport-netty4] [2017-11-15T05:27:09,275][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded plugin [ingest-geoip] [2017-11-15T05:27:09,275][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded plugin [ingest-user-agent] [2017-11-15T05:27:09,275][INFO ][o.e.p.PluginsService ] [DPjUrvL] loaded plugin [x-pack] [2017-11-15T05:27:14,754][INFO ][o.e.d.DiscoveryModule ] [DPjUrvL] using discovery type [zen] [2017-11-15T05:27:16,541][INFO ][o.e.n.Node ] initialized [2017-11-15T05:27:16,543][INFO ][o.e.n.Node ] [DPjUrvL] starting ... [2017-11-15T05:27:16,985][INFO ][o.e.t.TransportService ] [DPjUrvL] publish_address {10.42.7.99:9300}, bound_addresses {[::]:9300} [2017-11-15T05:27:17,019][INFO ][o.e.b.BootstrapChecks ] [DPjUrvL] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks ERROR: [1] bootstrap checks failed [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [2017-11-15T05:27:17,041][INFO ][o.e.n.Node ] [DPjUrvL] stopping ... [2017-11-15T05:27:17,191][INFO ][o.e.n.Node ] [DPjUrvL] stopped [2017-11-15T05:27:17,191][INFO ][o.e.n.Node ] [DPjUrvL] closing ... [2017-11-15T05:27:17,223][INFO ][o.e.n.Node ] [DPjUrvL] closed
- blocks
-
OOM-535 Upgrade Kubernetes from 1.8.6 to 1.9.2
- Closed
-
OOM-542 two pods not starting under OOM (CrashLoopBackOff) - under Rancher 1.6.12
- Closed
-
LOG-101 OOM Logging fixes tracking Epic
- Closed
- is blocked by
-
OOM-390 Wrong configuration of log/elasticsearch
- Closed
- relates to
-
LOG-287 Rancher 2.1 deployment POC
- Closed
- links to
- mentioned in
-
Page Loading...