Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-294

OPEN-LAB OOM multiple container failures on clean 16g VM (OOM subset)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Medium Medium
    • Beijing Release
    • Beijing Release
    • None

      branch: master: 20171123

      started subset that fits in 9g of 16g

      expected only vnc-portal to fail - all pods except a couple db pods fail 

      investigating

       

       config

      in oom tenant

      oom-cd-obrien-cd0 (not found)
      admin-private-mgmt * 10.10.2.15
      • 10.12.25.117
      		 m1.xlarge openlab_oom_key Active nova None Running 16 hours, 20 minutes

      Nameoom-cd-obrien-cd0ID1fe78720-e418-47f7-bcfd-b6b93c791448StatusActiveAvailability ZonenovaCreatedNov. 23, 2017, 12:40 p.m.Time Since Created16 hours, 17 minutes

      Specs

      Flavor Namem1.xlargeFlavor ID1534679b-8601-497e-bc73-500b6e435275RAM16GBVCPUs8 VCPUVCPUs (min/cur/max)8/8/8Disk160GB

      Network Interfaces (NICs)

      nic1Port ID600dffe5-90c3-41d1-8ff9-50e0ea5938b9Networkadmin-private-mgmtVIF ModelvirtioMTU1500MAC Addressfa:16:3e:2f:1c:7eVirtual PCI Address

      IP Addresses

      Admin-Private-Mgmt10.10.2.15,  10.12.25.117

       

       

      ubuntu@oom-cd-obrien-cd0:~$ kubectl -n onap-log logs -f elasticsearch-6df4f65775-hkx6m
[2017-11-24T12:55:27,005][INFO ][o.e.n.Node               ] [] initializing ...
[2017-11-24T12:55:27,023][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.5.0.jar:5.5.0]
Caused by: java.lang.IllegalStateException: Failed to create node environment
at org.elasticsearch.node.Node.<init>(Node.java:267) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.node.Node.<init>(Node.java:244) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.5.0.jar:5.5.0]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_131]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_131]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_131]
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:221) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.node.Node.<init>(Node.java:244) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) ~[elasticsearch-5.5.0.jar:5.5.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.5.0.jar:5.5.0]
... 6 more
ubuntu@oom-cd-obrien-cd0:~$ free
              total        used        free      shared  buff/cache   available
Mem:       16431868     9483284     2346680       48308     4601904     6410580
Swap:             0           0           0
ubuntu@oom-cd-obrien-cd0:~$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             8208844        0   8208844   0% /dev
tmpfs            1643188    24452   1618736   2% /run
/dev/vda1      101584140 51252020  50315736  51% /
tmpfs            8215932    12476   8203456   1% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
tmpfs            8215932        0   8215932   0% /sys/fs/cgroup
tmpfs            1643188        0   1643188   0% /run/user/1000

       

      ubuntu@oom-cd-obrien-cd0:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS              RESTARTS   AGE
kube-system   heapster-76b8cd7b5-lfr64                1/1       Running             0          11h
kube-system   kube-dns-5d7b4487c9-8lm2f               3/3       Running             0          11h
kube-system   kubernetes-dashboard-5ffb9c9bb7-dcxg6   1/1       Running             0          11h
kube-system   monitoring-grafana-997796fcf-5fmnh      1/1       Running             0          11h
kube-system   monitoring-influxdb-56fdcd96b-rg48l     1/1       Running             0          11h
kube-system   tiller-deploy-79c88c5d98-5sgh9          1/1       Running             0          11h
onap-aai      aai-resources-5455bbf989-zblft          0/2       CrashLoopBackOff    176        11h
onap-aai      aai-service-68b6684dfb-tqnft            0/1       CrashLoopBackOff    136        11h
onap-aai      aai-traversal-6767dc77c6-2zk47          0/2       CrashLoopBackOff    271        11h
onap-aai      data-router-94d778cff-wjj28             0/1       CrashLoopBackOff    134        11h
onap-aai      elasticsearch-6b577bf757-nk74j          0/1       CrashLoopBackOff    136        11h
onap-aai      hbase-576777bd56-z4d96                  1/1       Running             0          11h
onap-aai      model-loader-service-7b57d5d84c-s822n   0/2       CrashLoopBackOff    269        11h
onap-aai      search-data-service-568bc99b9c-tclnd    0/2       CrashLoopBackOff    270        11h
onap-aai      sparky-be-765d5458f9-fd49g              0/2       CrashLoopBackOff    269        11h
onap-log      elasticsearch-6df4f65775-hkx6m          0/1       CrashLoopBackOff    136        11h
onap-log      kibana-846489d66d-75766                 1/1       Running             0          11h
onap-log      logstash-68f8d87968-mmkgv               1/1       Running             0          11h
onap-portal   portalapps-547d5686cb-jddt6             0/2       RunContainerError   274        11h
onap-portal   portaldb-6d6f7f849c-vmrlg               1/2       RunContainerError   137        11h
onap-portal   portalwidgets-859d4844bd-pp5z6          0/1       CrashLoopBackOff    135        11h
onap-portal   vnc-portal-5b45665475-x57cp             0/1       CrashLoopBackOff    136        11h
onap-sdc      sdc-be-7f4ffdc488-tgtqt                 0/2       ContainerCreating   0          11h
onap-sdc      sdc-cs-688597fdfc-nhvjt                 0/1       CrashLoopBackOff    136        11h
onap-sdc      sdc-es-85fdb4ddf5-7q7bb                 1/1       Running             0          11h
onap-sdc      sdc-fe-6757d7c995-vpgrj                 0/2       CrashLoopBackOff    273        11h
onap-sdc      sdc-kb-6dc85f9d8f-ck52n                 1/1       Running             0          11h
onap-uui      uui-6d6b45cc6b-xbkq6                    1/1       Running             0          11h

       

            michaelobrien michaelobrien
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: