Loading...

Type: Task
Resolution: Done
Priority: Medium
Fix Version/s: Casablanca Release
Affects Version/s: None
Labels:
None

After 1 day running OOM Beijing, I see one node more 90% disk space utilization of the 100 GB /var/lib/docker paritition, which is due to uui-server container logs taking ~60 GB disk space.

Now, I will have to delete the uui-server pod to protect the node.

What might be the reason for this behaviour ?

root@olc-bjng-1:~# free -h

total used free shared buff/cache available

Mem: 51G 41G 469M 84M 9.2G 8.8G

Swap: 0B 0B 0B

root@olc-bjng-1:~# df -h /dev/vda1 /dev/vdb

Filesystem Size Used Avail Use% Mounted on

/dev/vda1 49G 3.4G 46G 7% /

/dev/vdb 99G 45G 49G 48% /var/lib/docker

root@olc-bjng-1:~#

root@olc-bjng-2:~# free -h

total used free shared buff/cache available

Mem: 51G 14G 467M 59M 36G 35G

Swap: 0B 0B 0B

root@olc-bjng-2:~# df -h /dev/vda1 /dev/vdb

Filesystem Size Used Avail Use% Mounted on

/dev/vda1 49G 1.5G 47G 4% /

/dev/vdb 99G 88G 5.7G 94% /var/lib/docker <<<<<<<<<<<<<<<<<<<< 94% disk space utilization

root@olc-bjng-2:~#

root@olc-bjng-2:/var/lib/docker/containers/f2ce99b10b3e5706142444cef9fda518b45d5ac8c67fc23066d0261f8ee06c03# ls -l

total 62476500

drwx------ 2 root root 4096 Jun 6 12:14 checkpoints

~~rw-r~~r- 1 root root 48170 Jun 6 12:14 config.v2.json

~~rw-r~~---- 1 root root 63913391670 Jun 8 04:42 f2ce99b10b3e5706142444cef9fda518b45d5ac8c67fc23066d0261f8ee06c03-json.log <<<<<<<< 60 GB log file for uui-server container

~~rw-r~~r- 1 root root 1696 Jun 6 12:14 hostconfig.json

root@olc-bjng-2:/var/lib/docker/containers/f2ce99b10b3e5706142444cef9fda518b45d5ac8c67fc23066d0261f8ee06c03#

root@olc-bjng-3:~# free -h

total used free shared buff/cache available

Mem: 51G 38G 629M 130M 12G 12G

Swap: 0B 0B 0B

root@olc-bjng-3:~# df -h /dev/vda1 /dev/vdb

Filesystem Size Used Avail Use% Mounted on

/dev/vda1 49G 2.0G 47G 5% /

/dev/vdb 99G 52G 42G 56% /var/lib/docker

root@olc-bjng-3:~#

root@olc-rancher:~# kubectl get pods --all-namespaces

NAMESPACE NAME READY STATUS RESTARTS AGE

kube-system heapster-76b8cd7b5-72qcs 1/1 Running 1 1d

kube-system kube-dns-5d7b4487c9-rvgld 3/3 Running 3 1d

kube-system kubernetes-dashboard-f9577fffd-ttw84 1/1 Running 1 1d

kube-system monitoring-grafana-997796fcf-6hbq9 1/1 Running 1 1d

kube-system monitoring-influxdb-56fdcd96b-r8cnn 1/1 Running 1 1d

kube-system tiller-deploy-54bcc55dd5-kxn5x 1/1 Running 0 1d

onap dep-config-binding-service-6d44b5bbb4-rss4p 2/2 Running 0 1d

onap dep-dcae-tca-analytics-6896d58fd8-fjbj4 2/2 Running 0 1d

onap dep-dcae-ves-collector-7bfb6886b5-ntcdl 2/2 Running 0 1d

onap dep-deployment-handler-68b8d69879-6vtlx 2/2 Running 0 1d

onap dep-holmes-engine-mgmt-665755f7-br5j5 1/1 Running 0 1d

onap dep-holmes-rule-mgmt-765d65cbcc-2n9gv 1/1 Running 0 1d

onap dep-inventory-697d48fdbb-jb767 1/1 Running 0 1d

onap dep-policy-handler-5fd6989b7f-dq6gw 2/2 Running 0 1d

onap dep-pstg-write-787c4bb65b-84nzh 1/1 Running 0 1d

onap dep-service-change-handler-6996fdcccd-ncd5m 1/1 Running 11 1d

onap onap-aaf-cm-77fc7898c5-9w95w 1/1 Running 0 1d

onap onap-aaf-cs-6684b45df5-v9rv8 1/1 Running 0 1d

onap onap-aaf-fs-987668d95-8vcdb 1/1 Running 0 1d

onap onap-aaf-gui-75cb5c9c6-8b2nt 1/1 Running 0 1d

onap onap-aaf-hello-ddd9bc589-q7kxn 1/1 Running 0 1d

onap onap-aaf-locate-7c8b87989-tg2rt 1/1 Running 0 1d

onap onap-aaf-oauth-64dc475b6-gdlp8 1/1 Running 0 1d

onap onap-aaf-service-857cd86f8-cj4f4 1/1 Running 0 1d

onap onap-aaf-sms-6559d77ff8-2fcn8 1/1 Running 2 1d

onap onap-aaf-sms-quorumclient-0 1/1 Running 0 1d

onap onap-aaf-sms-quorumclient-1 1/1 Running 0 1d

onap onap-aaf-sms-quorumclient-2 1/1 Running 0 1d

onap onap-aaf-sms-vault-0 2/2 Running 1 1d

onap onap-aai-7968dd7685-qr7fx 1/1 Running 0 1d

onap onap-aai-babel-94544b766-njq6d 2/2 Running 0 1d

onap onap-aai-cassandra-0 1/1 Running 0 1d

onap onap-aai-cassandra-1 1/1 Running 0 1d

onap onap-aai-cassandra-2 1/1 Running 1 1d

onap onap-aai-champ-7d55cbb956-d47d5 0/1 Running 0 1d

onap onap-aai-data-router-65b8bcfd97-47cnk 1/1 Running 0 1d

onap onap-aai-elasticsearch-5668d555d7-vf9rk 1/1 Running 0 1d

onap onap-aai-gizmo-776d5bb56f-92788 2/2 Running 0 1d

onap onap-aai-hbase-db48bf969-cbkxk 1/1 Running 0 1d

onap onap-aai-modelloader-5947c7fc9b-9527g 2/2 Running 0 1d

onap onap-aai-resources-6768958d8f-z2cll 2/2 Running 0 1d

onap onap-aai-search-data-68dd4875fb-7c2xb 2/2 Running 0 1d

onap onap-aai-sparky-be-68755bc98c-2kvnf 2/2 Running 0 1d

onap onap-aai-traversal-5c667666f4-bdpg7 2/2 Running 0 1d

onap onap-appc-0 2/2 Running 0 1d

onap onap-appc-cdt-5dd69dc86f-jsw6g 1/1 Running 0 1d

onap onap-appc-db-0 2/2 Running 0 1d

onap onap-appc-dgbuilder-56dd4dcd48-8zmsp 1/1 Running 0 1d

onap onap-brmsgw-5498675564-qjn7v 1/1 Running 0 1d

onap onap-clamp-59456c78bf-wcmkg 2/2 Running 0 1d

onap onap-clamp-dash-es-7bbf5f6db4-wcqf8 1/1 Running 0 1d

onap onap-clamp-dash-kibana-5bb7fbd596-2gltb 1/1 Running 0 1d

onap onap-clamp-dash-logstash-775c4b7bcc-59ddw 1/1 Running 0 1d

onap onap-clampdb-9cfb679f6-6bxm6 1/1 Running 0 1d

onap onap-cli-6967bcbf64-srs96 1/1 Running 0 1d

onap onap-consul-7b9cbfd8c8-nqg5v 1/1 Running 0 1d

onap onap-consul-server-0 1/1 Running 0 1d

onap onap-consul-server-1 1/1 Running 0 1d

onap onap-consul-server-2 1/1 Running 0 1d

onap onap-dbc-pg-0 1/1 Running 0 1d

onap onap-dbc-pg-1 1/1 Running 0 1d

onap onap-dcae-cloudify-manager-854dbcdb4b-94q92 1/1 Running 0 1d

onap onap-dcae-db-0 1/1 Running 0 1d

onap onap-dcae-db-1 1/1 Running 0 1d

onap onap-dcae-healthcheck-746f9f4b9f-fkdnb 1/1 Running 0 1d

onap onap-dcae-redis-0 1/1 Running 0 1d

onap onap-dcae-redis-1 1/1 Running 0 1d

onap onap-dcae-redis-2 1/1 Running 0 1d

onap onap-dcae-redis-3 1/1 Running 0 1d

onap onap-dcae-redis-4 1/1 Running 0 1d

onap onap-dcae-redis-5 1/1 Running 0 1d

onap onap-dmaap-bus-controller-54b9589ccb-6lxp6 1/1 Running 0 1d

onap onap-drools-0 1/1 Running 0 1d

onap onap-esr-9df4786fc-zfvs7 2/2 Running 0 1d

onap onap-esr-gui-7cc64966d6-xj6hv 1/1 Running 0 1d

onap onap-kube2msb-5997c9fff9-bm8qj 1/1 Running 0 1d

onap onap-log-elasticsearch-7554b888f6-p5t7l 1/1 Running 0 1d

onap onap-log-kibana-85dc7958c4-64g56 1/1 Running 0 1d

onap onap-log-logstash-dl6jt 1/1 Running 0 1d

onap onap-log-logstash-gp6jr 1/1 Running 0 1d

onap onap-log-logstash-rkc88 1/1 Running 0 1d

onap onap-message-router-6b946854b9-8gchz 1/1 Running 0 1d

onap onap-message-router-kafka-9d5c55dff-z9rjf 1/1 Running 0 1d

onap onap-message-router-zookeeper-bbc7f4b8f-v754s 1/1 Running 0 1d

onap onap-msb-consul-5f79c7fd47-4f5gt 1/1 Running 0 1d

onap onap-msb-discovery-5b47cb74f5-glkwh 2/2 Running 0 1d

onap onap-msb-eag-7bb6d58bf6-px5nq 2/2 Running 0 1d

onap onap-msb-iag-7fc9c9c75-p7kf9 2/2 Running 0 1d

onap onap-multicloud-84f67b85b-994tz 2/2 Running 0 1d

onap onap-multicloud-ocata-56687995bf-pf8b7 2/2 Running 0 1d

onap onap-multicloud-vio-56c6d75965-9wb5h 2/2 Running 0 1d

onap onap-multicloud-windriver-6b7f9f897f-5gdtj 2/2 Running 0 1d

onap onap-nbi-859dcb94f5-svf7b 1/1 Running 0 1d

onap onap-nbi-mariadb-665b756dc-hs42k 1/1 Running 0 1d

onap onap-nbi-mongo-0 1/1 Running 0 1d

onap onap-nexus-54ddfc9497-hwqnw 1/1 Running 0 1d

onap onap-oof-bc6b58dd7-hkjjs 1/1 Running 0 1d

onap onap-oof-has-api-cbc5cb4f7-xxjsx 1/1 Running 0 1d

onap onap-oof-has-cassandra-85d4d65946-kw8z4 1/1 Running 0 1d

onap onap-oof-has-controller-668b57d9f8-p2zj9 1/1 Running 0 1d

onap onap-oof-has-data-8cdbc8588-xr9n5 1/1 Running 0 1d

onap onap-oof-has-music-7c85798b5f-6tdhw 1/1 Running 0 1d

onap onap-oof-has-reservation-6cf99fffc6-kwk2f 1/1 Running 0 1d

onap onap-oof-has-solver-78869cbd84-spndn 1/1 Running 0 1d

onap onap-oof-has-zookeeper-778b6b597b-7qjtv 1/1 Running 0 1d

onap onap-pap-5c9f7dbbd4-spg62 2/2 Running 0 1d

onap onap-pdp-0 2/2 Running 0 1d

onap onap-policydb-c8f56cd86-55kpj 1/1 Running 0 1d

onap onap-portal-app-765f57f9b7-g98dp 2/2 Running 0 1d

onap onap-portal-cassandra-fdd79fccc-b8gj4 1/1 Running 0 1d

onap onap-portal-db-7d9b8f4f4c-gvf77 1/1 Running 0 1d

onap onap-portal-sdk-6d4757d4fc-7twxl 2/2 Running 0 1d

onap onap-portal-widget-64c9d944f7-8fd7j 1/1 Running 0 1d

onap onap-portal-zookeeper-696c5f95cb-8ghnk 1/1 Running 0 1d

onap onap-robot-54899d4b69-2mk76 1/1 Running 0 1d

onap onap-sdc-be-777759bcb9-mpdcd 2/2 Running 0 1d

onap onap-sdc-cs-68649db4bc-kjgv2 1/1 Running 0 1d

onap onap-sdc-es-66ffbcd8fd-m6lwk 1/1 Running 0 1d

onap onap-sdc-fe-75fb4965bd-5n8sl 2/2 Running 0 1d

onap onap-sdc-kb-786db5585c-clclk 1/1 Running 0 1d

onap onap-sdc-onboarding-be-56487dd77b-79gg8 2/2 Running 0 1d

onap onap-sdc-wfd-7787cc697d-lhj4q 1/1 Running 0 1d

onap onap-sdnc-0 2/2 Running 0 1d

onap onap-sdnc-ansible-server-6c67cd5fc7-cql2c 1/1 Running 0 1d

onap onap-sdnc-db-0 2/2 Running 0 1d

onap onap-sdnc-dgbuilder-8b5d48776-ksbnv 1/1 Running 0 1d

onap onap-sdnc-dmaap-listener-8bb599776-p6pkj 1/1 Running 0 1d

onap onap-sdnc-portal-577694b6f6-4xlgn 1/1 Running 0 1d

onap onap-sdnc-ueb-listener-77d8bcc7f5-47nsv 1/1 Running 0 1d

onap onap-sniro-emulator-645cbb88c8-42jd6 1/1 Running 0 1d

onap onap-so-db-5665b9bb6d-4xgc9 1/1 Running 0 1d

onap onap-so-df8d8d5b6-qqwdm 2/2 Running 0 1d

onap onap-uui-687788b787-qkbg8 1/1 Running 0 1d

onap onap-uui-server-78b6bf799b-wlcl7 1/1 Running 0 1d

onap onap-vfc-catalog-b755ccff9-mfnk2 2/2 Running 0 1d

onap onap-vfc-ems-driver-76c89ccfb5-8c955 1/1 Running 5 1d

onap onap-vfc-generic-vnfm-driver-6cb87b49f6-62g6w 2/2 Running 0 1d

onap onap-vfc-huawei-vnfm-driver-8477d6cfbf-q4vvx 2/2 Running 0 1d

onap onap-vfc-juju-vnfm-driver-5b5b785554-b9g8k 2/2 Running 0 1d

onap onap-vfc-multivim-proxy-67b95d689b-q8ddv 1/1 Running 9 1d

onap onap-vfc-nokia-v2vnfm-driver-5c6cf89959-tbczx 1/1 Running 0 1d

onap onap-vfc-nokia-vnfm-driver-7bd7dcdb6c-l7scq 2/2 Running 0 1d

onap onap-vfc-nslcm-659c68b6d-dhw7q 2/2 Running 0 1d

onap onap-vfc-resmgr-5647f9b879-pc6hv 2/2 Running 0 1d

onap onap-vfc-vnflcm-7bd594bfb8-bbvxd 2/2 Running 0 1d

onap onap-vfc-vnfmgr-7464c44776-kzzq2 2/2 Running 0 1d

onap onap-vfc-vnfres-7767f4f5c7-xvj7p 2/2 Running 0 1d

onap onap-vfc-workflow-6bd85dd8f6-9mxhb 1/1 Running 9 1d

onap onap-vfc-workflow-engine-68547b7475-cvfl7 1/1 Running 5 1d

onap onap-vfc-zte-sdnc-driver-854cb6c8fd-kswsv 1/1 Running 2 1d

onap onap-vfc-zte-vnfm-driver-db575b5f9-qk2fn 2/2 Running 0 1d

onap onap-vid-79bb866c56-bbrhl 2/2 Running 0 1d

onap onap-vid-mariadb-galera-0 1/1 Running 0 1d

onap onap-vnfsdk-7d5dfc8978-qffmk 1/1 Running 0 1d

onap onap-vnfsdk-postgres-69f57c7897-vs5pl 1/1 Running 0 1d

root@olc-rancher:~# cd oom/kubernetes/robot/

root@olc-rancher:~/oom/kubernetes/robot# ./ete-k8s.sh onap health

Starting Xvfb on display :88 with res 1280x1024x24

Executing robot tests at log level TRACE

==============================================================================

OpenECOMP ETE

==============================================================================

OpenECOMP ETE.Robot

==============================================================================

OpenECOMP ETE.Robot.Testsuites

==============================================================================

OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp components are...

==============================================================================

Basic A&AI Health Check | PASS |

------------------------------------------------------------------------------

Basic AAF Health Check | PASS |

------------------------------------------------------------------------------

Basic AAF SMS Health Check | PASS |

------------------------------------------------------------------------------

Basic APPC Health Check | PASS |

------------------------------------------------------------------------------

Basic CLI Health Check | PASS |

------------------------------------------------------------------------------

Basic CLAMP Health Check | PASS |

------------------------------------------------------------------------------

Basic DCAE Health Check | PASS |

------------------------------------------------------------------------------

Basic DMAAP Message Router Health Check | PASS |

------------------------------------------------------------------------------

Basic External API NBI Health Check | PASS |

------------------------------------------------------------------------------

Basic Log Elasticsearch Health Check | PASS |

------------------------------------------------------------------------------

Basic Log Kibana Health Check | PASS |

------------------------------------------------------------------------------

Basic Log Logstash Health Check | PASS |

------------------------------------------------------------------------------

Basic Microservice Bus Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-ocata API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-titanium_cloud API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-vio API Health Check | PASS |

------------------------------------------------------------------------------

Basic OOF-Homing Health Check | PASS |

------------------------------------------------------------------------------

Basic OOF-SNIRO Health Check | PASS |

------------------------------------------------------------------------------

Basic Policy Health Check | PASS |

------------------------------------------------------------------------------

Basic Portal Health Check | PASS |

------------------------------------------------------------------------------

Basic SDC Health Check | PASS |

------------------------------------------------------------------------------

Basic SDNC Health Check | PASS |

------------------------------------------------------------------------------

Basic SO Health Check | PASS |

------------------------------------------------------------------------------

Basic UseCaseUI API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC catalog API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC emsdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC gvnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC huaweivnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC jujuvnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC multivimproxy API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC nokiavnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC nokiav2driver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC nslcm API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC resmgr API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC vnflcm API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC vnfmgr API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC vnfres API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC workflow API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC ztesdncdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC ztevnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VID Health Check | PASS |

------------------------------------------------------------------------------

Basic VNFSDK Health Check | PASS |

------------------------------------------------------------------------------

OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp compo... | PASS |

43 critical tests, 43 passed, 0 failed

43 tests total, 43 passed, 0 failed

==============================================================================

OpenECOMP ETE.Robot.Testsuites | PASS |

43 critical tests, 43 passed, 0 failed

43 tests total, 43 passed, 0 failed

==============================================================================

OpenECOMP ETE.Robot | PASS |

43 critical tests, 43 passed, 0 failed

43 tests total, 43 passed, 0 failed

==============================================================================

OpenECOMP ETE | PASS |

43 critical tests, 43 passed, 0 failed

43 tests total, 43 passed, 0 failed

==============================================================================

Output: /share/logs/ETE_0002_health/output.xml

Log: /share/logs/ETE_0002_health/log.html

Report: /share/logs/ETE_0002_health/report.html

root@olc-rancher:~/oom/kubernetes/robot# kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME

olc-bjng-1.novalocal Ready <none> 1d v1.8.10-rancher1 84.39.41.54 Ubuntu 16.04.1 LTS 4.4.0-127-generic docker://17.3.2

olc-bjng-2.novalocal Ready <none> 1d v1.8.10-rancher1 84.39.51.156 Ubuntu 16.04.1 LTS 4.4.0-127-generic docker://17.3.2

olc-bjng-3.novalocal Ready <none> 1d v1.8.10-rancher1 84.39.32.104 Ubuntu 16.04.1 LTS 4.4.0-127-generic docker://17.3.2

root@olc-rancher:~/oom/kubernetes/robot#

relates to

LOG-453 Syslog filling up cluster VM's - one VM at 12MB

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates