Details
-
Story
-
Status: Closed
-
High
-
Resolution: Done
-
None
-
None
Description
melliott - these are notes for helping adjust the RTDs
Environment: master 20181210
https://wiki.onap.org/display/DW/OOM+Helm+%28un%29Deploy+plugins
Need to triage the logs for failed deployments
2 issues
sdc does not deploy - how to get the logs
aai deploys but is not listed - so we cannot use the plugin - and the base install/upgrade commands are blocked by the configmap issue
onap onap-aai-aai-resources-f75d854cd-l4spw 2/2 Running 0 22h
ubuntu@a-cd-cas:~$ helm list NAME REVISION UPDATED STATUS CHART NAMESPACE onap 1 Mon Dec 10 13:10:39 2018 DEPLOYED onap-3.0.0 onap onap-aaf 3 Mon Dec 10 13:10:40 2018 DEPLOYED aaf-3.0.0 onap onap-appc 3 Mon Dec 10 13:10:53 2018 DEPLOYED appc-3.0.0 onap onap-clamp 3 Mon Dec 10 13:11:00 2018 DEPLOYED clamp-3.0.0 onap onap-cli 3 Mon Dec 10 13:11:03 2018 DEPLOYED cli-3.0.0 onap onap-consul 3 Mon Dec 10 13:11:04 2018 DEPLOYED consul-3.0.0 onap onap-dcaegen2 3 Mon Dec 10 13:11:07 2018 DEPLOYED dcaegen2-3.0.0 onap onap-dmaap 3 Mon Dec 10 13:11:10 2018 DEPLOYED dmaap-3.0.0 onap onap-esr 3 Mon Dec 10 13:11:15 2018 DEPLOYED esr-3.0.0 onap onap-log 3 Mon Dec 10 13:11:16 2018 DEPLOYED log-3.0.0 onap onap-msb 3 Mon Dec 10 13:11:18 2018 DEPLOYED msb-3.0.0 onap onap-multicloud 3 Mon Dec 10 13:11:21 2018 DEPLOYED multicloud-3.0.0 onap onap-nbi 3 Mon Dec 10 13:11:25 2018 DEPLOYED nbi-3.0.0 onap onap-oof 3 Mon Dec 10 13:11:27 2018 DEPLOYED oof-3.0.0 onap onap-policy 3 Mon Dec 10 13:11:33 2018 DEPLOYED policy-3.0.0 onap onap-pomba 3 Mon Dec 10 13:11:40 2018 DEPLOYED pomba-3.0.0 onap onap-portal 3 Mon Dec 10 13:11:53 2018 DEPLOYED portal-3.0.0 onap onap-robot 3 Mon Dec 10 13:12:02 2018 DEPLOYED robot-3.0.0 onap onap-sdc 1 Mon Dec 10 00:35:00 2018 FAILED sdc-3.0.0 onap onap-sdnc 3 Mon Dec 10 13:12:04 2018 DEPLOYED sdnc-3.0.0 onap onap-so 3 Mon Dec 10 13:12:16 2018 DEPLOYED so-3.0.0 onap onap-uui 3 Mon Dec 10 13:12:24 2018 DEPLOYED uui-3.0.0 onap onap-vid 3 Mon Dec 10 13:12:27 2018 DEPLOYED vid-3.0.0 onap cannot use direct helm commands because of the known resource limit issue this plugin fixesubuntu@a-cd-master:~/oom/kubernetes$ sudo helm upgrade -i onap local/onap --namespace onap -f ../../dev.yaml --set aai.enabled=true Error: UPGRADE FAILED: ConfigMap "onap.v2" is invalid: []: Too long: must have at most 1048576 characters redeploying lists aai but does not bounce it ubuntu@a-cd-master:~/oom/kubernetes$ helm undeploy onap-aai --purge ubuntu@a-cd-master:~/oom/kubernetes$ helm undeploy onap-sdc --purge release "onap-sdc" deleted ubuntu@a-cd-master:~/oom/kubernetes$ sudo helm deploy onap local/onap --namespace onap -f ../../dev.yaml fetching local/onap release "onap" deployed release "onap-aaf" deployed release "onap-aai" deployed aai still missing ubuntu@a-cd-master:~$ helm list NAME REVISION UPDATED STATUS CHART NAMESPACE onap 2 Tue Dec 11 11:09:48 2018 DEPLOYED onap-3.0.0 onap onap-aaf 4 Tue Dec 11 11:09:49 2018 DEPLOYED aaf-3.0.0 onap onap-appc 4 Tue Dec 11 11:09:57 2018 DEPLOYED appc-3.0.0 onap ubuntu@a-cd-master:~/oom/kubernetes$ sudo helm deploy onap local/onap --namespace onap -f ../../dev.yaml --set aai.enabled=false fetching local/onap release "onap" deployed release "onap-aaf" deployed release "onap-appc" deployed release "onap-clamp" deployed this works as expected ubuntu@a-cd-master:~/oom/kubernetes$ helm undeploy onap-esr --purge release "onap-esr" deleted onap onap-esr-esr-gui-7946c6b68d-2lbhs 1/1 Terminating 0 22h onap onap-esr-esr-server-d96fcc7d7-7fnrd 2/2 Terminating 0 22h and the bounce ubuntu@a-cd-master:~/oom/kubernetes$ sudo helm deploy onap local/onap --namespace onap -f ../../dev.yaml onap onap-esr-esr-gui-7946c6b68d-nstwc 0/1 Running 0 2m onap onap-esr-esr-server-d96fcc7d7-tzwmv 0/2 ContainerCreating 0 2m trying so ubuntu@a-cd-cas:~$ kubectl get pods --all-namespaces | grep so onap onap-aai-aai-resources-f75d854cd-jmztq 2/2 Running 0 22h onap onap-oof-cmso-db-0 0/1 CrashLoopBackOff 244 22h onap onap-oof-oof-cmso-service-6454cf5994-4zjcs 0/1 Init:0/2 126 22h onap onap-oof-oof-has-solver-7d6cdb957f-2zgpf 0/1 Init:2/4 126 22h onap onap-so-so-75bb4b68bb-mdmk4 0/1 Terminating 230 22h onap onap-so-so-bpmn-infra-56f99599b6-88m7g 0/1 Terminating 229 22h onap onap-so-so-catalog-db-adapter-7994778fd9-k4clc 0/1 Terminating 231 22h onap onap-so-so-mariadb-795cf844d8-5pgl2 1/1 Terminating 0 22h onap onap-so-so-monitoring-b75b95f76-fpthv 1/1 Terminating 0 22h onap onap-so-so-openstack-adapter-78dc84b88f-sxxqn 0/1 Terminating 233 22h onap onap-so-so-request-db-adapter-64ff55cb79-5zc6z 1/1 Terminating 235 22h onap onap-so-so-sdc-controller-d89d44595-mmnbk 0/1 Terminating 231 22h onap onap-so-so-sdnc-adapter-8db65dfb8-l8ng7 1/1 Terminating 0 22h onap onap-so-so-vfc-adapter-6c4d54b6fd-5w48n 0/1 Terminating 234 22h ubuntu@a-cd-cas:~$ kubectl get pods --all-namespaces | grep so onap onap-aai-aai-resources-f75d854cd-jmztq 2/2 Running 0 22h onap onap-oof-cmso-db-0 0/1 CrashLoopBackOff 245 22h onap onap-oof-oof-cmso-service-6454cf5994-4zjcs 0/1 Init:0/2 126 22h onap onap-oof-oof-has-solver-7d6cdb957f-2zgpf 0/1 Init:2/4 126 22h ubuntu@a-cd-cas:~$ sudo helm deploy onap local/onap --namespace onap -f ../../dev.yaml fetching local/onap Error: open ../../dev.yaml: no such file or directory release "onap" deployed ^C ubuntu@a-cd-cas:~$ cd oom/kubernetes/ ubuntu@a-cd-cas:~/oom/kubernetes$ sudo helm deploy onap local/onap --namespace onap -f ../../dev.yaml fetching local/onap release "onap" deployed release "onap-aaf" deployed release "onap-aai" deployed release "onap-appc" deployed release "onap-clamp" deployed release "onap-cli" deployed release "onap-consul" deployed release "onap-dcaegen2" deployed release "onap-dmaap" deployed release "onap-esr" deployed release "onap-log" deployed release "onap-msb" deployed release "onap-multicloud" deployed release "onap-nbi" deployed release "onap-oof" deployed release "onap-policy" deployed release "onap-pomba" deployed release "onap-portal" deployed release "onap-robot" deployed release "onap-sdc" deployed release "onap-sdnc" deployed release "onap-so" deployed release "onap-uui" deployed release "onap-vid" deployed onap-sdc 1 Mon Dec 10 00:35:00 2018 FAILED sdc-3.0.0 onap ubuntu@a-cd-cas:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep so onap onap-aai-aai-resources-f75d854cd-jmztq 2/2 Running 0 22h onap onap-oof-cmso-db-0 0/1 CrashLoopBackOff 247 22h onap onap-oof-oof-cmso-service-6454cf5994-4zjcs 0/1 Init:0/2 127 22h onap onap-oof-oof-has-solver-7d6cdb957f-2zgpf 0/1 Init:2/4 127 22h onap onap-so-so-75bb4b68bb-bm7rs 0/1 Init:0/1 0 6m onap onap-so-so-bpmn-infra-56f99599b6-x7ds2 0/1 Init:0/1 0 6m onap onap-so-so-catalog-db-adapter-7994778fd9-shcpg 0/1 Init:0/1 0 6m onap onap-so-so-mariadb-795cf844d8-ktvc5 0/1 Running 0 6m onap onap-so-so-monitoring-b75b95f76-gw647 0/1 PodInitializing 0 6m onap onap-so-so-openstack-adapter-78dc84b88f-8f6ng 0/1 Init:0/1 0 6m onap onap-so-so-request-db-adapter-64ff55cb79-x5b8b 0/1 Init:0/1 0 6m onap onap-so-so-sdc-controller-d89d44595-b62s6 0/1 Init:0/1 0 6m onap onap-so-so-sdnc-adapter-8db65dfb8-km65w 1/1 Running 0 6m onap onap-so-so-vfc-adapter-6c4d54b6fd-2hw5s 0/1 Init:0/1 0 6m ubuntu@a-cd-cas:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep so onap onap-aai-aai-resources-f75d854cd-jmztq 2/2 Running 0 22h onap onap-oof-cmso-db-0 0/1 Running 248 22h onap onap-oof-oof-cmso-service-6454cf5994-4zjcs 0/1 Init:0/2 127 22h onap onap-oof-oof-has-solver-7d6cdb957f-2zgpf 0/1 Init:2/4 127 22h onap onap-so-so-75bb4b68bb-bm7rs 0/1 Init:0/1 0 12m onap onap-so-so-bpmn-infra-56f99599b6-x7ds2 0/1 Init:0/1 0 12m onap onap-so-so-catalog-db-adapter-7994778fd9-shcpg 0/1 Init:0/1 0 12m onap onap-so-so-mariadb-795cf844d8-ktvc5 0/1 Running 0 12m onap onap-so-so-monitoring-b75b95f76-gw647 1/1 Running 0 12m onap onap-so-so-openstack-adapter-78dc84b88f-8f6ng 0/1 Init:0/1 0 12m onap onap-so-so-request-db-adapter-64ff55cb79-x5b8b 0/1 Init:0/1 0 12m onap onap-so-so-sdc-controller-d89d44595-b62s6 0/1 Init:0/1 0 12m onap onap-so-so-sdnc-adapter-8db65dfb8-km65w 1/1 Running 0 12m onap onap-so-so-vfc-adapter-6c4d54b6fd-2hw5s 0/1 Init:0/1 0 12m ubuntu@a-cd-cas:~/oom/kubernetes$ ubuntu@a-cd-cas:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep so onap onap-aai-aai-resources-f75d854cd-jmztq 2/2 Running 0 1d onap onap-oof-cmso-db-0 0/1 Running 275 1d onap onap-oof-oof-cmso-service-6454cf5994-4zjcs 0/1 Init:0/2 142 1d onap onap-oof-oof-has-solver-7d6cdb957f-2zgpf 0/1 Init:2/4 142 1d onap onap-so-so-75bb4b68bb-bm7rs 0/1 CrashLoopBackOff 30 2h onap onap-so-so-bpmn-infra-56f99599b6-x7ds2 0/1 CrashLoopBackOff 30 2h onap onap-so-so-catalog-db-adapter-7994778fd9-shcpg 0/1 CrashLoopBackOff 30 2h onap onap-so-so-mariadb-795cf844d8-ktvc5 1/1 Running 0 2h onap onap-so-so-monitoring-b75b95f76-gw647 1/1 Running 0 2h onap onap-so-so-openstack-adapter-78dc84b88f-8f6ng 0/1 CrashLoopBackOff 30 2h onap onap-so-so-request-db-adapter-64ff55cb79-x5b8b 0/1 CrashLoopBackOff 30 2h onap onap-so-so-sdc-controller-d89d44595-b62s6 0/1 CrashLoopBackOff 30 2h onap onap-so-so-sdnc-adapter-8db65dfb8-km65w 1/1 Running 0 2h onap onap-so-so-vfc-adapter-6c4d54b6fd-2hw5s 0/1 CrashLoopBackOff 31 2h #ref my other identical azure instance is ok ubuntu@a-cd-master:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep so onap onap-aai-aai-resources-f75d854cd-l4spw 2/2 Running 0 1d onap onap-oof-cmso-db-0 1/1 Running 5 1d onap onap-oof-oof-cmso-service-6454cf5994-tpnmh 1/1 Running 0 1d onap onap-oof-oof-has-solver-7d6cdb957f-9x9bm 1/1 Running 0 1d onap onap-so-so-75bb4b68bb-v968h 1/1 Running 0 1d onap onap-so-so-bpmn-infra-56f99599b6-whw5p 1/1 Running 0 1d onap onap-so-so-catalog-db-adapter-7994778fd9-lrwlb 1/1 Running 0 1d onap onap-so-so-mariadb-795cf844d8-tvfq8 1/1 Running 0 1d onap onap-so-so-monitoring-b75b95f76-7c8x5 1/1 Running 0 1d onap onap-so-so-openstack-adapter-78dc84b88f-vqw84 1/1 Running 0 1d onap onap-so-so-request-db-adapter-64ff55cb79-k9fqk 1/1 Running 0 1d onap onap-so-so-sdc-controller-d89d44595-zkzw2 1/1 Running 0 1d onap onap-so-so-sdnc-adapter-8db65dfb8-cmvb5 1/1 Running 0 1d onap onap-so-so-vfc-adapter-6c4d54b6fd-6k9sg 1/1 Running 0 1d
Attachments
Issue Links
- is blocked by
-
OOM-1547
ONAP consistent deployment of pods in dependency order - to avoid failed random deployments
-
- Closed
-
- relates to
-
TSC-79 LF Nexus3 routing slowdown starting 20181217 - 80-100x slower download times totalling 120+ hours - using nexus3/4.onap.cloud proxy for now
-
- Closed
-
-
-
- Closed
-
-
LOG-898
ONAP Deployment Resiliency changes to deploy without POD failures
-
- Closed
-
-
LOG-899
ONAP vFW vFirewall Use Case validation - so we can use it for transaction tracing
-
- Closed
-
- mentioned in
-
Page Loading...
(1 mentioned in)