Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-2444

DCAE/Dmaap pods failing in OOM setup for SMO deployment

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: High High
    • None
    • El Alto Release, Frankfurt Release
    • None
    • Hide

      SUMMARY

      dcae and dmaap pods are in CLBO state and are unable to re-deploy

      OS / ENVIRONMENT

      • Kubernetes version:
        ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ kubectl version
        Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
        Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
      • Helm version:
        Client: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
        Server: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
      • Kubernetes mode of installation:
        <!-- add also configuration file if relevant -->
        <!-- please run:
        docker run -e DEPLOY_SCENARIO=k8s-test \
        -v <the kube config>:/root/.kube/config \
        opnfv/functest-kubernetes-healthcheck:latest
        -->
        <!-- and upload the result directory as a zip file -->
      • CNI Used for Kubernetes:ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ kubectl get nodes -n onap -o=wide
        NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
        onap-controller Ready controlplane,etcd 11d v1.15.11 10.12.6.3 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
        onap-k8s-1 Ready worker 11d v1.15.11 10.12.6.14 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
        onap-k8s-2 Ready worker 11d v1.15.11 10.12.6.20 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
        ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$
      • type of installation: 1 Controller, 2 worker nodes and 1 NFS node.
      • onap-discuss link:

      https://lists.onap.org/g/onap-discuss/message/21539

       

      OOM VERSION

      git clone b frankfurt http://gerrit.onap.org/r/oom -recurse-submodules oom_smo 

      SOP followed link: https://wiki.o-ran-sc.org/display/OAM/Deployment

      CONFIGURATION

      deploy_smo_bronze.yaml provided in the link which has been followed

      https://wiki.o-ran-sc.org/display/OAM/Deployment

      STEPS TO REPRODUCE

      <!-- line used to create helm charts -->

      cp -R /workspace/oom_smo/kubernetes/helm/plugins/ /.helm

      helm serve &

      helm repo add local http://127.0.0.1:8879

      helm repo list

      make all; make onap

      make onap -e SKIP_LINT=TRUE

      helm search onap -l

      <!-- line used to deploy ONAP -->

      helm install --debug local/onap -n jun24-smo --namespace onap -f ~/workspace/smo/deploy_smo_bronze.yaml --timeout 900

      <!-- add any necessary relevant command done -->

      tried deleting pods and tried undeploying the dcae and dmaap pods, but pods didn't get undeployed

      ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$

      ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ helm undeploy jun24-dcae --purge

      ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$

      ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ kubectl get pods -n onap -o=wide | grep jun24-dcae

      jun24-dcae-bootstrap-845967888-2pn2n                 0/1     Init:1/3                493        5d4h   10.42.1.12   onap-k8s-1   <none>           <none>

      jun24-dcae-cloudify-manager-5fdf98d4d8-8glm6         0/1     Init:CrashLoopBackOff   1357       5d1h   10.42.1.32   onap-k8s-1   <none>           <none>

      jun24-dcae-config-binding-service-6d9b9dbfcb-dw5wf   0/4     Init:1/2                1359       5d1h   10.42.1.31   onap-k8s-1   <none>           <none>

      jun24-dcae-dashboard-5654857ddc-w7j8f                0/2     Init:0/2                165        5d4h   10.42.1.18   onap-k8s-1   <none>           <none>

      jun24-dcae-dashboard-pg-primary-55bb79b655-vrlgp     1/1     Running                 0          5d4h   10.42.1.13   onap-k8s-1   <none>           <none>

      jun24-dcae-dashboard-pg-replica-7d5959499f-qvk42     1/1     Running                 0          5d4h   10.42.2.15   onap-k8s-2   <none>           <none>

      jun24-dcae-db-primary-6868ff6674-6hlf8               1/1     Running                 0          5d4h   10.42.2.23   onap-k8s-2   <none>           <none>

      jun24-dcae-db-replica-775d445d46-5p2gt               1/1     Running                 0          5d4h   10.42.2.19   onap-k8s-2   <none>           <none>

      jun24-dcae-deployment-handler-8cd7d9fdb-xqg4w        0/2     Init:0/3                165        5d4h   10.42.2.12   onap-k8s-2   <none>           <none>

      jun24-dcae-healthcheck-5b9976754c-wscsf              1/1     Running                 0          5d4h   10.42.1.16   onap-k8s-1   <none>           <none>

      jun24-dcae-inv-pg-primary-86f6488cf6-rjdd5           1/1     Running                 0          5d4h   10.42.1.17   onap-k8s-1   <none>           <none>

      jun24-dcae-inv-pg-replica-557dbb9bcb-jdjqb           1/1     Running                 0          5d4h   10.42.2.22   onap-k8s-2   <none>           <none>

      jun24-dcae-inventory-api-56d74bc77b-6pnks            0/2     Init:CrashLoopBackOff   1386       5d1h   10.42.2.36   onap-k8s-2   <none>           <none>

      jun24-dcae-mongo-0                                   1/1     Running                 0          5d4h   10.42.1.11   onap-k8s-1   <none>           <none>

      jun24-dcae-policy-handler-5dfcb7d7d6-fmcst           0/2     Init:0/3                165        5d4h   10.42.2.13   onap-k8s-2   <none>           <none>

      jun24-dcae-redis-0                                   1/1     Running                 0          5d4h   10.42.1.19   onap-k8s-1   <none>           <none>

      jun24-dcae-redis-1                                   1/1     Running                 0          5d4h   10.42.2.27   onap-k8s-2   <none>           <none>

      jun24-dcae-redis-2                                   1/1     Running                 0          5d4h   10.42.2.29   onap-k8s-2   <none>           <none>

      jun24-dcae-servicechange-handler-9bcb89dc9-vkc8c     0/1     Init:0/2                165        5d4h   10.42.2.6    onap-k8s-2   <none>           <none>

       

      EXPECTED RESULTS

      <!--- Describe what you expected to happen when running the steps above -->

      dcae and dmaap podsare failing and deleting and re-deploying doesn't help here

      helm undeploy jun24-dcae -- purge  ---> not deleting the pod to try re-deployment

      helm undeploy jun24-dmaap --purge ---> not deleting the pod to try re-deployment

       

      ACTUAL RESULTS

      <!--- Describe what actually happened. -->
      <!-- please run: docker run -v <the kube config>:/root/.kube/config -v \
      <result directory>:/var/lib/xtesting/results \ registry.gitlab.com/orange-opensource/lfn/onap/integration/xtesting/infra-healthcheck:latest
      -->
      <!-- and upload the result directory as a zip file -->
      <!-- cd where/your/oom/install is -->
      <!-- launch healthchecks: ./kubernetes/robot/ete-k8s.sh YOUR_DEPLOYMENT_NAME health -->
      <!-- and upload the result directory as a zip file -->
      <!-- it should be /dockerdata-nfs/onap/robot/logs/0000_ete_health/ (0000 must be the biggest number) -->

      Show
      SUMMARY dcae and dmaap pods are in CLBO state and are unable to re-deploy OS / ENVIRONMENT Kubernetes version: ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ kubectl version Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Helm version: Client: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"} Kubernetes mode of installation: <!-- add also configuration file if relevant --> <!-- please run: docker run -e DEPLOY_SCENARIO=k8s-test \ -v <the kube config>:/root/.kube/config \ opnfv/functest-kubernetes-healthcheck:latest --> <!-- and upload the result directory as a zip file --> CNI Used for Kubernetes:ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ kubectl get nodes -n onap -o=wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME onap-controller Ready controlplane,etcd 11d v1.15.11 10.12.6.3 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5 onap-k8s-1 Ready worker 11d v1.15.11 10.12.6.14 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5 onap-k8s-2 Ready worker 11d v1.15.11 10.12.6.20 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5 ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ type of installation: 1 Controller, 2 worker nodes and 1 NFS node. onap-discuss link: https://lists.onap.org/g/onap-discuss/message/21539   OOM VERSION git clone b frankfurt http: //gerrit .onap.org /r/oom   -recurse-submodules oom_smo   SOP followed link:  https://wiki.o-ran-sc.org/display/OAM/Deployment CONFIGURATION deploy_smo_bronze.yaml provided in the link which has been followed https://wiki.o-ran-sc.org/display/OAM/Deployment STEPS TO REPRODUCE <!-- line used to create helm charts --> cp   -R /workspace/oom_smo/kubernetes/helm/plugins/   /.helm helm serve & helm repo add local http://127.0.0.1:8879 helm repo list make all; make onap make onap -e SKIP_LINT=TRUE helm search onap -l <!-- line used to deploy ONAP --> helm install --debug local/onap -n jun24-smo --namespace onap -f ~/workspace/smo/deploy_smo_bronze.yaml --timeout 900 <!-- add any necessary relevant command done --> tried deleting pods and tried undeploying the dcae and dmaap pods, but pods didn't get undeployed ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ helm undeploy jun24-dcae --purge ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ ubuntu@onap-controller:~/workspace/oom_smo/kubernetes$ kubectl get pods -n onap -o=wide | grep jun24-dcae jun24-dcae-bootstrap-845967888-2pn2n                 0/1     Init:1/3                493        5d4h   10.42.1.12   onap-k8s-1   <none>           <none> jun24- dcae-cloudify-manager -5fdf98d4d8-8glm6         0/1     Init:CrashLoopBackOff    1357       5d1h   10.42.1.32   onap-k8s-1   <none>           <none> jun24-dcae-config-binding-service-6d9b9dbfcb-dw5wf   0/4     Init:1/2                1359       5d1h   10.42.1.31   onap-k8s-1   <none>           <none> jun24-dcae-dashboard-5654857ddc-w7j8f                0/2     Init:0/2                165        5d4h   10.42.1.18   onap-k8s-1   <none>           <none> jun24-dcae-dashboard-pg-primary-55bb79b655-vrlgp     1/1     Running                 0          5d4h   10.42.1.13   onap-k8s-1   <none>           <none> jun24-dcae-dashboard-pg-replica-7d5959499f-qvk42     1/1     Running                 0          5d4h   10.42.2.15   onap-k8s-2   <none>           <none> jun24-dcae-db-primary-6868ff6674-6hlf8               1/1     Running                 0          5d4h   10.42.2.23   onap-k8s-2   <none>           <none> jun24-dcae-db-replica-775d445d46-5p2gt               1/1     Running                 0          5d4h   10.42.2.19   onap-k8s-2   <none>           <none> jun24-dcae-deployment-handler-8cd7d9fdb-xqg4w        0/2     Init:0/3                165        5d4h   10.42.2.12   onap-k8s-2   <none>           <none> jun24-dcae-healthcheck-5b9976754c-wscsf              1/1     Running                 0          5d4h   10.42.1.16   onap-k8s-1   <none>           <none> jun24-dcae-inv-pg-primary-86f6488cf6-rjdd5           1/1     Running                 0          5d4h   10.42.1.17   onap-k8s-1   <none>           <none> jun24-dcae-inv-pg-replica-557dbb9bcb-jdjqb           1/1     Running                 0          5d4h   10.42.2.22   onap-k8s-2   <none>           <none> jun24- dcae -inventory-api-56d74bc77b-6pnks            0/2     Init:CrashLoopBackOff    1386       5d1h   10.42.2.36   onap-k8s-2   <none>           <none> jun24-dcae-mongo-0                                   1/1     Running                 0          5d4h   10.42.1.11   onap-k8s-1   <none>           <none> jun24-dcae-policy-handler-5dfcb7d7d6-fmcst           0/2     Init:0/3                165        5d4h   10.42.2.13   onap-k8s-2   <none>           <none> jun24-dcae-redis-0                                   1/1     Running                 0          5d4h   10.42.1.19   onap-k8s-1   <none>           <none> jun24-dcae-redis-1                                   1/1     Running                 0          5d4h   10.42.2.27   onap-k8s-2   <none>           <none> jun24-dcae-redis-2                                   1/1     Running                 0          5d4h   10.42.2.29   onap-k8s-2   <none>           <none> jun24-dcae-servicechange-handler-9bcb89dc9-vkc8c     0/1     Init:0/2                165        5d4h   10.42.2.6    onap-k8s-2   <none>           <none>   EXPECTED RESULTS <!--- Describe what you expected to happen when running the steps above --> dcae and dmaap podsare failing and deleting and re-deploying doesn't help here helm undeploy jun24-dcae -- purge  ---> not deleting the pod to try re-deployment helm undeploy jun24-dmaap --purge ---> not deleting the pod to try re-deployment   ACTUAL RESULTS <!--- Describe what actually happened. --> <!-- please run: docker run -v <the kube config>:/root/.kube/config -v \ <result directory>:/var/lib/xtesting/results \ registry.gitlab.com/orange-opensource/lfn/onap/integration/xtesting/infra-healthcheck:latest --> <!-- and upload the result directory as a zip file --> <!-- cd where/your/oom/install is --> <!-- launch healthchecks: ./kubernetes/robot/ete-k8s.sh YOUR_DEPLOYMENT_NAME health --> <!-- and upload the result directory as a zip file --> <!-- it should be /dockerdata-nfs/onap/robot/logs/0000_ete_health/ (0000 must be the biggest number) -->

          sdesbure sdesbure
          himeshshukla himeshshukla
          Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

            Created:
            Updated:
            Resolved: