Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-904

OOM problems bringing up components and passing healthchecks

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: Beijing Release
    • Fix Version/s: Beijing Release
    • Labels:

      Description

      This is an ongoing activity to clean up problems related to instantiation of ONAP components. 

      The following issues have been found and resolved:

      1 , modify nexus image version : https://gerrit.onap.org/r/#/c/42091/  (merged) 
      2. miscellaneous problems standing up drools cluster: https://gerrit.onap.org/r/#/c/42097/  (merged)

      3. mariadb and pap initialization: https://gerrit.onap.org/r/43223 (in progress)

         The following changes are also necessary, being worked by Mahendra Raghuwanshi:
        https://gerrit.onap.org/r/#/c/42701/

      This changes is also necessary to allow pap to come up.   It has to be included in a review.    Requesting Mahendra to add it to his, to avoid merge conflicts.   This is the section that has to be commented out here:

      root@k8s:~/oom/kubernetes/policy# git diff templates/deployment.yaml
      diff --git a/kubernetes/policy/templates/deployment.yaml b/kubernetes/policy/templates/deployment.yaml
      index 88b3de3..40ebcbc 100644
      — a/kubernetes/policy/templates/deployment.yaml
      +++ b/kubernetes/policy/templates/deployment.yaml
      @@ -45,19 +45,19 @@ spec:
      image: "{{ .Values.global.readinessRepository }}/{{ .Values.global.readinessImage }}"
      imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
      name: {{ include "common.name" . }}-readiness

      • - command: ["/bin/bash", "-c", "echo $POD_IP > /config/ip_addr.txt"]
      • env:
      • - name: POD_IP
      • valueFrom:
      • fieldRef:
      • apiVersion: v1
      • fieldPath: status.podIP
      • volumeMounts:
      • - mountPath: /config/
      • name: pe
      • image: {{ .Values.global.ubuntuImage }}
      • imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
      • name: generate-ip-addr-txt
        + # - command: ["/bin/bash", "-c", "echo $POD_IP > /config/ip_addr.txt"]
        + # env:
        + # - name: POD_IP
        + # valueFrom:
        + # fieldRef:
        + # apiVersion: v1
        + # fieldPath: status.podIP
        + # volumeMounts:
        + # - mountPath: /config/
        + # name: pe
        + # image: {{ .Values.global.ubuntuImage }}
        + # imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
        + # name: generate-ip-addr-txt

        With all previous changes from all code reviews merged and in progress, healthchecks (from within the pdp-d pass).   See:

      NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
      oom jhh-brmsgw-896c4f648-7mtkp 1/1 Running 0 13m 10.42.87.30 k8s
      oom jhh-dmaap-64d67cf8bf-s5fd5 0/1 Init:0/1 1 13m 10.42.222.125 k8s
      oom jhh-drools-0 1/1 Running 0 13m 10.42.43.145 k8s
      oom jhh-drools-1 1/1 Running 0 12m 10.42.201.28 k8s
      oom jhh-drools-2 1/1 Running 0 11m 10.42.103.36 k8s
      oom jhh-drools-3 1/1 Running 0 11m 10.42.138.57 k8s
      oom jhh-global-kafka-57df6996df-cpg6k 0/1 ImagePullBackOff 0 13m 10.42.55.50 k8s
      oom jhh-log-elasticsearch-6ddc6d5bc4-z5sdg 1/1 Running 0 13m 10.42.40.196 k8s
      oom jhh-log-kibana-6dc79464b7-jgwfz 1/1 Running 0 13m 10.42.164.177 k8s
      oom jhh-log-logstash-5bd987797c-s9fx6 1/1 Running 0 13m 10.42.98.123 k8s
      oom jhh-nexus-58b86f9485-zb2vg 1/1 Running 0 13m 10.42.89.150 k8s
      oom jhh-pap-795b744c5f-d7wpf 2/2 Running 0 13m 10.42.92.228 k8s
      oom jhh-pdp-0 2/2 Running 0 13m 10.42.95.11 k8s
      oom jhh-pdp-1 2/2 Running 0 11m 10.42.96.124 k8s

      {{}}

      root@k8s:~/oom/kubernetes/policy/templates# kubectl exec -it jhh-drools-0 -n oom bash
      policy@jhh-drools-0:/tmp/policy-install$ stty rows 65 cols 140
      policy@jhh-drools-0:/tmp/policy-install$ policy status

      [drools-pdp-controllers]
      L []: Policy Management (pid 1584) is running
      1 cron jobs installed.

      [features]
      name version status
      ---- ------- ------
      eelf 1.2.0-SNAPSHOT disabled
      test-transaction 1.2.0-SNAPSHOT disabled
      session-persistence 1.2.0-SNAPSHOT disabled
      state-management 1.2.0-SNAPSHOT disabled
      active-standby-management 1.2.0-SNAPSHOT disabled
      healthcheck 1.2.0-SNAPSHOT enabled

      The healthcheck configuration:

      policy@jhh-drools-0:/tmp/policy-install$ curl --silent --user 'healthcheck:zb!XztG34' -X GET http://localhost:6969/healthcheck/configuration | python -m json.tool
      {
      "alive": true,
      "clients": [
      {
      "alive": true,
      "basePath": "pap/test",
      "baseUrl": "http://jhh-pap:9091/pap/test",
      "hostname": "jhh-pap",
      "https": false,
      "name": "PAP",
      "port": 9091,
      "selfSignedCerts": false,
      "userName": "testpap"
      },
      {
      "alive": true,
      "basePath": "pdp/test",
      "baseUrl": "http://jhh-pdp:8081/pdp/test",
      "hostname": "jhh-pdp",
      "https": false,
      "name": "PDP",
      "port": 8081,
      "selfSignedCerts": false,
      "userName": "testpdp"
      }
      ],
      "servers": [
      {
      "alive": true,
      "host": "0.0.0.0",
      "name": "HEALTHCHECK",
      "port": 6969,
      "user": "healthcheck"
      }
      ]
      }

       

      The actual healthcheck:

      policy@jhh-drools-0:/tmp/policy-install$ curl --silent --user 'healthcheck:zb!XztG34' -X GET http://localhost:6969/healthcheck | python -m json.tool
      {
      "details": [
      {
      "code": 200,
      "healthy": true,
      "message": "alive",
      "name": "PDP-D",
      "url": "self"
      },
      {
      "code": 200,
      "healthy": true,
      "message": "",
      "name": "PAP",
      "url": "http://jhh-pap:9091/pap/test"
      },
      {
      "code": 200,
      "healthy": true,
      "message": "",
      "name": "PDP",
      "url": "http://jhh-pdp:8081/pdp/test"
      }
      ],
      "healthy": true
      }

       

       

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              • Assignee:
                jhh Jorge Hernandez
                Reporter:
                jhh Jorge Hernandez
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: