Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-428

Add log container healthcheck to mark failed creations - see OOM-427

    Details

      Description

      There was a change done to optimize the yamls - the log containers failed to create - but were not caught by the CD or the healtcheck because we are missing a call to one or all of the ELK containers

       

       The OOM CD job picked this up at 18:07 UTC 20171114 http://jenkins.onap.info/job/oom-cd/93/console

      18:07:15 ********** Creating deployments for log ********** 
      18:07:15 
      18:07:15 Creating namespace **********
      18:07:15 namespace "onap-log" created
      18:07:15 
      18:07:15 Creating service account **********
      18:07:15 clusterrolebinding "onap-log-admin-binding" created
      18:07:15 
      18:07:15 Creating registry secret **********
      18:07:15 secret "onap-docker-registry-key" created
      18:07:15 
      18:07:15 Creating deployments and services **********
      18:07:15 Error: parse error in "log/templates/logstash-deployment.yaml": template: log/templates/logstash-deployment.yaml:83: function "tpl" not defined
      18:07:15 The command helm returned with error code 1 
      18:07:15 wait for all pods up for 10 min
      
      A rollback of the 4 changes fixes the pods
      ubuntu@ip-172-31-25-96:~/oom/kubernetes/oneclick$ kubectl get pods --all-namespaces | grep log
      onap-log              elasticsearch-1942187295-kkg1n           1/1       Running            0          10m
      onap-log              kibana-3372627750-drmlf                  1/1       Running            0          10m
      onap-log              logstash-1708188010-1fsgl                1/1       Running            0          10m

      see kibana query

      http://kibana.onap.info:5601/app/kibana#/discover?_g=(refreshInterval:(display:'5%20seconds',pause:!f,section:1,value:5000),time:(from:now-7d,mode:quick,to:now))&_a=(columns:!(_source),index:AV-vxHzpUcDKD8zJ1zbn,interval:auto,query:(query_string:(query:'message:%20%22parse%20error%22')),sort:!('@timestamp',desc)) 

       

        Attachments

        1. loghealth1.PNG
          loghealth1.PNG
          61 kB
        2. loghealth2.PNG
          loghealth2.PNG
          47 kB
        3. loghealth3.PNG
          loghealth3.PNG
          24 kB

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              • Assignee:
                mayankg2703 Mayank Gupta
                Reporter:
                michaelobrien Michael O'Brien
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: