Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-813

ConfigMap issues with regression in new K8S 1.8.9 in Rancher 1.6.14 up from 1.8.5 since 20180317 1.6.15 release - revert to 1.6.12 with 1.8.3

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Highest Highest
    • Casablanca Release
    • Beijing Release
    • None

      20180321: not an ONAP or OOM issue - is a regression in Kubernetes 1.8.9 being fixed

      Noticed this busted behaviour specific to Azure:ubuntu over the weekend - thought it was just me but getting more reports
      20170318:1700 update: Gary Wu has noticed that the kubernetes server for some reason is now running 1.8.9 instead of 1.8.5
      20180320: beijing is busted across all azure/aws vms that run the latest (since friday) version of 1.6.14
      Testing 1.6.13 (also updated to 1.8.9) and 1.6.14+kubectl 1.8.9
      1.6.12 under 1.12 and k8s 1.8.3 is ok, 1.6.15 under 1.9.2 is not

      state
      AWS, amsterdam, ubuntu user = ?
      AWS, beijing, ubuntu user = OK (20180319) - only for older 1.6.14 install
      AWS, beijing, root user = ?
      Azure, amsterdam, ubuntu user = OK (20180319) http://jenkins.onap.cloud/job/oom_azure_deployment/
      Azure, beijing, ubuntu user = BUSTED (20180319)
      Azure, beijing, root user = in progress now (ete 75 min)

      AWS is fine only because it is still running a 1.6.14 from before friday (AWS CD is static, Azure CD is dynamic)
      http://jenkins.onap.info/job/oom-cd/2410/console
      Azure has issues on master not amsterdam on the ubuntu user
      http://jenkins.onap.cloud/job/oom-cd-master/13/console

      master
      pending containers=39
      onap          aaf-6c64db8fdd-fgwxb                           0/1       Running                           0          27m
      onap          aai-data-router-6fbb8695d4-9s6w2               0/1       CreateContainerConfigError        0          27m
      onap          aai-elasticsearch-7f66545fdf-q7gnh             0/1       CreateContainerConfigError        0          27m
      onap          aai-model-loader-service-7768db4744-lj9bg      0/2       CreateContainerConfigError        0          27m
      onap          aai-resources-9f95b9b6d-qrhs5                  0/2       CreateContainerConfigError        0          27m
      onap          aai-search-data-service-99dff479c-fr8bh        0/2       CreateContainerConfigError        0          27m
      onap          aai-service-5698ddc455-npsm6                   0/1       Init:0/1                          2          27m
      onap          aai-sparky-be-57bd9944b5-cmqvc                 0/2       CreateContainerConfigError        0          27m
      onap          aai-traversal-df4b45c4-sjtlx                   0/2       Init:0/1                          0          27m
      onap          appc-67c6b9d477-n64mk                          0/2       CreateContainerConfigError        0          27m
      onap          appc-dgbuilder-68c68ff84b-x6dst                0/1       Init:0/1                          0          27m
      onap          clamp-6889598c4-76mww                          0/1       Init:0/1                          2          27m
      onap          clamp-mariadb-78c46967b8-2w922                 0/1       CreateContainerConfigError        0          27m
      onap          log-elasticsearch-6ff5b5459d-2zq2b             0/1       CreateContainerConfigError        0          27m
      onap          log-kibana-54c978c5fc-457gb                    0/1       Init:0/1                          2          27m
      onap          log-logstash-5f6fbc4dff-t2hh9                  0/1       Init:0/1                          2          27m
      onap          mso-555464596b-t5fc2                           0/2       Init:0/1                          2          28m
      onap          mso-mariadb-5448666ccc-kddh6                   0/1       CreateContainerConfigError        0          28m
      onap          multicloud-framework-57687dc8c-nf7pk           0/2       CreateContainerConfigError        0          27m
      onap          multicloud-vio-5bfb9f68db-g6j7h                0/2       CreateContainerConfigError        0          27m
      onap          policy-brmsgw-5f445cfcfb-wzb88                 0/1       Init:0/1                          2          27m
      onap          policy-drools-5b67c475d6-pv6kt                 0/2       CreateContainerConfigError        0          27m
      onap          policy-pap-79577c6947-fhfxb                    0/2       Init:CrashLoopBackOff             8          27m
      onap          policy-pdp-7d5c76bf8d-st7js                    0/2       Init:0/1                          2          27m
      onap          portal-apps-7ddfc4b6bd-g7nhk                   0/2       Init:CreateContainerConfigError   0          27m
      onap          portal-vnc-7dcbf79f66-7c6p6                    0/1       Init:0/4                          2          27m
      onap          portal-widgets-6979b47c48-5kr86                0/1       CreateContainerConfigError        0          27m
      onap          robot-f6d55cc87-t2wgd                          0/1       CreateContainerConfigError        0          27m
      onap          sdc-fe-6d4b87c978-2v5x2                        0/2       CreateContainerConfigError        0          27m
      onap          sdnc-0                                         0/2       Init:0/1                          2          28m
      onap          sdnc-dbhost-0                                  0/2       Pending                           0          28m
      onap          sdnc-dgbuilder-794d686f78-296zf                0/1       Init:0/1                          2          28m
      onap          sdnc-dmaap-listener-8595c8f6c-vgzxt            0/1       Init:0/1                          2          28m
      onap          sdnc-portal-69b79b6646-p4x8k                   0/1       Init:0/1                          2          28m
      onap          sdnc-ueb-listener-6897f6dd55-fq9j5             0/1       Init:0/1                          2          28m
      onap          vfc-ztevnfmdriver-fcf4ddf68-65pb5              0/1       ImagePullBackOff                  0          27m
      onap          vid-mariadb-6788c598fb-kbfnw                   0/1       CreateContainerConfigError        0          28m
      onap          vid-server-87d5d87cf-9rbx4                     0/2       Init:0/1                          2          28m
      onap          vnfsdk-refrepo-55f544c5f5-9b6jj                0/1       ImagePullBackOff                  0          27m
      

      http://beijing.onap.cloud:8880/r/projects/1a7/kubernetes-dashboard:9090/#!/pod?namespace=_all

      Checking amserdam on azure running ubuntu user = OK
      root@ons-auto-201803191109z:/var/lib/waagent/custom-script/download/0# tail -f stdout
      root@ons-auto-201803191109z:/var/lib/waagent/custom-script/download/0/oom# git status
      On branch amsterdam
      Your branch is up-to-date with 'origin/amsterdam'.
      4 pending > 0 at the 62th 15 sec interval
      onap-aaf aaf-1993711932-3lnwd 0/1 Running 0 19m
      onap-vnfsdk refrepo-1924147637-1x10v 0/1 ErrImagePull 0 19m
      3 pending > 0 at the 63th 15 sec interval

            michaelobrien michaelobrien
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: