Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-286

Docker pulls involving protected nexus server secrets failing after Kubernetes 1.9.0 upgrade from 1.8.4

XMLWordPrintable

       Fix: Stay on Kubernetes 1.8 - not 1.9.0 - as there is a problem creating the secret for protected nexus repositories in the new version of Kubernetes - as it just came out of RC

      instead of 

      #curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

      use

      curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.4/bin/linux/amd64/kubectl

       see our raised

      https://github.com/kubernetes/kubernetes/issues/57528 

      This is failing
      create_registry_key() {
        cmd=`echo kubectl --namespace $1-$2 create secret docker-registry $3 --docker-server=$4 --docker-username=$5 --docker-password=$6 --docker-email=$7`
        eval ${cmd}
        check_return_code $cmd
      }

      note: 1.9.0 was released 6 days ago
      https://github.com/kubernetes/kubernetes/releases/tag/v1.9.0 
      ubuntu@ip-172-31-83-79:~$ kubectl versionClient Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.7-rancher1", GitCommit:"a1ea37c6f6d21f315a07631b17b9537881e1986a", GitTreeState:"clean", BuildDate:"2017-10-02T21:33:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

       Adjust

      https://wiki.onap.org/display/DW/ONAP+on+Kubernetes#ONAPonKubernetes-QuickstartInstallation

      https://wiki.onap.org/display/DW/3.+Set+Up+the+Undercloud#id-3.SetUptheUndercloud-Downloadkubectlontheserver

      https://github.com/obrienlabs/onap-root/blob/master/oom_rancher_setup_1.sh

       

       Issue

      On certain systems - and on my new physical nuc system - the pulls from nexus3 involving the refactored secret config are failing.

      Two other developers are also getting secured docker pulls failing

      On the Jenkins CD system we pass because the version of kubectl was below 1.9.0

      On my VM systems both the unsecured dockerhub and the secured nexus3 pulls work fine

      on this system out of the box - the secured pulls are failing

       AAI

       
      search-data-service
      onap-aai
      app: search-data-service0 / 19 hours
      nexus3.onap.org:10001/onap/search-data-service:v1.1.0
      docker.elastic.co/beats/filebeat:5.5.0more_vert
      Failed to pull image "nexus3.onap.org:10001/onap/search-data-service:v1.1.0": rpc error: code = 2 desc = unauthorized: authentication required
      Error syncing pod

      michael@oskub0:~$ sudo su -
      
      [sudo] password for michael:
      
      root@oskub0:~# kubectl get pods --all-namespaces -a
      
      NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
      
      kube-system   heapster-4285517626-hp1b8               1/1       Running            0          1d
      
      kube-system   kube-dns-638003847-7fkx0                3/3       Running            0          1d
      
      kube-system   kubernetes-dashboard-716739405-5j9j5    1/1       Running            0          1d
      
      kube-system   monitoring-grafana-2360823841-64wc4     1/1       Running            0          1d
      
      kube-system   monitoring-influxdb-2323019309-jnv7z    1/1       Running            0          1d
      
      kube-system   tiller-deploy-737598192-8tqcb           1/1       Running            0          1d
      
      onap          config                                  0/1       Completed          0          9h
      
      onap-aai      aai-resources-122881374-s5qd4           1/2       ImagePullBackOff   0          9h
      
      onap-aai      aai-service-749944520-8r3bq             0/1       Init:0/1           54         9h
      
      onap-aai      aai-traversal-3810717509-npnls          0/2       Init:0/1           54         9h
      
      onap-aai      data-router-3434587794-5kl0z            0/1       ImagePullBackOff   0          9h
      
      onap-aai      elasticsearch-622738319-6fggp           1/1       Running            0          9h
      
      onap-aai      hbase-1949550546-nxnmx                  1/1       Running            0          9h
      
      onap-aai      model-loader-service-4144225433-mnshh   1/2       ImagePullBackOff   0          9h
      
      onap-aai      search-data-service-378072033-3dvmv     1/2       ImagePullBackOff   0          9h
      
      onap-aai      sparky-be-3094577325-npj57              1/2       ImagePullBackOff   0          9h
      
      
      root@oskub0:~# kubectl -n onap-aai logs -f model-loader-service-4144225433-mnshh model-loader-service
      
      Error from server (BadRequest): container "model-loader-service" in pod "model-loader-service-4144225433-mnshh" is waiting to start: trying and failing to pull image

      If I pull manually outside of K8S - we are fine

       

       

       

            michaelobrien michaelobrien
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: