Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-805

createAll.bash on single component on clean k8s cluster fails to pull nexus3 images -a option does not work

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Medium Medium
    • Casablanca Release
    • Beijing Release
    • None
    • None

      workaround: use HELM_APPS in setenv.bash to bring up a single or set of components

      running
      ./createAll.bash -n onap
      and using the HELM_APPS variable works

      But running
      ./createAll.bash -n onap -a aai
      will fail for nexus3 images (dockerhub pulls fine)

      See question from Kiran on
      https://wiki.onap.org/display/DW/ONAP+on+Kubernetes?refresh=1521067241237&refresh=1521067706865&refresh=1521068454088&refresh=1521069135810&focusedCommentId=25441419&refresh=1521069869672#comment-25441419

      Kiran Kamineni
      ./createall.sh seems to have a bug where creating a single component on a clean Rancher/k8s installation will fail with image pulling errors.
      
      This is because, create_registry_key and create_service_account are not called when single components are installed.
      
      I might be doing something wrong here though. Any suggestions?
      
      ReplyLikeabout 2 hours ago
       User icon: michaelobrien
      Michael O'Brien
      Testing on an Azure VM right now
      
      
      see oom_rancher_install.sh details in a comment below - hit the 32767 char limit in jira
      
      root@cd-s:~# # run parts of cd.sh manually to prep oom for a single component
      root@cd-s:~# sysctl -w vm.max_map_count=262144
      vm.max_map_count = 262144
      root@cd-s:~# git clone -b master http://gerrit.onap.org/r/oom
      Cloning into 'oom'...
      remote: Counting objects: 902, done
      remote: Finding sources: 100% (24/24)
      remote: Total 8008 (delta 0), reused 8001 (delta 0)
      Receiving objects: 100% (8008/8008), 19.24 MiB | 11.50 MiB/s, done.
      Resolving deltas: 100% (3633/3633), done.
      Checking connectivity... done.
      root@cd-s:~# source oom/kubernetes/oneclick/setenv.bash
      root@cd-s:~# cp onap-parameters.yaml oom/kubernetes/config
      root@cd-s:~# cd oom/kubernetes/config
      root@cd-s:~/oom/kubernetes/config# ./createConfig.sh -n onap
      **** Creating configuration for ONAP instance: onap
      namespace "onap" created
      NAME:   onap-config
      LAST DEPLOYED: Wed Mar 14 23:06:39 2018
      NAMESPACE: onap
      STATUS: DEPLOYED
      RESOURCES:
      ==> v1/ConfigMap
      NAME                   DATA  AGE
      global-onap-configmap  15    0s
      ==> v1/Pod
      NAME    READY  STATUS             RESTARTS  AGE
      config  0/1    ContainerCreating  0         0s
      **** Done ****
       
       
      root@cd-s:~/oom/kubernetes/config# kubectl get pods --all-namespaces -a
      NAMESPACE     NAME                                   READY     STATUS      RESTARTS   AGE
      kube-system   heapster-76b8cd7b5-5fxgl               1/1       Running     0          10m
      kube-system   kube-dns-5d7b4487c9-b6ddl              3/3       Running     0          10m
      kube-system   kubernetes-dashboard-f9577fffd-bqfhz   1/1       Running     0          10m
      kube-system   monitoring-grafana-997796fcf-q8wcd     1/1       Running     0          10m
      kube-system   monitoring-influxdb-56fdcd96b-c656h    1/1       Running     0          10m
      kube-system   tiller-deploy-cc96d4f6b-h2szs          1/1       Running     0          10m
      onap          config                                 0/1       Completed   0          1m
       
      root@cd-s:~/oom/kubernetes/config# cd ../oneclick/
      root@cd-s:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a robot
      ********** Creating instance 1 of ONAP with port range 30200 and 30399
      ********** Creating ONAP:
      ********** Creating deployments for robot **********
      Creating deployments and services **********
      NAME:   onap-robot
      LAST DEPLOYED: Wed Mar 14 23:08:33 2018
      NAMESPACE: onap
      STATUS: DEPLOYED
      RESOURCES:
      ==> v1beta1/Deployment
      NAME   DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
      robot  1        1        1           0          1s
      ==> v1/ConfigMap
      NAME                              DATA  AGE
      robot-eteshare-configmap          4     1s
      robot-resources-configmap         3     1s
      lighttpd-authorization-configmap  1     1s
      ==> v1/Service
      NAME   CLUSTER-IP   EXTERNAL-IP  PORT(S)       AGE
      robot  10.43.15.78  <nodes>      88:30209/TCP  1s
      **** Done ****
      root@cd-s:~/oom/kubernetes/oneclick# watch kubectl get pods --all-namespaces
       
       
      onap          robot-f6d55cc87-fbsqt                  0/1       ErrImagePull   0          3m
      later
      onap          robot-f6d55cc87-fbsqt                  0/1       ImagePullBackOff   0          3m
       
       
      does not look good so far - also a "docker images" shows no nexus3 images yet
       
       
      AAI has no dependencies and will also come up by itself - testing
      root@cd-s:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a aai
      ********** Creating instance 1 of ONAP with port range 30200 and 30399
      ********** Creating ONAP:
      ********** Creating deployments for aai **********
      Creating deployments and services **********
      NAME:   onap-aai
      LAST DEPLOYED: Wed Mar 14 23:13:21 2018
      NAMESPACE: onap
      STATUS: DEPLOYED
      RESOURCES:
      ==> v1/Secret
      NAME                             TYPE    DATA  AGE
      aai-data-router-secret           Opaque  3     5s
      aai-model-loader-secret          Opaque  1     5s
      aai-search-data-keystone-secret  Opaque  1     5s
      aai-sparky-be-secret             Opaque  3     5s
      ==> v1/ConfigMap
      NAME                                   DATA  AGE
      aai-deployment-configmap               1     5s
      aai-filebeat-configmap                 1     5s
      aai-traversal-log-configmap            1     5s
      aai-chef-config-configmap              1     5s
      aai-resources-environments-configmap   3     5s
      aai-resources-log-configmap            1     5s
      aai-data-router-dynamic-configmap      2     5s
      aai-data-router-prop-configmap         1     5s
      aai-data-router-model-v11-configmap    1     5s
      aai-data-router-model-v10-configmap    1     5s
      aai-data-router-model-v8-configmap     1     4s
      aai-data-router-model-v9-configmap     1     4s
      aai-elasticsearch-configmap            1     4s
      aai-model-loader-prop-configmap        1     4s
      aai-model-loader-log-configmap         1     4s
      aai-search-policy-configmap            1     4s
      aai-search-data-configmap              3     4s
      aai-search-data-service-log-configmap  1     4s
      aai-sparky-be-portal-configmap         2     4s
      aai-sparky-be-model-configmap          1     4s
      aai-sparky-be-log-configmap            1     4s
      aai-sparky-be-configmap                6     4s
      ==> v1/Service
      NAME                  CLUSTER-IP     EXTERNAL-IP  PORT(S)                                                            AGE
      aai-traversal         None           <none>       8446/TCP,5005/TCP                                                  4s
      aai-elasticsearch     None           <none>       9200/TCP                                                           4s
      gremlin               None           <none>       8182/TCP                                                           4s
      hbase                 None           <none>       2181/TCP,8080/TCP,8085/TCP,9090/TCP,16000/TCP,16010/TCP,16201/TCP  4s
      model-loader-service  10.43.27.176   <nodes>      8443:30229/TCP,8080:30210/TCP                                      4s
      aai-resources         None           <none>       8447/TCP,5005/TCP                                                  4s
      search-data-service   None           <none>       9509/TCP                                                           4s
      aai-service           10.43.255.254  <nodes>      8443:30233/TCP,8080:30232/TCP                                      4s
      sparky-be             None           <none>       9517/TCP                                                           4s
      ==> v1beta1/Deployment
      NAME                      DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
      aai-service               1        1        1           0          4s
      aai-resources             1        1        1           0          4s
      aai-traversal             1        1        1           0          4s
      aai-data-router           1        1        1           0          4s
      aai-elasticsearch         1        1        1           0          4s
      aai-hbase                 1        1        1           0          4s
      aai-model-loader-service  1        1        1           0          4s
      aai-search-data-service   1        1        1           0          4s
      aai-sparky-be             1        1        1           0          4s
      **** Done ****
       
       
      This is a bug - only dockerhub images download (nexus3 ones don't) - without as you state @Kiran
      I have (about 2 months ago raised a clean ubuntu VM in VMWare on my laptop running only aai - so this looks like a recent regression
      onap          aai-data-router-6fbb8695d4-xphpv            0/1       ErrImagePull       0          2m
      onap          aai-elasticsearch-7f66545fdf-rmmrr          1/1       Running            0          2m
      onap          aai-hbase-5bcb5884d4-547s9                  1/1       Running            0          2m
      onap          aai-model-loader-service-7768db4744-2j7wz   1/2       ErrImagePull       0          2m
      onap          aai-resources-58d4966887-zcl5r              1/2       ErrImagePull       0          2m
      onap          aai-search-data-service-99dff479c-t8qfd     1/2       ImagePullBackOff   0          2m
      onap          aai-service-5698ddc455-7mz5p                0/1       Init:0/1           0          2m
      onap          aai-sparky-be-57bd9944b5-zwrk4              1/2       ImagePullBackOff   0          2m
      onap          aai-traversal-557f8fc78d-xm7gt              0/2       Init:0/1           0          2m
      onap          robot-f6d55cc87-fbsqt                       0/1       ImagePullBackOff   0          7m
       
       
      the images downloaded are non-nexus3 - this is why the config, aai-hbase, aai-elasticsearch containers are running
       
       
      root@cd-s:~/oom/kubernetes/oneclick# docker images
      REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
      oomk8s/config-init                                     2.0.0-SNAPSHOT      8a0891e2f059        10 days ago         703 MB
      oomk8s/readiness-check                                 1.1.0               c0cce3bbe795        3 weeks ago         585 MB
      aaionap/hbase                                          1.2.0               3568a85848ab        6 months ago        432 MB
      docker.elastic.co/beats/filebeat                       5.5.0               b61327632415        8 months ago        271 MB
      elasticsearch                                          2.4.1               8e3cf79edcc3        16 months ago       346 MB
       
      root@cd-s:~/oom/kubernetes/oneclick#
       
       
      good one - raising jira
      Every 2.0s: kubectl get pods --all-namespaces                                                                                                         Wed Mar 14 23:22:57 2018
       
      NAMESPACE     NAME                                        READY     STATUS             RESTARTS   AGE
      kube-system   heapster-76b8cd7b5-5fxgl                    1/1       Running            0          25m
      kube-system   kube-dns-5d7b4487c9-b6ddl                   3/3       Running            0          25m
      kube-system   kubernetes-dashboard-f9577fffd-bqfhz        1/1       Running            0          25m
      kube-system   monitoring-grafana-997796fcf-q8wcd          1/1       Running            0          25m
      kube-system   monitoring-influxdb-56fdcd96b-c656h         1/1       Running            0          25m
      kube-system   tiller-deploy-cc96d4f6b-h2szs               1/1       Running            0          25m
      onap          aai-data-router-6fbb8695d4-xphpv            0/1       ImagePullBackOff   0          9m
      onap          aai-elasticsearch-7f66545fdf-rmmrr          1/1       Running            0          9m
      onap          aai-hbase-5bcb5884d4-547s9                  1/1       Running            0          9m
      onap          aai-model-loader-service-7768db4744-2j7wz   1/2       ImagePullBackOff   0          9m
      onap          aai-resources-58d4966887-zcl5r              1/2       ImagePullBackOff   0          9m
      onap          aai-search-data-service-99dff479c-t8qfd     1/2       ImagePullBackOff   0          9m
      onap          aai-service-5698ddc455-7mz5p                0/1       Init:0/1           0          9m
      onap          aai-sparky-be-57bd9944b5-zwrk4              1/2       ImagePullBackOff   0          9m
      onap          aai-traversal-557f8fc78d-xm7gt              0/2       Init:0/1           0          9m
      onap          robot-f6d55cc87-fbsqt                       0/1       ImagePullBackOff   0          14m
      

            david.sauvageau david.sauvageau
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: