Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-543

SDNC adjust docker pullPolicy to IfNotPresent to speed up initial deployment slowdown introduced by SDNC-163

    Details

    • Type: Task
    • Status: Closed
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: Beijing Release
    • Labels:
    • Sprint:
      OOM Sprint 9 - Beijing freeze

      Description

      Issue: during initial SDNC OOM creation we will time out using the new docker images introduced in SDNC-163
      This will occur on a clean machine with no previous docker pullls
      On a 2nd deployment we are ok on a pre-pulled vm

      Underlying issue is that pod startup is sensitive to docker pull times - this is a separate jira

      Marco,
      Hi, there were changes done recently in SDNC to enable clustering (notice the new sdnc-0 pod name) https://jira.onap.org/browse/SDNC-163
      – when this went in I had the same issue bringing up the pod tree – starting with dbhost. However it looks related to pulling the docker image where the readiness check times out before the retries are exhausted – itself a separate issue - OOM-544. If you delete/recreate SDNC you will be OK – also when running again - like we do on the CD server because by that point you have already loaded the docker image.
      Part of this does not make sense to me yet because if the dynamic prepull has pulled the docker layers for a particular image - and later during pod startup the image is pulled again – then I should be seeing the same slowdown on the 2nd startup of SDNC with cached docker images – need to add logs from a clean startup on a clean VM – TODO.

      So, restarting SDNC is not a proper workaround – I originally closed the following bug – but after talking with Rahul of the SDNC team – a proposal to not pull the image again (for even though the generated docker pre-pull works) – Kubernetes will still pull the image – slowing down the startup just enough to fail the first time we pull the new SDNC images in values.yaml – need to verify this theory with a test of the change tomorrow.

      Previously closed
      https://jira.onap.org/browse/OOM-537
      created
      https://jira.onap.org/browse/OOM-543

      However, this also is not a proper workaround – I will raise another OOM jira that addresses the fact that our pod startup needs to handle “any” docker pull time – or be configurable. Because the underlying issue to all of this is the fact that we are exposed to startup failures on long docker pulls.

      https://jira.onap.org/browse/OOM-544

      /michael

      From: PLATANIA, MARCO (MARCO) platania@research.att.com
      Sent: Wednesday, January 3, 2018 14:43
      To: Yury Novitsky <Yury.Novitsky@Amdocs.com>; alexis.de_talhouet@bell.ca; Michael O'Brien <Frank.Obrien@amdocs.com>; Roger Maitland <Roger.Maitland@amdocs.com>; Mike Elliott <Mike.Elliott@amdocs.com>
      Subject: Re: VID/SDC error in ONAP K8S

      Hi Yury and All,

      I actually got past that error but issues between AAI and MSO happened, so I decided to rebuild ONAP from scratch. Since this morning, all my attempts failed due to incomplete SDNC installation. Please see below. I tried to reinstall ONAP a few times or bounce SDNC containers, but it didn’t work so far.

      onap-sdnc nfs-provisioner-4051743034-vqvmm 1/1 Running 0 41m
      onap-sdnc sdnc-0 0/2 Init:0/1 4 41m
      onap-sdnc sdnc-dbhost-0 0/2 Pending 0 41m
      onap-sdnc sdnc-dgbuilder-4267203648-6hqnk 0/1 Init:0/1 4 41m
      onap-sdnc sdnc-portal-2558294154-bqsvb 0/1 Init:0/1 4 41m

      Have you seen this issue before? If so, how can I get past it?

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              • Assignee:
                michaelobrien Michael O'Brien
                Reporter:
                michaelobrien Michael O'Brien
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: