Uploaded image for project: 'Clamp'
  1. Clamp
  2. CLAMP-208

OOM Container Restarts

    XMLWordPrintable

Details

    • Task
    • Status: Closed
    • High
    • Resolution: Done
    • Beijing Release
    • Casablanca Release
    • CLAMP-R3 Sprint 1

    Description

      Over the past days, it has been noticed that the clamp container has restarted several times in the different environments. Additional information can be found on the following dashboard:

      http://onapci.org/grafana/d/kRvfoqKmz/oom-container-restarts?orgId=1

       
      Additional information from Gary on Aug 7th

      Generally speaking, a container restart means that:

      1. The liveness probe initial delay is too low, and
      2. The liveness probe, when triggered, is returning the wrong status while the container is still initializing (i.e. not dead).

       

      The default liveness probe in OOM is a TCP probe with a short (10 seconds?) initial delay.  This means that if the TCP port specified is not responding at 10s after container start, then the container will be killed and restarted.

       

      There are two ways to remedy this:

      1. Make the initial delay long enough to guarantee that by then the TCP port will be up, keeping in mind that ONAP might be running on arbitrarily slow hardware.
      2. Change the liveness probe to something else, perhaps to a shell script probe inside the container that can do more sophisticated checks like process status, etc.

       

      The deployments are done daily at midnight Pacific so anyone who is actively working a restart issue should have plenty of time to gather information and try out liveness probe changes.

      Please contact Gary Wu if you have any query

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sebdet Sébastien Determe
              Katel34 Catherine Lefevre
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: