Uploaded image for project: 'Data Collection, Analytics, and Events'
  1. Data Collection, Analytics, and Events
  2. DCAEGEN2-514

docker stats memory values are misleading

XMLWordPrintable

      The observed increase of docker stats MEM USAGE caused a concern of the possible memory leak in policy-handler.The policy-handler shows ever growing MEM USAGE and MEM % values reported by docker stats command. The observed growth is of order of magnitude of about 1GB per day. 

       

      Policy-handler has a feature of periodic catchup that is automatically being performed every 20 minutes.  The growth of docker stats MEM USAGE correlates and is proportional to the amount of data passed through the policy-handler between the communication with the policy-engine and deployment-handler.  The big policy config size of DROOL rules under CLAMP scope (about 50KB each) is the biggest contributor to the size of the messaging.  

       

      Further research and collection of the memory stats show that RSS data reported inside the docker container does not change that much after reaching about 50MB.  Small fluctuation of about +/-(2-5)MB around 50MB is observed and is attributed to the periodic catchup activity of the policy-handler followed by the return of the heap memory back to OS. 

       


      Here is the sample log. 

      On Wednesday the docker stats shows MEM USAGE = 456.1 MB
      Wed May 16 17:36:07 EDT 2018
      $ docker stats --no-stream policy_handler
      CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
      policy_handler 0.96% 456.1 MB / 33.74 GB 1.35% 27.13 MB / 1.537 MB 0 B / 0 B

      related memory inside the docker container shows RSS of only 49.9MiB
      "mem_info": {"pss": "46.4MiB", "uss": "43.3MiB", "swap": "0B", "rss": "49.9MiB"}


      Next day - the docker stats shows the dramatically increased MEM USAGE = 2.466 GB            
       Thu May 17 08:10:04 EDT 2018
      $ docker stats --no-stream policy_handler
      CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
       policy_handler 1.20% 2.466 GB / 33.74 GB 7.31% 163.8 MB / 9.256 MB 16.38 kB / 0 B

      related memory inside the docker container shows almost unchanged RSS of only 51.4MiB
        "mem_info": {"pss": "48.0MiB", "uss": "44.8MiB", "swap": "0B", "rss": "51.4MiB"}
      $ ps afxv
      PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND

      3722 ? Ss 0:00 0 966 20965 3364 0.0 bash
      3728 ? R+ 0:00 0 84 10691 844 0.0 _ ps afxv
      1 ? Ss 0:00 0 966 20797 3128 0.0 /bin/bash ./run_policy
      26 ? Sl 11:42 0 2 2090533 51460 0.1 python -m policyhandl


      Further research showed that docker stats measures the MEM USAGE as the sum of RSS + cache that is not an indication of the memory leak !

      https://github.com/moby/moby/issues/10824

      For more detailed memory stats of the docker container the proper API end point of (/containers/(id)/stats) should be used

      https://docs.docker.com/engine/reference/commandline/stats/#extended-description

      For instance,
      $ curl http://<<docker-host>>:2376/containers/policy_handler/stats?stream=false
      returns json like this
      { ..."memory_stats": {"usage": 3063111680,..."stats": {... "cache": 3017359360,..."rss": 45752320,...}}}
      where the reported usage (about 3GB) = cache (about 3GB) + rss (about 45MB)


      Bottom line - do not assume that the output of docker stats command indicates the memory leaking
       

            alex_sh alex_sh
            alex_sh alex_sh
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: