-
Story
-
Resolution: Done
-
Medium
-
None
-
None
-
DCAE R2 Sprint RC1
The observed increase of docker stats MEM USAGE caused a concern of the possible memory leak in policy-handler.The policy-handler shows ever growing MEM USAGE and MEM % values reported by docker stats command. The observed growth is of order of magnitude of about 1GB per day.
Policy-handler has a feature of periodic catchup that is automatically being performed every 20 minutes. The growth of docker stats MEM USAGE correlates and is proportional to the amount of data passed through the policy-handler between the communication with the policy-engine and deployment-handler. The big policy config size of DROOL rules under CLAMP scope (about 50KB each) is the biggest contributor to the size of the messaging.
Further research and collection of the memory stats show that RSS data reported inside the docker container does not change that much after reaching about 50MB. Small fluctuation of about +/-(2-5)MB around 50MB is observed and is attributed to the periodic catchup activity of the policy-handler followed by the return of the heap memory back to OS.
Here is the sample log.
On Wednesday the docker stats shows MEM USAGE = 456.1 MB
Wed May 16 17:36:07 EDT 2018
$ docker stats --no-stream policy_handler
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
policy_handler 0.96% 456.1 MB / 33.74 GB 1.35% 27.13 MB / 1.537 MB 0 B / 0 B
related memory inside the docker container shows RSS of only 49.9MiB
"mem_info": {"pss": "46.4MiB", "uss": "43.3MiB", "swap": "0B", "rss": "49.9MiB"}
Next day - the docker stats shows the dramatically increased MEM USAGE = 2.466 GB
Thu May 17 08:10:04 EDT 2018
$ docker stats --no-stream policy_handler
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
policy_handler 1.20% 2.466 GB / 33.74 GB 7.31% 163.8 MB / 9.256 MB 16.38 kB / 0 B
related memory inside the docker container shows almost unchanged RSS of only 51.4MiB
"mem_info": {"pss": "48.0MiB", "uss": "44.8MiB", "swap": "0B", "rss": "51.4MiB"}
$ ps afxv
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
3722 ? Ss 0:00 0 966 20965 3364 0.0 bash
3728 ? R+ 0:00 0 84 10691 844 0.0 _ ps afxv
1 ? Ss 0:00 0 966 20797 3128 0.0 /bin/bash ./run_policy
26 ? Sl 11:42 0 2 2090533 51460 0.1 python -m policyhandl
Further research showed that docker stats measures the MEM USAGE as the sum of RSS + cache that is not an indication of the memory leak !
https://github.com/moby/moby/issues/10824
For more detailed memory stats of the docker container the proper API end point of (/containers/(id)/stats) should be used
https://docs.docker.com/engine/reference/commandline/stats/#extended-description
For instance,
$ curl http://<<docker-host>>:2376/containers/policy_handler/stats?stream=false
returns json like this
{ ..."memory_stats": {"usage": 3063111680,..."stats": {... "cache": 3017359360,..."rss": 45752320,...}}}
where the reported usage (about 3GB) = cache (about 3GB) + rss (about 45MB)
Bottom line - do not assume that the output of docker stats command indicates the memory leaking