Background:

We faced a big performance degradation during a manual 80k (20k node) ENM's CM discovery.
According to the reporter this process usually takes 10-11 minutes on the used environment, but for the given run it was around 2 hours.

From the logs we saw a lot of the following errors thrown by NCMP's DB:

ERROR: no unpinned buffers available

As I checked this error is usually appears if the DB runs out of its configured shared buffer memory, the default value for this configuration is 128 MB and this is what we are also using.

Logs:

IDUN-113119.tgz (I couldn't make them to be under 10 MB, let me know if there is an access issue)

• DDP logs:
  • NCMP microservice log
  • NCMP DB log
  • ENM Discovery Adapter log
• K8S logs:
  • NCMP microservice log

Faulty version:

CPS 3.4.6

Reproduction:

According to the reporter they did the manual 80k ENM discovery on a fresh EIC deployment, and there was nothing else running parallely.

Expected behaviour:

80k ENM discovery finishes in 10-11 minutes