Description

Previously I tested the solution proposal of CPS-1933 with one ncmp instance and most of the functionalities were working fine, however I missed to check the DB content if all the cmHandles has all the modules and all the nodes our discovered in our topology discovery.

I started to test the 3.3.9 release with automated tests and I saw issues.

So I took again the https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=4266fbbaad4ab3741accf1b3911278835ca19795  commit and cherrypicked https://gerrit.onap.org/r/c/cps/+/136379 correction on top of that and we have issues around topology discovery, because the module data content is not correct in NCMP db.

What I see:

• Testing with one ncmp instance, we have clean database, no cmhandle registration before
  • All cmhandles went to READY
  • It seems that with one instance ncmp stores some of the yang resources but not all of them and not all of the cmhandles has all of the yang resources bound
  • This causing that our topology discovery is not complete, because the module data in NCMP is inconsistent
• Testing with one ncmp instance, if we already have some modules stored in ncmp DB but not all of them
  • All cmhandles went to READY
  • Ncmp does not store additional yang resources at new discovery, we have just the already downloaded yang resources in ncmp DB
  • This causing that our topology discovery is not complete, because the module data in NCMP is inconsistent
• Testing with two instances
  • All cmhandles went to READY
  • Ncmp does not store any yang resources at all and no yang resources bound to cmHandles
  • This causing that our topology discovery fully failing because it can not get any modules for any cmhandles
  • Our automatic tests in our nightly pipelines are running with two instances and we saw the same from 3/3 executions
  • Logs ncmp_nomodules_2instance_debug_2.log ncmp_nomodules_2instance_debug_1.log

In prometheus metrics I see that cmhandles went to ready:

There were some yang processing on ncmp side so it got the modules, but if I check the DB those are not stored and when I filter the cmhandles on modules I got empty response

Based on these we are still blocked and can not take the latest 3.3.9 release

It is still https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=4266fbbaad4ab3741accf1b3911278835ca19795 commit which is causing it, because with the commit before that it is working fine.

In the meantime I was able turn on the debug logging on application yaml level.
I collected a log from a cmhandle discovery but I can not attach here because of sensitive data
What is strange that when the debug logging was turned on than it was able to download all the modules and bound them to cmhandles
So I guess that a timing/threading/caching issue could be here

Expected behavior:
Have all of the yang resources in the database and bound to correct cmhandles