Current behavior
With latest CPS master (86cda427e084b45c0f76d67f0eeebe74ee419ff2) we faced an issue during cmhandle registration.
Previously registration (reaching ADVISED to READY state) took around 8 minutes for 100 cmhandles
Now we register 100 cmHandles and it can not finish in our test timeout (20 minutes), it takes at least 30 minutes or longer.
How I see it is because strange state transitions and long time periods where no state transitions happening. Under strange state transitions I mean that some READY cmhandles going to LOCKED
There are search issues too during the tests. From the time when the first LOCKED states appearing we got 500 errors both on ncmp/v1/ch/id-searches and cps/api/v1/dataspaces/NFP-Operational/anchors searches:

{     "status": "500 INTERNAL_SERVER_ERROR",     "message": "Unable to find org.onap.cps.spi.entities.SchemaSetEntity with id 647",     "details": "Check logs for details." }

   

 

Expected behavior:
Successful ADVISED-READY state transitions for high amount of cmHandles, without hanging procedure and without unexpected state transitions, with same performance as before.
No SchemaSetEntity not found errors during the search requests.

 

Reproduction

• Use a near real model adapter which simulates some kind of processing/latency too to have bit longer response times.
• Register high amount of cmHandles. We registered only 100, but for reproduction it is better to have more, to have longer discovery procedure
• Check the database content in each minute and not the cmHandle states. There will be some cmHandles whose state will change from READY to LOCKED.
• After the first cmHandle with LOCKED state appears try to perform ncmp/v1/ch/id-searches and cps/api/v1/dataspaces/NFP-Operational/anchors search requests too, to see the SchemaSetEntity not found errors.
   

Tested CPS commits

• Version what we have in our current release is working fine:
    Default CMHandles to READY during upgrade 28/131428/2   https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=8b37702647755949165dc822116ed10dca5c1c2f
• Issue appearing first with:
    TTL for module and data sync 16/131416/8   https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=0165959c0218b4666a57d47d43a78fab1be6a894
• It got even worse and 100% reproducible from (maybe from older commits too, but I not tested every commit between the previous one and this)
    Merge "Improve code coverage"  https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=1377bf984e9d3ae97b1dc58c7d62ff1cafe60bb9
• Failing with latest commit too (2023.01.09)
    DocBuilderFactory one instance https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=86cda427e084b45c0f76d67f0eeebe74ee419ff2

 

Test environment:

• Real cluster
• No mocks in the system
• Postgres database
• Real model adapter, same version for every test
• Every test performed after clean installation

 

Collected logs:

ncmp-full.log
cmhandle_states_during_reproduction.txt