1. Some of the modules are removed from the DB at cmhandle removal, but the modules of last removed cmhandle type is always kept
   1. Register two cmhandles with different modulesets
   2. Remove first cmhandle, all modules kept
   3. Remove second, the modules of first one are removed (the ones which are not used by second)
   4. It seems that orphaned modules are removed from the DB before cmhandle removal and not after that.
   5. It is not causing problem if modulesetTag is not set because than modules downloaded every time.
   6. Not a new issue in 3.4.1, we had it in 3.4.0 and I guess in older versions too, but It is triggering other problems when we use moduleSetTag, see below.
      No symptoms but would like to improve this behavior -> separate Jira. @Sourabh to create - CPS-2031
2. If previously known modules are removed from the DB NCMP still thinking based on the cache content that newly registered cmhandle modules are known, that is causing READY cmhandles with no modules
   1. Register first cmhandle again, no modules downloaded because already known moduleSetTag, but the cmhandle wnet to ready without modules.
   2. Register second  cmhandle again, no modules downloaded because already known moduleSetTag, but as the modules were already there everything went fine.
   3. So if we register a cmhandle with a moduleSetTag which was known before but the modules were removed from the DB in the meantime, the new cmhandle will be moved to ready without modules.
   4. Here some inconsistency can be between DB and cached data
      Fixed after latest Patch
3. If I register cmhandles after a pod restart it downloads the previously known modules again, but in schema_set_yang_resources table we have duplicated entries
   1. If I restart the pod (eg an upgrade with recreate policy happens) it seems that the cached moduleSetData is partly lost, because if I register a cmhandle with moduleSetTag which was known before than it starts downloading and cmhandle goes to ready. 
   2. However in this case schema_set_yang_resources table contains the cmhandle-module pairs, but every cmhandle-module pairs appearing twice (like coming both from cache and download) It is not really causing functional problem I think, but need to think about it.
      Fixed after latest Patch
4. Upgrade procedure
   1. We can upgrade theModuleSetTag only which is not enough, sure it will trigger a model request but for that we need all the other updated properties because model adapter is working based on that.
   2. Just updating the moduleSetTag is not enough now because if private properties are not changed model adapter will give back the old modules if it is not cached and ncmp is requesting it.
   3. Even if we change modelDmiPlugin behavior to work based on moduleSetTag we might need the moduleSetTag sent to the modelDmiPlugin at module-set request. How I see currently it is not sent.
      Private property (which should have some information as in ModuleSetTag) is not updated and this causes problem in model adapter. Separate Jira for new requirement. @Csaba to create new Jira
5. Cmhandle version upgrade is failing on DB level
   1. If I send in a simple moduleSetTag update the model adapter will respond with same model set we have now because the properties not changed.
   2. It even can be that ENM contains the same module set for two different versions which is causing failing upgrade in NCMP.
   3. {"reason": "MODULE_UPGRADE_FAILED", "details": "Upgrade to ModuleSetTag: 20.Q2-R6A06 Attempt #1 failed: Already defined exception"}
   4. {"cm-handle-state": "LOCKED", "last-update-time": "2024-01-11T05:34:37.150+0000"}
   5. "ERROR: duplicate key value violates unique constraint \"schema_set_name_dataspace_id_key\"\n  Detail: Key (name, dataspace_id)=(725E4E790691461E84FA030AFFFB500F, 1) already exists.
      CPS Team to test with Overlapping module Sets.
6. Once a module upgrade went to MODULE_UPGRADE_FAILED new module upgrade wont be started
   1. If I try to rollback moduleSetTag to the previous version it writing back the moduleSetTag, but it does not initiate module update because it stuck in MODULE_UPGRADE_FAILED
   2. I was waiting for 10 minutes but nothing happened.- Still have "Upgrade to ModuleSetTag: 20.Q2-R6A06 Attempt #1 failed" however moduleSetTag is written back to 20.Q1-R88A07
   3. {"reason": "MODULE_UPGRADE_FAILED", "details": "Upgrade to ModuleSetTag: 20.Q2-R6A06 Attempt #1 failed: Already defined exception"}
   4. {"cm-handle-state": "LOCKED", "last-update-time": "2024-01-11T05:34:37.150+0000"}

      Related to #5, retry mechanism not working (maybe fix #6 before #5)

7. Cmhandle version upgrade to a version which is already known by NCMP
1. CmHandle went to ready state. No module download happened (Sure because those are downloaded)
2. No modules bound to that cmHandle even after it went to ready state Maybe only after latest fix. sourabh_sourabh to supply (REST API) method to check all modules on a given cm handle. @Csaba to add more detail after more testing (with and without latest fix)

A/C

Test Upgrade with Overlapping modules

Tag1 : M1 & M2
Tag2 : M1 & M3

Tag3 : M1 & M2 (Identical to Tag1)

Test 1.1 - Testing adding 2 cm handles with overlapping module sets
Add CM-H1, Tag1
Add CM-H2, Tag2

Test 1.2 - Upgrading 1 cm handle with overlapping module sets
Add CM-H1, Tag1
Upgrade CM-H1, Tag 2

Test 1.3 - Testing adding 2 cm handles with identical module sets (Different Tags)
Add CM-H1, Tag1
Add CM-H2, Tag3

Test 1.4 - Upgrading cm handle with identical module sets (Different Tags)
Add CM-H1, Tag1
Upgrade CM-H1, Tag3

============================

Delete Scenarios Test 2

Add CM-H1, Tag1
Add CM-H2, Tag2
Delete CM-H2
Upgrade CM-H1 to Tag 2

Test 3

Demo retry mechanism by stopping the DMI-Plugin shall work