Uploaded image for project: 'Configuration Persistence Service'
  1. Configuration Persistence Service
  2. CPS-1126

CmHandle creation performance degradation


    • Icon: Bug Bug
    • Resolution: Done
    • Icon: High High
    • Kohn Release
    • Jakarta Release
    • NCMP
    • None


      Current behavior
      I saw there is an ongoing implementation (in frame of notification handling) around cmHandle registration which will change the way it worked before, I mean the newly introduced states and the way how NCMP gets the modules.
      I tried the CPS master as there are several bug corrections which is needed by us. During the cmHandle registration I saw that it is changed now.
      With previous versions we were able to register 10k cmhandles (in 4 threads) in 12 minutes.
      With the current CPS master we sent the 10k cmHandles  around 10 minutes to NCMP, so it is not faster than before. What I see is that NCMP starts getting modules in a background thread (seems to be one thread) and handling one cmhandle takes more seconds.
      Currenlty in 12 minutes only 150 cmHandles got available with ready state in NCMP, because getting modules and changing the state is quite slow this way. 

      Expected behavior

      With previous versions we were able to register 10k cmhandles (in 4 threads) in 12 minutes.


      Register huge amount of cmhandles and try to search them.
      After 12 minutes there were only ~150cmHandles inready state


      CPS version



      Collected logs

        1. cps1126_0912_ncmp_1.zip
          369 kB
        2. cps1126_0914_correction.zip
          513 kB
        3. cps1126_0914.zip
          577 kB
        4. cps1126_gap_outage_dbcontent.txt
          404 kB
        5. cps1126_gap_outage_discoservice.log
          18 kB
        6. cps1126_gap_outage_modelservice.log
          906 kB
        7. cps1126_gap_outage_ncmp.log
          396 kB
        8. cps1126_ncmp_0912_2.zip
          358 kB
        9. cps1126_ncmp_0912_3.zip
          357 kB
        10. EofException.txt
          14 kB
        11. eric-oss-ncmp-55564fbb89-bt8cq.log
          3.26 MB
        12. eric-oss-ncmp-7f48bd867f-q4f97.log
          2.57 MB
        13. ExceptionAndOutageReproduction_ncmp_log_0909.txt
          54 kB
        14. Gap5thread.zip
          236 kB
        15. GapAndExceptionReproduction_ncmp_log_0909.txt
          272 kB
        16. GapReproductionDb.txt
          2 kB
        17. ncmp_database_fragment_state_20220715_patchset5.csv
          296 kB
        18. ncmp_discovery_nodebug.log
          0.8 kB
        19. ncmp_discovery.zip
          5.02 MB
        20. ncmp_reprod_0719_ps6.log
          1.73 MB
        21. ncmp_reproduction_20220715_patchset5.log
          1.66 MB
        22. ncmp-sql02000.log
          439 kB
        23. ncmp-sql02000-show-sql.log
          8.50 MB
        24. ready_cmhandle_times.txt
          1.04 MB
        25. summary_100thread_100iteration.csv
          0.4 kB
        26. summary_30thread_300iteration.csv
          0.3 kB
        27. UnfinishedDiscovery5thread.zip
          244 kB
        Performance Improvement: Postgres Pool Configuration Sub-task Closed Toine Siebelink
        Performance Improvement: Async Notification Pool Configuration Sub-task Closed Toine Siebelink
        Performance Improvement: Increase Connection Time-Out Sub-task Closed Priyank Maheshwari
        Performance Improvement: Save/update batch of CM-Handle States in one operation Sub-task Closed lukegleeson
        Performance Improvement: Updating (many) child datanodes/datanode tree into one SQL operation Sub-task Closed kissand
        Performance Improvement: Native query to more efficiently get all yangResourceIds Sub-task Closed Toine Siebelink
        Performance Improvement: Insert all yang resources for a schema set into one operation Sub-task Closed Toine Siebelink
        Performance Improvement: Detaching fragment entities early Sub-task Closed Priyank Maheshwari
        Performance Improvement: Decouple Temporal DB Notification from main thread Sub-task Closed Priyank Maheshwari
        Performance Improvement: Use Hazelcast to store Advised CM handles upon registration Sub-task Closed Sourabh Sourabh
        Performance Improvement: Configurable parallelism in Watchdog Executor Sub-task Closed Unassigned
        Performance Improvement: Use Priority Queue in Hazelcast (instead of concurrent map) Sub-task Closed Toine Siebelink
        Performance Improvement: Save cm handles in batch capability Sub-task Closed Priyank Maheshwari
        Performance Improvement: Watchdog Parallel execution Sub-task Closed Sourabh Sourabh
        Use state-save-batch capability (depends sub-task12, 13) Sub-task Closed Priyank Maheshwari
        Performance Improvement: Use save batches of cmhandles Sub-task Closed Sourabh Sourabh
        Handle partial failure when saving batches (cm handle create) Sub-task Closed Priyank Maheshwari
        Investigation SQL Error fetching Resource Ids by Module References Sub-task Closed Priyank Maheshwari

            sourabh_sourabh Sourabh Sourabh
            csaba.eder csaba Eder
            0 Vote for this issue
            7 Start watching this issue