Uploaded image for project: 'Configuration Persistence Service'
  1. Configuration Persistence Service
  2. CPS-1126

CmHandle creation performance degradation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: High High
    • Kohn Release
    • Jakarta Release
    • NCMP
    • None

       

      Current behavior
       
      I saw there is an ongoing implementation (in frame of notification handling) around cmHandle registration which will change the way it worked before, I mean the newly introduced states and the way how NCMP gets the modules.
      I tried the CPS master as there are several bug corrections which is needed by us. During the cmHandle registration I saw that it is changed now.
       
      With previous versions we were able to register 10k cmhandles (in 4 threads) in 12 minutes.
      With the current CPS master we sent the 10k cmHandles  around 10 minutes to NCMP, so it is not faster than before. What I see is that NCMP starts getting modules in a background thread (seems to be one thread) and handling one cmhandle takes more seconds.
      Currenlty in 12 minutes only 150 cmHandles got available with ready state in NCMP, because getting modules and changing the state is quite slow this way. 

      Expected behavior
      With previous versions we were able to register 10k cmhandles (in 4 threads) in 12 minutes.

      Reproduction

      Register huge amount of cmhandles and try to search them.
      After 12 minutes there were only ~150cmHandles inready state
       

      CPS version

      https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=7914c8924723092345e8b4d829f15d2a3a5c72c8

       

      Collected logs
      NA

        4. cps1126_gap_outage_dbcontent.txt
          404 kB
          csaba Eder
        5. cps1126_gap_outage_discoservice.log
          18 kB
          csaba Eder
        6. cps1126_gap_outage_modelservice.log
          906 kB
          csaba Eder
        7. cps1126_gap_outage_ncmp.log
          396 kB
          csaba Eder
        10. EofException.txt
          14 kB
          csaba Eder
        11. eric-oss-ncmp-55564fbb89-bt8cq.log
          3.26 MB
          csaba Eder
        12. eric-oss-ncmp-7f48bd867f-q4f97.log
          2.57 MB
          csaba Eder
        13. ExceptionAndOutageReproduction_ncmp_log_0909.txt
          54 kB
          csaba Eder
        15. GapAndExceptionReproduction_ncmp_log_0909.txt
          272 kB
          csaba Eder
        16. GapReproductionDb.txt
          2 kB
          csaba Eder
        17. ncmp_database_fragment_state_20220715_patchset5.csv
          296 kB
          csaba Eder
        18. ncmp_discovery_nodebug.log
          0.8 kB
          csaba Eder
        20. ncmp_reprod_0719_ps6.log
          1.73 MB
          csaba Eder
        21. ncmp_reproduction_20220715_patchset5.log
          1.66 MB
          csaba Eder
        22. ncmp-sql02000.log
          439 kB
          Csaba Kocsis
        23. ncmp-sql02000-show-sql.log
          8.50 MB
          Csaba Kocsis
        24. ready_cmhandle_times.txt
          1.04 MB
          csaba Eder
        25. summary_100thread_100iteration.csv
          0.4 kB
          csaba Eder
        26. summary_30thread_300iteration.csv
          0.3 kB
          csaba Eder
        1.
        		 Performance Improvement: Postgres Pool Configuration Sub-task Closed Toine Siebelink
        2.
        		 Performance Improvement: Async Notification Pool Configuration Sub-task Closed Toine Siebelink
        3.
        		 Performance Improvement: Increase Connection Time-Out Sub-task Closed Priyank Maheshwari
        4.
        		 Performance Improvement: Save/update batch of CM-Handle States in one operation Sub-task Closed lukegleeson
        5.
        		 Performance Improvement: Updating (many) child datanodes/datanode tree into one SQL operation Sub-task Closed kissand
        6.
        		 Performance Improvement: Native query to more efficiently get all yangResourceIds Sub-task Closed Toine Siebelink
        7.
        		 Performance Improvement: Insert all yang resources for a schema set into one operation Sub-task Closed Toine Siebelink
        8.
        		 Performance Improvement: Detaching fragment entities early Sub-task Closed Priyank Maheshwari
        9.
        		 Performance Improvement: Decouple Temporal DB Notification from main thread Sub-task Closed Priyank Maheshwari
        10.
        		 Performance Improvement: Use Hazelcast to store Advised CM handles upon registration Sub-task Closed Sourabh Sourabh
        11.
        		 Performance Improvement: Configurable parallelism in Watchdog Executor Sub-task Closed Unassigned
        12.
        		 Performance Improvement: Use Priority Queue in Hazelcast (instead of concurrent map) Sub-task Closed Toine Siebelink
        13.
        		 Performance Improvement: Save cm handles in batch capability Sub-task Closed Priyank Maheshwari
        14.
        		 Performance Improvement: Watchdog Parallel execution Sub-task Closed Sourabh Sourabh
        15.
        		 Use state-save-batch capability (depends sub-task12, 13) Sub-task Closed Priyank Maheshwari
        16.
        		 Performance Improvement: Use save batches of cmhandles Sub-task Closed Sourabh Sourabh
        17.
        		 Handle partial failure when saving batches (cm handle create) Sub-task Closed Priyank Maheshwari
        18.
        		 Investigation SQL Error fetching Resource Ids by Module References Sub-task Closed Priyank Maheshwari

            sourabh_sourabh Sourabh Sourabh
            csaba.eder csaba Eder
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: