Uploaded image for project: 'Service Orchestrator'
  1. Service Orchestrator
  2. SO-1419

Impossible to delete service instances with the SO (VNF-API)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • High
    • Resolution: Done
    • Casablanca Release, Dublin Release, Casablanca Maintenance Release
    • Dublin Release

    Description

      when onboardind then instantiating a VNF using the old VNF-API (old but not deprecated), everything is almost fine

      • Onboarding OK: from the upload of the heat files to the model distribution in the different modules.
      • Instantiation OK: creation of service instance => creation of the VNF instance(s) => preload(s) => creation of the VF module(s)

      But there is resource cleaning problem

      delete of the VF module(s) => delete of the VNF instance(s) => delete of the preload(s) OK

      but delete of the service instance never ends and keeps "IN_PROGRESS"

      it seems that with the old API, it is impossible to delete any service instance (except if we do it directly in the DB, which is not the nominal case...)

      A race condition in the SO is suspected

      As the instances are not cleaned, it keeps on generating traffic and using resources

      ------------------------------------------------------------------------------------------

      according to the SO logs

      • creation then deletion of the VNF COMPLETED
        • --------------------------> create VNF
          • /app/logs/apih # grep 54d8314d-8b76-4a76-ad8b-270a39a4164b audit.log 
          • 2019-01-22T13:00:33.981Z|2019-01-22T13:00:35.444Z|95de6d44-c262-4c38-b602-d2abc3682f57||http-nio-8080-exec-7||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
          • 2019-01-22T*13:01:58.084Z*|2019-01-22T13:01:59.570Z|5c97e518-c9df-4424-ba08-dd0adb3743f5||http-nio-8080-exec-8||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules/c83f0095-bbc3-4327-a33e-ed12b1e3d26c||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
        • --------------------------> clean
          • 2019-01-22T*13:02:45.542Z*|2019-01-22T13:02:47.011Z|9d764278-bfce-489c-be9d-6bf3f324c5e3||http-nio-8080-exec-2||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting
      • Service Instance creation and deletion
        • --------------------------> create service instance
          • 2019-01-22T*13:00:11.867Z*|2019-01-22T13:00:13.181Z|8c54f59d-0bf4-4f96-be11-71f02e694294||http-nio-8080-exec-2||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
          • 2019-01-22T*13:00:33.981Z*|2019-01-22T13:00:35.444Z|95de6d44-c262-4c38-b602-d2abc3682f57||http-nio-8080-exec-7||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
          • 2019-01-22T*13:01:58.084Z*|2019-01-22T13:01:59.570Z|5c97e518-c9df-4424-ba08-dd0adb3743f5||http-nio-8080-exec-8||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules/c83f0095-bbc3-4327-a33e-ed12b1e3d26c||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
        • --------------------------> clean
          • 2019-01-22T*13:02:45.542Z*|2019-01-22T13:02:47.011Z|9d764278-bfce-489c-be9d-6bf3f324c5e3||http-nio-8080-exec-2||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
          • 2019-01-22T*13:02:57.974Z*|2019-01-22T13:02:59.266Z|02ad82a2-198b-4e43-87c1-ddfde03c0b42||http-nio-8080-exec-4||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
        • --------------------------> manual attempt with the VID some minutes later
          • 2019-01-22T*13:24:45.*698Z|2019-01-22T13:24:45.764Z|a4be4eb1-7387-417d-b0a7-889c5d0e4ca9||http-nio-8080-exec-10||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||*ERROR|*409||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|2|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.

      From BPMN logs, It looks OK.

       in the debug.log of the SO (see full logs attached) around 13:02:58, I can see the deletion state moving from PENDING (13:02:58) to IN_PROGRESS (13:02:59)

      but it stays in IN_PROGRESS and is never COMPLETED

      that is why when I try manually to delete the instance through the VID I got

       

      2019-01-22T*13:24:45.763Z*|a4be4eb1-7387-417d-b0a7-889c5d0e4ca9|http-nio-8080-exec-10|onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||||ERROR|||Error: Locked instance - This service (6536a8ee-a7fc-4443-a722-6c0918ba361d) already has a request being worked with a status of IN_PROGRESS (RequestId - 02ad82a2-198b-4e43-87c1-ddfde03c0b42). The existing request must finish or be cleaned up before proceeding.

      org.onap.so.apihandlerinfra.exceptions.DuplicateRequestException: Error: Locked instance - This service (6536a8ee-a7fc-4443-a722-6c0918ba361d) already has a request being worked with a status of IN_PROGRESS (RequestId - 02ad82a2-198b-4e43-87c1-ddfde03c0b42). The existing request must finish or be cleaned up before proceeding.

      2019-01-22T*13:24:45.763Z*|a4be4eb1-7387-417d-b0a7-889c5d0e4ca9|http-nio-8080-exec-10|onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||||ERROR|400|Error: Locked instance - This service (6536a8ee-a7fc-4443-a722-6c0918ba361d) already has a request being worked with a status of IN_PROGRESS (RequestId - 02ad82a2-198b-4e43-87c1-ddfde03c0b42). The existing request must finish or be cleaned up before proceeding.|APIH_DUPLICATE_FOUND

      And on the VID, at the end I can see lots of resources as no  instantation can be fully deleted (VNF and the VF module have been deleted but not the service instance)

      ---------------------------------------------------------------------------------------------------

      With my testing tool, once I call the SO to delete the VF module, the VNF or the service instance, I check directly in the AA&I that the references have been effectively cleaned.

      it is OK for the Module and the VNF but the service reference keeps in the A&AI

      However we do not get message in the VID saying that the VNF has been cleaned

      I got the same behavior using my test framework (https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests) or performing all the operations manually with the VID.

       

      Note: I just created an instance (but no VNF, VF module) then delete it and I do not have the problem.

       

      Attachments

        1. debug_so + clienttest.zip
          258 kB
        2. error_SO-UI.png
          error_SO-UI.png
          57 kB
        3. logs.tar.gz
          1.41 MB
        4. so_master_staging_14022019.txt
          83 kB

        Issue Links

          blocks

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. TSC-101 Dublin kubernetes deployment stability tracking

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. SDC-2096 SDC-FE installation fails

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mrichomme Morgan Richomme
              mrichomme Morgan Richomme
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: