when onboardind then instantiating a VNF using the old VNF-API (old but not deprecated), everything is almost fine

• Onboarding OK: from the upload of the heat files to the model distribution in the different modules.
• Instantiation OK: creation of service instance => creation of the VNF instance(s) => preload(s) => creation of the VF module(s)

But there is resource cleaning problem

delete of the VF module(s) => delete of the VNF instance(s) => delete of the preload(s) OK

but delete of the service instance never ends and keeps "IN_PROGRESS"

it seems that with the old API, it is impossible to delete any service instance (except if we do it directly in the DB, which is not the nominal case...)

A race condition in the SO is suspected

As the instances are not cleaned, it keeps on generating traffic and using resources

------------------------------------------------------------------------------------------

according to the SO logs

• creation then deletion of the VNF COMPLETED
  • --------------------------> create VNF
    • /app/logs/apih # grep 54d8314d-8b76-4a76-ad8b-270a39a4164b audit.log 
    • 2019-01-22T13:00:33.981Z|2019-01-22T13:00:35.444Z|95de6d44-c262-4c38-b602-d2abc3682f57||http-nio-8080-exec-7||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
    • 2019-01-22T*13:01:58.084Z*|2019-01-22T13:01:59.570Z|5c97e518-c9df-4424-ba08-dd0adb3743f5||http-nio-8080-exec-8||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules/c83f0095-bbc3-4327-a33e-ed12b1e3d26c||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
  • --------------------------> clean
    • 2019-01-22T*13:02:45.542Z*|2019-01-22T13:02:47.011Z|9d764278-bfce-489c-be9d-6bf3f324c5e3||http-nio-8080-exec-2||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting
• Service Instance creation and deletion

• • --------------------------> create service instance
    • 2019-01-22T*13:00:11.867Z*|2019-01-22T13:00:13.181Z|8c54f59d-0bf4-4f96-be11-71f02e694294||http-nio-8080-exec-2||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
    • 2019-01-22T*13:00:33.981Z*|2019-01-22T13:00:35.444Z|95de6d44-c262-4c38-b602-d2abc3682f57||http-nio-8080-exec-7||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
    • 2019-01-22T*13:01:58.084Z*|2019-01-22T13:01:59.570Z|5c97e518-c9df-4424-ba08-dd0adb3743f5||http-nio-8080-exec-8||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b/vfModules/c83f0095-bbc3-4327-a33e-ed12b1e3d26c||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
  • --------------------------> clean
    • 2019-01-22T*13:02:45.542Z*|2019-01-22T13:02:47.011Z|9d764278-bfce-489c-be9d-6bf3f324c5e3||http-nio-8080-exec-2||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d/vnfs/54d8314d-8b76-4a76-ad8b-270a39a4164b||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
    • 2019-01-22T*13:02:57.974Z*|2019-01-22T13:02:59.266Z|02ad82a2-198b-4e43-87c1-ddfde03c0b42||http-nio-8080-exec-4||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||COMPLETED|202||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|0|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.
  • --------------------------> manual attempt with the VID some minutes later
    • 2019-01-22T*13:24:45.*698Z|2019-01-22T13:24:45.764Z|a4be4eb1-7387-417d-b0a7-889c5d0e4ca9||http-nio-8080-exec-10||onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||*ERROR|*409||b1307e77-a122-4be4-993c-c3a5fd6f1b8d|INFO|2|||onap-so-so-85f675cfd9-6qsmg/10.42.102.18|||||||||Exiting.

From BPMN logs, It looks OK.

 in the debug.log of the SO (see full logs attached) around 13:02:58, I can see the deletion state moving from PENDING (13:02:58) to IN_PROGRESS (13:02:59)

but it stays in IN_PROGRESS and is never COMPLETED

that is why when I try manually to delete the instance through the VID I got

2019-01-22T*13:24:45.763Z*|a4be4eb1-7387-417d-b0a7-889c5d0e4ca9|http-nio-8080-exec-10|onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||||ERROR|||Error: Locked instance - This service (6536a8ee-a7fc-4443-a722-6c0918ba361d) already has a request being worked with a status of IN_PROGRESS (RequestId - 02ad82a2-198b-4e43-87c1-ddfde03c0b42). The existing request must finish or be cleaned up before proceeding.

org.onap.so.apihandlerinfra.exceptions.DuplicateRequestException: Error: Locked instance - This service (6536a8ee-a7fc-4443-a722-6c0918ba361d) already has a request being worked with a status of IN_PROGRESS (RequestId - 02ad82a2-198b-4e43-87c1-ddfde03c0b42). The existing request must finish or be cleaned up before proceeding.

2019-01-22T*13:24:45.763Z*|a4be4eb1-7387-417d-b0a7-889c5d0e4ca9|http-nio-8080-exec-10|onap/so/infra/serviceInstantiation/v7/serviceInstances/6536a8ee-a7fc-4443-a722-6c0918ba361d||||ERROR|400|Error: Locked instance - This service (6536a8ee-a7fc-4443-a722-6c0918ba361d) already has a request being worked with a status of IN_PROGRESS (RequestId - 02ad82a2-198b-4e43-87c1-ddfde03c0b42). The existing request must finish or be cleaned up before proceeding.|APIH_DUPLICATE_FOUND

And on the VID, at the end I can see lots of resources as no  instantation can be fully deleted (VNF and the VF module have been deleted but not the service instance)

---------------------------------------------------------------------------------------------------

With my testing tool, once I call the SO to delete the VF module, the VNF or the service instance, I check directly in the AA&I that the references have been effectively cleaned.

it is OK for the Module and the VNF but the service reference keeps in the A&AI

However we do not get message in the VID saying that the VNF has been cleaned

I got the same behavior using my test framework (https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests) or performing all the operations manually with the VID.

Note: I just created an instance (but no VNF, VF module) then delete it and I do not have the problem.