We automatically run stability tests consisting in 10 parallel basic_vm during 24h

The success rate is dropping and after a while all the tests are Fail.

The MariaDB is still up&running, the errors seem to be applicative now.

the different errors are listed as follows

first one that seems to be due to SDC

there is 1 error due to AAI the all the other ones seem to be due to SO or something behind SO (but from the tester perspective the error is on the a la carte so PBMN call).

Note that the same tests executed on honolulu-weekly (https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_honolulu/2021-05/27_10-47/)

are not fail after 24h

 

• first error (serie 27)
  test 1,2,9 (other tests are OK) => occurs only once (+ MichaelMorris ChrisC )
   _2021-05-29 04:30:52,469 ERROR 212:onap_service.py(1) - [SDC][Deep Load Service] request failed: Code: 500. Info: { "requestError": { "policyException": { "messageId": "POL5000", "text": "Error: Internal Server Error. Please try again later.", "variables": [] }

  }}._
   

• first full error (serie 34)
  test1, 4, 5, 6, 7, 8, 9, 10
   2021-05-29 05:32:11,555 ERROR 148:vf_module_ala_carte.py(1) - VfModule instantiation basic_vm0..BasicVm..base_ubuntu20..module-0 failed
   test2 _2021-05-29 05:30:37,634 ERROR 212:onap_service.py(1) - [None][Create 9a0a4e09-37a9-48af-880e-5d15590bf338 VF moduledeletion request] request failed: Code: 409. Info: {"requestError":{"serviceException": {"messageId":"SVC2000","text":"*Error: Locked instance* - This vfModule (9a0a4e09-37a9-48af-880e-5d15590bf338) already has a request being worked with a status of IN_PROGRESS (RequestId - db4d6742-5328-4ec7-806f-7d5f6cb7214f). The existing request must finish or be cleaned up before proceeding."}

  }}._
  test 3
   _2021-05-29 05:29:33,835 ERROR 174:vf_module_ala_carte.py(1) - VfModule deletion AAI failed
  _
  wreehil
   

• last error (serie 86)
  test1, 2, 3, 4, 7, 8, 9, 10

2021-05-30 02:06:05,151 ERROR 240:service_ala_carte.py(1) - Service instantiation basic_vm-99cc570a-0e84-4af0-970d-e03bd168c5e6 failed
 test 5 _2021-05-30 01:45:47,153 ERROR 212:onap_service.py(1) - [None][*Instantiate basic_vm service* a'la carte] request failed: Code: 500. Info:

{"message":"*Unexpected Internal Exceptio*n"}

._
 test 6 _2021-05-30 01:46:14,388 ERROR 212:onap_service.py(1) - [None][Instantiate basic_vm service a'la carte] request failed: Code: 502. Info: {"requestError":{"serviceException":

{"messageId":"SVC2000","text":"*Request Failed due to BPEL error with HTTP Status = 500 INTERNAL_SERVER_ERROR* <aetgt:WorkflowException xmlns:aetgt=\"http://org.onap/so/workflow/schema/v1\"><aetgt:ErrorMessage>Exception while setting execution list. 500 : [\{\"timestamp\":\"2021-05-30T01:45:59.996+00:00\",\"status\":500,\"error\":\"Internal Server Error\",\"message\":\"\",\"path\":\"/northbound_request_ref_lookup/search/findOneByActionAndRequestScopeAndIsAlacarteAndCloudOwnerAndServiceType\"}

]</aetgt:ErrorMessage><aetgt:ErrorCode>7000</aetgt:ErrorCode></aetgt:WorkflowException>"}}}.
_