Description

For the past few weeks we've been having issues with Policy->CDS timeouts. We've increased policy timeout but CDS still having issues. Most of our CDS executions take <5s to complete but there are times where it takes over a minute and its not due to the code.

In example A The prepare Env response is returned at 20:21:05, after that it takes a whole 00:01:50 to receive execute command request at 20:22:55.

2021-07-15 20:21:05,525|ea6c446c-213a-45eb-b092-bc41cf659f6a|0|POLICY|ThreadPoolExecutor-0_1|CommandExecutorServer|INFO|Prepare Env Response returned :
{'cds_is_successful': True, 'results_log': ['Installed pip packages:\n requirements.txt\n\n']}
2021-07-15 20:22:55,048|ea6c446c-213a-45eb-b092-bc41cf659f6a|0|POLICY|ThreadPoolExecutor-0_13|CommandExecutorServer|INFO|wanr-prod/1.0.3/5d51a76c-47a5-48f2-a89e-024b27eb5fb8 - Received executeCommand request

In example B the same CBA takes <1s to go from prepare ENV execute command. This CBA has been deployed for a few weeks so theres no reason package installation should take take long

2021-07-15 22:28:49,799|f5c83062-71b5-467f-a7f0-939f8718ace7|0|POLICY|ThreadPoolExecutor-0_12|CommandExecutorServer|INFO|Prepare Env Response returned :
{'cds_is_successful': True, 'results_log': ['Installed pip packages:\n requirements.txt\n\n']}
2021-07-15 22:28:49,803|f5c83062-71b5-467f-a7f0-939f8718ace7|0|POLICY|ThreadPoolExecutor-0_10|CommandExecutorServer|INFO|wanr-prod/1.0.3/5d51a76c-47a5-48f2-a89e-024b27eb5fb8 - Received executeCommand request

How to reproduce ?

1. Run 2 different instances of command-executor locally

1. 1. first one listens on port 50051
   2. second one listens on port 50052
2. upload CBAs :
   1. cba-100.zip
   2. cba-101.zip
3. Execute run_benchmark.sh from benchmark.zip
4. Wait till all request from benchmark are sent to command-executor (port 50052)
5. Send manual request of cba-100 and check log from command-executor with port 50051 -> You should see that prepareEnv is executed but executeCommand is never executed (or take a long time to be executed)