Uploaded image for project: 'Common Controller SDK'
  1. Common Controller SDK
  2. CCSDK-1874

Blueprint processor liveness probe fails during high load

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Highest Highest
    • Frankfurt Release
    • El Alto Release
    • cds
    • None

      The blueprint processor had its liveness probe fail occasionally when the rate of incoming requests is higher than the rate it can execute.
      This was causing disruption of service during that time where incoming requests rejected until the running requests finish running.

      This situation was causing the endpoint “health-check” to fail to answer.

      In order to replicate this behaviour with command executor, please take a look CBA I added and the following script which basically calls CDS multiple times.

      #!/bin/bash
      N=15
      echo "Will run N=${N} times the load test"
      echo "will sleep for 5 seconds before running,CTRL+C to cancel"
      #sleep 5for i in `seq 1 ${N}`; do
      (curl -X POST "http://<ip-to-reach-cds>:<port-to-reach-cds>/api/v1/execution-service/process" \
      -H 'Accept: */*' \
      -H 'Accept-Encoding: gzip, deflate' \
      -H 'Authorization: Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==' \
      -H 'Content-Type: application/json' \
      -d "{ 
        \"commonHeader\": { 
          \"originatorId\": \"System\",
          \"requestId\": \"123456\",
          \"subRequestId\": \"1234-12234\"
        },
        \"actionIdentifiers\": {
          \"blueprintName\": \"remote_scripts\",
          \"blueprintVersion\": \"1.0.0\",
          \"actionName\": \"execute-remote-python\",
          \"mode\": \"sync\"
        },
        \"payload\": {
          \"execute-remote-python-request\": {
            \"input\": \"RUN_NUM_${i}\"
          }
        }
      }";

        1. Screen Shot 2019-10-29 at 6.32.42 PM.png
          129 kB
          prathameshmo
        2. Logs-From-Script
          25 kB
          prathameshmo
        3. Command-Executor-Logs
          73 kB
          prathameshmo

            prathameshmo prathameshmo
            prathameshmo prathameshmo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: