Uploaded image for project: 'Policy Framework'
  1. Policy Framework
  2. POLICY-2884

Dmaap connectivity failure in policy component logs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: Frankfurt Release
    • Fix Version/s: Frankfurt Release
    • Component/s: None
    • Labels:
    • Environment:

      Any

      Description

      In the recent testing, it was observed that intermittently the dmaap connection from policy pods (drools-pdp, apex-pdp etc.) comes back with "503 Service Unavailable".

      With the quick analysis it was found in dmaap logs that there were too many requests made at the same time. And there is some concurrency limits. 

      Issue needs to be investigated further on both policy and dmaap side.

      Sample error message

      19:12:57.928 [DMAAP-source-POLICY-PDP-PAP] INFO  c.att.nsa.apiClient.http.HttpClient - GET https://message-router:3905/events/POLICY-PDP-PAP/2d2648c1-a523-4c09-ac0d-abe46f9a4721/frapreprod-policy-apex-pdp-0 (anonymous) ...
      19:13:03.932 [DMAAP-source-POLICY-PDP-PAP] INFO  c.att.nsa.apiClient.http.HttpClient - 	--> HTTP/1.1 503 Service Unavailable
      19:13:03.932 [DMAAP-source-POLICY-PDP-PAP] ERROR o.o.p.c.e.e.c.b.i.BusConsumer$CambriaConsumerWrapper - CambriaConsumerWrapper [fetchTimeout=-1]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable - backoff for -1 ms.
      19:13:03.932 [DMAAP-source-POLICY-PDP-PAP] ERROR o.o.p.c.e.e.c.b.i.SingleThreadedBusTopicSource - SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=2d2648c1-a523-4c09-ac0d-abe46f9a4721, consumerInstance=frapreprod-policy-apex-pdp-0, fetchTimeout=-1, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=-1], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=10, locked=false, #topicListeners=1]]]]: cannot fetch
      java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable

        Attachments

        # Subject Branch Project Status CR V

          Activity

            People

            Assignee:
            ramverma Ram Krishna Verma
            Reporter:
            ramverma Ram Krishna Verma
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: