-
Bug
-
Resolution: Done
-
Medium
-
Frankfurt Release
-
None
-
Any
In the recent testing, it was observed that intermittently the dmaap connection from policy pods (drools-pdp, apex-pdp etc.) comes back with "503 Service Unavailable".
With the quick analysis it was found in dmaap logs that there were too many requests made at the same time. And there is some concurrency limits.
Issue needs to be investigated further on both policy and dmaap side.
Sample error message
19:12:57.928 [DMAAP-source-POLICY-PDP-PAP] INFO c.att.nsa.apiClient.http.HttpClient - GET https://message-router:3905/events/POLICY-PDP-PAP/2d2648c1-a523-4c09-ac0d-abe46f9a4721/frapreprod-policy-apex-pdp-0 (anonymous) ... 19:13:03.932 [DMAAP-source-POLICY-PDP-PAP] INFO c.att.nsa.apiClient.http.HttpClient - --> HTTP/1.1 503 Service Unavailable 19:13:03.932 [DMAAP-source-POLICY-PDP-PAP] ERROR o.o.p.c.e.e.c.b.i.BusConsumer$CambriaConsumerWrapper - CambriaConsumerWrapper [fetchTimeout=-1]: cannot fetch because of com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable - backoff for -1 ms. 19:13:03.932 [DMAAP-source-POLICY-PDP-PAP] ERROR o.o.p.c.e.e.c.b.i.SingleThreadedBusTopicSource - SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=2d2648c1-a523-4c09-ac0d-abe46f9a4721, consumerInstance=frapreprod-policy-apex-pdp-0, fetchTimeout=-1, fetchLimit=-1, consumer=CambriaConsumerWrapper [fetchTimeout=-1], alive=true, locked=false, uebThread=Thread[DMAAP-source-POLICY-PDP-PAP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=null, apiSecret=null, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=POLICY-PDP-PAP, effectiveTopic=POLICY-PDP-PAP, #recentEvents=10, locked=false, #topicListeners=1]]]]: cannot fetch java.io.IOException: com.att.nsa.apiClient.http.HttpException: HTTP/1.1 503 Service Unavailable