Affects Version/s: Casablanca Release
Fix Version/s: Dublin Release
Sprint:AAF Dublin M1 01/11-01/31, AAF Dublin M2 02/01-02/21
While debugging and re-testing
AAI-2082, found that the aai-resources REST calls started to return "Unauthorized" error on GET calls, even though the GET calls were just working a second ago:
The aai-resources container logs showed the transition from "authenticated" to "invalid":
Thereafter, every GET call to aai-resources returned the "Unauthorized" error.
Checking the pods with kubectl showed no particular recent changes:
Checking the logs of aaf-service showed:
So apparently the AAF database is down, but the authentication API continued to respond to calls. Apparently this condition was not picked up by healthchecks.
We are trying to do a workaround by restarting AAF pods, but presumably this kind of corrective action should be automated by kubernetes, assuming it detects the service is down.
After a restart of AAF pods, found this error in the logs: