Observed Problem:  All containers running the ready.py readiness check on init-container throw a 403 Forbidden error when trying to get the pod details:

2017-06-30 21:33:21,292 - INFO - Checking if hbase  is ready2017-06-30 21:33:21,301 - ERROR - Exception when calling list_namespaced_pod: (403)Reason: ForbiddenHTTP response headers: HTTPHeaderDict({'Date': 'Fri, 30 Jun 2017 21:33:21 GMT', 'Content-Length': '57', 'Content-Type': 'text/plain; charset=utf-8'})HTTP response body: Forbidden: "/api/v1/namespaces/onap-aai/pods?watch=False"

This causes the ready.py script to never return that the other container is running, so spin up of dependent containers is blocked.

Environment: Kubernetes 1.5 with RBAC enabled.  Specifically tested on the IBM Bluemix Kubernetes service

Background on Issue: The ready.py script is accessing the API from inside a pod using automatically mounted service account credentials, as described in Accessing the Cluster”

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

Also, see https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod

However, in Kubernetes 1.5 or 1.6 where RBAC is enabled, the automatically mounted service account credentials do not have rights to view their namespace information:

"Default RBAC policies grant scoped permissions to control-plane components, nodes, and controllers, but grant no permissions to service accounts outside the “kube-system” namespace (beyond discovery permissions given to all authenticated users)."

from this page: https://kubernetes.io/docs/admin/authorization/rbac/#api-overview

Suggested Resolution

There are two ways to address this, depending on the Kubernetes version.  

For v 1.6, you can issue a command such as:

kubectl create rolebinding serviceaccounts-view  --clusterrole=view --group=system:serviceaccounts:onap-aai --namespace=onap-aai

where onap-aai is the namespace.

For v1.5, this command won't work, and you have to create it via yaml with the following template:

apiVersion: rbac.authorization.k8s.io/v1alpha1

kind: ClusterRoleBinding

metadata:

  name: AAA-serviceaccounts-view

  namespace: NNN-AAA

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  }}{{name: view

subjects:

- kind: Group

  name: system:serviceaccounts:NNN-AAA

  apiVersion: rbac.authorization.k8s.io

 where NNN is the namespace prefix being used and AAA is the app portion of the namespace (aai, robot, sdc, etc.)

This template can be embedded in the all-services.yaml for each component... though you do have to figure out how to get the user-selected namespace-prefix into that yaml file.