-
Bug
-
Resolution: Done
-
High
-
Casablanca Release
-
None
-
Casablanca release (OOM) with Rancher, using the Integration Project tools
-
Dublin-1 (12/03-01/23)
In our labs (latest Casablanca release), after a couple of days, it is not possible to connect to pods via "kubectl exec" or "kubectl log", e.g.
kubectl exec -ti -n onap dmaap-message-router-kafka-7c5f9cbc69-7rxq2 bash Error from server: error dialing backend: dial tcp 10.0.0.23:10250: connect: cannot assign requested address
Reason is a known bug in Kubernetes (see https://github.com/kubernetes/kubernetes/issues/67659), which might be fixed in version 1.12.0
So I logged in to the k8s-1, which hosts the dmaap-message-router and checked the open file descriptors of the kubelet (as described):
ubuntu@onap-onap-oom-tm-c-k8s-1:~$ sudo ls -l /proc/3773/fd|wc -l 56473 ubuntu@onap-onap-oom-tm-c-k8s-1:~$ sudo netstat -plant|grep kubelet |wc -l 56189
I tried the suggested workaround by restarting the kubelet (via Rancher UI) and the connection to the pod was working again.
I don't know yet, if this affects also the inter-pod communication or only the kubectl function, therefor I set the priority to "High"
- duplicates
-
OOM-1520 K8S lab unstable after few days of operations - kublet restart required
- Closed
-
OOM-1516 ONAP health check fails and not recovered after a k8s cluster node failure
- Closed
- is blocked by
-
LOG-992 RKE 0.16 / Docker 18.06 for ONAP installation - migrate from Rancher for Dublin - script support
- Closed
-
OOM-1499 Update Dublin OOM user guide with Helm/K8s versions
- Closed
- relates to
-
LOG-895 Upgrade Rancher to 1.6.25 to address CVE-2018-1002105 and move to Kubernetes 1.11.5 (server side)
- Closed
-
OOM-1496 Update K8s and Helm for Dublin Release
- Closed