Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-1532

kubectl exec/log pod connection problems after x days - k8s 1.12 upgrade candidate

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: High High
    • Dublin Release
    • Casablanca Release
    • None

    • Casablanca release (OOM) with Rancher, using the Integration Project tools

    • Dublin-1 (12/03-01/23)

      In our labs (latest Casablanca release), after a couple of days, it is not possible to connect to pods via "kubectl exec" or "kubectl log", e.g.

      kubectl exec -ti -n onap dmaap-message-router-kafka-7c5f9cbc69-7rxq2 bash
Error from server: error dialing backend: dial tcp 10.0.0.23:10250: connect: cannot assign requested address

      Reason is a known bug in Kubernetes (see https://github.com/kubernetes/kubernetes/issues/67659), which might be fixed in version 1.12.0

      So I logged in to the k8s-1, which hosts the dmaap-message-router and checked the open file descriptors of the kubelet (as described):

      ubuntu@onap-onap-oom-tm-c-k8s-1:~$ sudo ls -l /proc/3773/fd|wc -l
56473
ubuntu@onap-onap-oom-tm-c-k8s-1:~$ sudo netstat -plant|grep kubelet |wc -l
56189

      I tried the suggested workaround by restarting the kubelet (via Rancher UI) and the connection to the pod was working again.
      I don't know yet, if this affects also the inter-pod communication or only the kubectl function, therefor I set the priority to "High"

            melliott melliott
            andreasgeissler Andreas Geissler
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: