Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-1532

kubectl exec/log pod connection problems after x days - k8s 1.12 upgrade candidate

    Details

    • Type: Bug
    • Status: Closed
    • Priority: High
    • Resolution: Done
    • Affects Version/s: Casablanca Release
    • Fix Version/s: Dublin Release
    • Environment:

      Casablanca release (OOM) with Rancher, using the Integration Project tools

    • Sprint:
      Dublin-1 (12/03-01/23)

      Description

      In our labs (latest Casablanca release), after a couple of days, it is not possible to connect to pods via "kubectl exec" or "kubectl log", e.g.

      kubectl exec -ti -n onap dmaap-message-router-kafka-7c5f9cbc69-7rxq2 bash
      Error from server: error dialing backend: dial tcp 10.0.0.23:10250: connect: cannot assign requested address

      Reason is a known bug in Kubernetes (see https://github.com/kubernetes/kubernetes/issues/67659), which might be fixed in version 1.12.0

      So I logged in to the k8s-1, which hosts the dmaap-message-router and checked the open file descriptors of the kubelet (as described):

      ubuntu@onap-onap-oom-tm-c-k8s-1:~$ sudo ls -l /proc/3773/fd|wc -l
      56473
      ubuntu@onap-onap-oom-tm-c-k8s-1:~$ sudo netstat -plant|grep kubelet |wc -l
      56189
      

      I tried the suggested workaround by restarting the kubelet (via Rancher UI) and the connection to the pod was working again.
      I don't know yet, if this affects also the inter-pod communication or only the kubectl function, therefor I set the priority to "High"

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              • Assignee:
                melliott Mike Elliott
                Reporter:
                andreasgeissler Andreas Geissler
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: