On some kubernetes deployments there is a conflict when a volume in docker image is mounted in kubernetes over NFS.

The observed behavior is that files / directories created inside the container are not visible inside / outside of the container.

Removing VOLUME declaration from the Dockerfile solve this problem.

Related defect with similar symptoms: OOM-258

Log:

[root@kubernetes-master-borislavki-1867-09-04-17 config]# kubectl describe pods -n onap config-init
Name: config-init
Namespace: onap
Node: kubernetes-node-borislavki-1867-09-04-17-2/10.247.58.31
Start Time: Thu, 07 Sep 2017 13:17:00 +0300
Labels: <none>
Annotations: <none>
Status: Succeeded
IP: 10.244.3.117
Containers:
config-init:
Container ID: docker://b6415e21bcb21b08ab1b97ebb9fca0517917f3ea4a07f948e0890c87a3352e58
Image: oomk8s/config-init:1.1.0
Image ID: docker-pullable://docker.io/oomk8s/config-init@sha256:8c9989e35f424110a82cae773bda785dedd9634ee7da019bb14bc1fdfe32a2fd
Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 07 Sep 2017 13:17:05 +0300
Finished: Thu, 07 Sep 2017 13:17:05 +0300
Ready: False
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/config-init/ from config-init-root (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-t8bfw (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
config-init-root:
Type: HostPath (bare host directory volume)
Path: /dockerdata-nfs/
default-token-t8bfw:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-t8bfw
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
43s 43s 1 default-scheduler Normal Scheduled Successfully assigned config-init to kubernetes-node-borislavki-1867-09-04-17-2
6s 6s 1 kubelet, kubernetes-node-borislavki-1867-09-04-17-2 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "config-init-root"
6s 6s 1 kubelet, kubernetes-node-borislavki-1867-09-04-17-2 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-t8bfw"
5s 5s 1 kubelet, kubernetes-node-borislavki-1867-09-04-17-2 spec.containers{config-init} Normal Pulling pulling image "oomk8s/config-init:1.1.0"
3s 3s 1 kubelet, kubernetes-node-borislavki-1867-09-04-17-2 spec.containers{config-init} Normal Pulled Successfully pulled image "oomk8s/config-init:1.1.0"
2s 2s 1 kubelet, kubernetes-node-borislavki-1867-09-04-17-2 spec.containers{config-init} Normal Created Created container
1s 1s 1 kubelet, kubernetes-node-borislavki-1867-09-04-17-2 spec.containers{config-init} Normal Started Started container
[root@kubernetes-master-borislavki-1867-09-04-17 config]# kubectl logs -n onap config-init
cp: cannot create directory '/config-init/onap/aai/aai-config': No such file or directory
cp: cannot create directory '/config-init/onap/aai/aai-data': No such file or directory
cp: cannot create directory '/config-init/onap/aai/etc': No such file or directory
cp: cannot create directory '/config-init/onap/aai/model-loader': No such file or directory
cp: cannot create directory '/config-init/onap/aai/aai-resources': No such file or directory
'/opt/config/src/aai/aai-traversal/docker-entrypoint.sh' -> '/config-init/onap/aai/aai-traversal/docker-entrypoint.sh'
cp: cannot create regular file '/config-init/onap/aai/aai-traversal/docker-entrypoint.sh': No such file or directory
cp: cannot create directory '/config-init/onap/aai/elasticsearch': No such file or directory
cp: cannot create directory '/config-init/onap/aai/sparky-be': No such file or directory
cp: cannot create directory '/config-init/onap/aai/data-router': No such file or directory
cp: cannot create directory '/config-init/onap/aai/haproxy': No such file or directory
cp: cannot create directory '/config-init/onap/aai/search-data-service': No such file or directory
cp: cannot create directory '/config-init/onap/appc': No such file or directory
cp: cannot create directory '/config-init/onap/dcae': No such file or directory
cp: cannot create directory '/config-init/onap/message-router': No such file or directory
cp: cannot create directory '/config-init/onap/mso': No such file or directory
cp: cannot create directory '/config-init/onap/policy': No such file or directory
cp: cannot create directory '/config-init/onap/portal': No such file or directory
cp: cannot create directory '/config-init/onap/robot': No such file or directory
cp: cannot create directory '/config-init/onap/sdc': No such file or directory
cp: cannot create directory '/config-init/onap/sdnc': No such file or directory
cp: cannot create directory '/config-init/onap/vid': No such file or directory
mkdir: cannot create directory '/config-init/onap/appc': No such file or directory
mkdir: cannot create directory '/config-init/onap/dcae': No such file or directory
mkdir: cannot create directory '/config-init/onap/portal': No such file or directory
mkdir: cannot create directory '/config-init/onap/portal': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdnc': No such file or directory
mkdir: cannot create directory '/config-init/onap/vid': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdc': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdc': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdc': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdc': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdc': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdc': No such file or directory
mkdir: cannot create directory '/config-init/onap/sdc': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/opt': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/model-loader': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/haproxy': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/aai-traversal/logs/': File exists
mkdir: cannot create directory '/config-init/onap/aai/aai-resources': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/sparky-be': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/elasticsearch': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/search-data-service': No such file or directory
mkdir: cannot create directory '/config-init/onap/aai/data-router': No such file or directory
mkdir: cannot create directory '/config-init/onap/mso': No such file or directory
chmod: cannot access '/config-init/onap/sdc/logs/': No such file or directory
chmod: cannot access '/config-init/onap/portal/logs/': No such file or directory
chmod: cannot access '/config-init/onap/aai/aai-config/': No such file or directory
chmod: cannot access '/config-init/onap/aai/aai-data/': No such file or directory
chmod: cannot access '/config-init/onap/aai/opt/aai/logroot/': No such file or directory
chmod: cannot access '/config-init/onap/aai/model-loader/logs/': No such file or directory
chmod: cannot access '/config-init/onap/aai/haproxy/log/': No such file or directory
chmod: cannot access '/config-init/onap/aai/aai-traversal/logs/': Stale file handle
chmod: cannot access '/config-init/onap/aai/aai-resources/logs/': No such file or directory
chmod: cannot access '/config-init/onap/aai/sparky-be/logs/': No such file or directory
chmod: cannot access '/config-init/onap/aai/elasticsearch/es-data/': No such file or directory
chmod: cannot access '/config-init/onap/aai/search-data-service/logs/': No such file or directory
chmod: cannot access '/config-init/onap/aai/data-router/logs/': No such file or directory
chmod: cannot access '/config-init/onap/policy/mariadb/': No such file or directory