Name: sdnc-dbhost-0 Namespace: onap-sdnc Node: k8s-1/10.53.202.92 Start Time: Tue, 23 Jan 2018 14:09:14 +0000 Labels: app=sdnc-dbhost controller-revision-hash=sdnc-dbhost-3120665460 Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc-dbhost","uid":"eb70d57c-0046-11e8-be6f-0267980d8a5a","ap... pod.alpha.kubernetes.io/init-container-statuses=[{"name":"init-mysql","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2018-01-23T14:09:16Z","finishedAt":"2018-01-23T14:09:16Z","c... pod.alpha.kubernetes.io/init-containers=[{"name":"init-mysql","image":"mysql:5.7","command":["bash","-c","set -ex\n# Generate mysql server-id from pod ordinal index.\n[[ `hostname` =~ -([0-9]+)$ ]] ||... pod.beta.kubernetes.io/init-container-statuses=[{"name":"init-mysql","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2018-01-23T14:09:16Z","finishedAt":"2018-01-23T14:09:16Z","co... pod.beta.kubernetes.io/init-containers=[{"name":"init-mysql","image":"mysql:5.7","command":["bash","-c","set -ex\n# Generate mysql server-id from pod ordinal index.\n[[ `hostname` =~ -([0-9]+)$ ]] || ... Status: Running IP: 10.42.7.155 Created By: StatefulSet/sdnc-dbhost Controlled By: StatefulSet/sdnc-dbhost Init Containers: init-mysql: Container ID: docker://bb3117a119c1b05476642fcf4bb9c646b3bdaf81dc9ca151f4e24d75cbc24849 Image: mysql:5.7 Image ID: docker-pullable://mysql@sha256:1f95a2ba07ea2ee2800ec8ce3b5370ed4754b0a71d9d11c0c35c934e9708dcf1 Port: Command: bash -c set -ex # Generate mysql server-id from pod ordinal index. [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} echo BASH_REMATCH=${BASH_REMATCH} echo [mysqld] > /mnt/conf.d/server-id.cnf # Add an offset to avoid reserved server-id=0 value. echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf # Copy appropriate conf.d files from config-map to emptyDir. if [[ $ordinal -eq 0 ]]; then cp /mnt/config-map/master.cnf /mnt/conf.d/ else cp /mnt/config-map/slave.cnf /mnt/conf.d/ fi State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 23 Jan 2018 14:09:16 +0000 Finished: Tue, 23 Jan 2018 14:09:16 +0000 Ready: True Restart Count: 0 Environment: Mounts: /mnt/conf.d from conf (rw) /mnt/config-map from config-map (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-bbxpb (ro) clone-mysql: Container ID: docker://87fecf8ca068cf52115068385b76ea095cd4cba36ea43e2b4deadc6ff1fea403 Image: gcr.io/google-samples/xtrabackup:1.0 Image ID: docker-pullable://gcr.io/google-samples/xtrabackup@sha256:29354f70c9d9207e757a1bae6a4cbf2f57a56b18fe5c2b0acc1198a053b24b38 Port: Command: bash -c set -ex # Skip the clone if data already exists. [[ -d /var/lib/mysql/mysql ]] && exit 0 # Skip the clone on master (ordinal index 0). [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} echo ${BASH_REMATCH} [[ $ordinal -eq 0 ]] && exit 0 # Clone data from previous peer. ncat --recv-only sdnc-dbhost-$(($ordinal-1)).dbhost.onap-sdnc 3307 | xbstream -x -C /var/lib/mysql # Prepare the backup. xtrabackup --user=root --password=$MYSQL_ROOT_PASSWORD --prepare --target-dir=/var/lib/mysql ls -l /var/lib/mysql State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 23 Jan 2018 14:10:27 +0000 Finished: Tue, 23 Jan 2018 14:10:27 +0000 Ready: True Restart Count: 0 Environment: MYSQL_ROOT_PASSWORD: openECOMP1.0 Mounts: /etc/mysql/conf.d from conf (rw) /var/lib/mysql from sdnc-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-bbxpb (ro) Containers: sdnc-db-container: Container ID: docker://d41fd9a0629f9b1d06e560089a86db6300ad5661ee242929ae3dcde444bbda4e Image: mysql:5.7 Image ID: docker-pullable://mysql@sha256:1f95a2ba07ea2ee2800ec8ce3b5370ed4754b0a71d9d11c0c35c934e9708dcf1 Port: 3306/TCP State: Running Started: Tue, 23 Jan 2018 14:10:28 +0000 Ready: True Restart Count: 0 Requests: cpu: 500m memory: 1Gi Liveness: exec [mysqladmin ping] delay=30s timeout=5s period=10s #success=1 #failure=3 Readiness: tcp-socket :3306 delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: MYSQL_ROOT_PASSWORD: openECOMP1.0 MYSQL_ROOT_HOST: % MYSQL_ALLOW_EMPTY_PASSWORD: 0 Mounts: /etc/mysql/conf.d from conf (rw) /var/lib/mysql from sdnc-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-bbxpb (ro) xtrabackup: Container ID: docker://6450898b2468b136ce274494fba71d3f962d3807e1b7bf6c1454f00a81091b7c Image: gcr.io/google-samples/xtrabackup:1.0 Image ID: docker-pullable://gcr.io/google-samples/xtrabackup@sha256:29354f70c9d9207e757a1bae6a4cbf2f57a56b18fe5c2b0acc1198a053b24b38 Port: 3307/TCP Command: bash -c set -ex cd /var/lib/mysql ls -l # Determine binlog position of cloned data, if any. if [[ -f xtrabackup_slave_info ]]; then echo "Inside xtrabackup_slave_info" # XtraBackup already generated a partial "CHANGE MASTER TO" query # because we're cloning from an existing slave. mv xtrabackup_slave_info change_master_to.sql.in # Ignore xtrabackup_binlog_info in this case (it's useless). rm -f xtrabackup_binlog_info elif [[ -f xtrabackup_binlog_info ]]; then echo "Inside xtrabackup_binlog_info" # We're cloning directly from master. Parse binlog position. [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1 rm xtrabackup_binlog_info echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\ MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in fi # Check if we need to complete a clone by starting replication. if [[ -f change_master_to.sql.in ]]; then echo "Waiting for mysqld to be ready (accepting connections)" [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} echo $ordinal until mysql --user=root --password=$MYSQL_ROOT_PASSWORD -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done echo "Initializing replication from clone position" # In case of container restart, attempt this at-most-once. mv change_master_to.sql.in change_master_to.sql.orig mysql --user=root --password=$MYSQL_ROOT_PASSWORD -h 127.0.0.1 < Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s node.alpha.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 7m (x6 over 7m) default-scheduler PersistentVolumeClaim is not bound: "sdnc-data-sdnc-dbhost-0" Normal Scheduled 6m default-scheduler Successfully assigned sdnc-dbhost-0 to k8s-1 Normal SuccessfulMountVolume 6m kubelet, k8s-1 MountVolume.SetUp succeeded for volume "localtime" Normal SuccessfulMountVolume 6m kubelet, k8s-1 MountVolume.SetUp succeeded for volume "conf" Normal SuccessfulMountVolume 6m kubelet, k8s-1 MountVolume.SetUp succeeded for volume "config-map" Normal SuccessfulMountVolume 6m kubelet, k8s-1 MountVolume.SetUp succeeded for volume "default-token-bbxpb" Normal SuccessfulMountVolume 6m kubelet, k8s-1 MountVolume.SetUp succeeded for volume "pvc-eb7edbd9-0046-11e8-be6f-0267980d8a5a" Normal Pulled 6m kubelet, k8s-1 Container image "mysql:5.7" already present on machine Normal Created 6m kubelet, k8s-1 Created container Normal Pulled 6m kubelet, k8s-1 Container image "gcr.io/google-samples/xtrabackup:1.0" already present on machine Normal Started 6m kubelet, k8s-1 Started container Normal Created 5m kubelet, k8s-1 Created container Normal Started 5m kubelet, k8s-1 Started container Normal Pulled 5m kubelet, k8s-1 Container image "gcr.io/google-samples/xtrabackup:1.0" already present on machine Normal Pulled 5m kubelet, k8s-1 Container image "mysql:5.7" already present on machine Normal Created 5m kubelet, k8s-1 Created container Normal Started 5m kubelet, k8s-1 Started container Normal Created 5m kubelet, k8s-1 Created container Normal Started 5m kubelet, k8s-1 Started container Warning Unhealthy 5m kubelet, k8s-1 Readiness probe failed: dial tcp 10.42.7.155:3306: getsockopt: connection refused Warning DNSSearchForming 33s (x14 over 6m) kubelet, k8s-1 Search Line limits were exceeded, some dns names have been omitted, the applied search line is: onap-sdnc.svc.cluster.local svc.cluster.local cluster.local kubelet.kubernetes.rancher.internal kubernetes.rancher.internal rancher.internal