SUMMARY
Commit [[COMMON][MARIADB] Upgrade Mariadb DB galera version (Ib9976227) · Gerrit Code Review (onap.org)|https://gerrit.onap.org/r/c/oom/+/115482] introduced change that causes mariaDB galera to fail at startup on k8s dual stack environment.
OS / ENVIRONMENT
- Kubernetes version:
1.18.8
- Helm version:
v2.16.6
- Kubernetes mode of installation:
manual installation
- CNI Used for Kubernetes: calico
OOM VERSION
master
STEPS TO REPRODUCE
Run helm deploy on dual stack k8s environment
EXPECTED RESULTS
All pods are up & running.
ACTUAL RESULTS
Many pods that are using new mariaDB galera image, are in a state that pod *-0 is up and pod *-1 is in a CrashLoop, e.g.
dev-appc-db-0 2/2 Running 0 18h
dev-appc-db-1 1/2 CrashLoopBackOff 206 18h
dev-dmaap-dr-db-0 2/2 Running 0 18h
dev-dmaap-dr-db-1 1/2 CrashLoopBackOff 205 18h
dev-mariadb-galera-0 2/2 Running 0 18h
dev-mariadb-galera-1 1/2 CrashLoopBackOff 206 18h
dev-vfc-mariadb-0 2/2 Running 0 17h
dev-vfc-mariadb-1 1/2 CrashLoopBackOff 201 17h
In logs of failing pods we can see:
2020-12-30 15:37:02 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'fd00:100::7194 10.242.113.148' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --parent '1' --binlog 'mysql-bin' --mysqld-args --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid'
WSREP_SST: [INFO] Streaming with xbstream (20201230 15:37:03.665)
WSREP_SST: [INFO] Using socat as streamer (20201230 15:37:03.677)
WSREP_SST: [INFO] Stale sst_in_progress file: /bitnami/mariadb/data//sst_in_progress (20201230 15:37:03.694)
Error: "100::7194" does not look like a port.
Cannot parse dst/src address.
WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:100::7194 10.242.113.148,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20201230 15:37:03.859)
2020/12/30 15:37:03 socat[436] E exactly 2 addresses required (there are 3); use option "-h" for help
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20201230 15:37:03.889)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20201230 15:37:03.905)
Error: "100::7194" does not look like a port.
Cannot parse dst/src address.
Error: "100::7194" does not look like a port.
Cannot parse dst/src address.
Error: "100::7194" does not look like a port.
Cannot parse dst/src address.
Error: "100::7194" does not look like a port.
Cannot parse dst/src address.
Looks like scripts in image are having problems with handling situation where container is obtaining 2 IPs.
I think similar situation for bitnami/mariaDB-galera image is described here: https://github.com/bitnami/charts/issues/4077
Main difference is that in this case, even *-0 pod is not comming up (I have also tried "clean" bitnami/mariadb-galera helm chart and I confirm that, same problems as described in link occurs).
Attaching some logs of failing pods.