Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-2661

MariadDB galera pods are failing to start on dual stack environment

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: High High
    • Honolulu Release
    • Honolulu Release
    • None
    • None
    • Hide

      SUMMARY

      Commit [[COMMON][MARIADB] Upgrade Mariadb DB galera version (Ib9976227) · Gerrit Code Review (onap.org)|https://gerrit.onap.org/r/c/oom/+/115482] introduced change that causes mariaDB galera to fail at startup on k8s dual stack environment.

      OS / ENVIRONMENT

      • Kubernetes version:
        1.18.8
      • Helm version:
        v2.16.6
      • Kubernetes mode of installation:
        manual installation
      • CNI Used for Kubernetes: calico

      OOM VERSION

      master

      STEPS TO REPRODUCE

      Run helm deploy on dual stack k8s environment

      EXPECTED RESULTS

      All pods are up & running.

      ACTUAL RESULTS

      Many pods that are using new mariaDB galera image, are in a state that pod *-0 is up and pod *-1 is in a CrashLoop, e.g.

      dev-appc-db-0 2/2 Running 0 18h
      dev-appc-db-1 1/2 CrashLoopBackOff 206 18h
      dev-dmaap-dr-db-0 2/2 Running 0 18h
      dev-dmaap-dr-db-1 1/2 CrashLoopBackOff 205 18h
      dev-mariadb-galera-0 2/2 Running 0 18h
      dev-mariadb-galera-1 1/2 CrashLoopBackOff 206 18h
      dev-vfc-mariadb-0 2/2 Running 0 17h
      dev-vfc-mariadb-1 1/2 CrashLoopBackOff 201 17h

       

      In logs of failing pods we can see:

      2020-12-30 15:37:02 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'fd00:100::7194 10.242.113.148' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --parent '1' --binlog 'mysql-bin' --mysqld-args --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid'
      WSREP_SST: [INFO] Streaming with xbstream (20201230 15:37:03.665)
      WSREP_SST: [INFO] Using socat as streamer (20201230 15:37:03.677)
      WSREP_SST: [INFO] Stale sst_in_progress file: /bitnami/mariadb/data//sst_in_progress (20201230 15:37:03.694)
      Error: "100::7194" does not look like a port.
      Cannot parse dst/src address.
      WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:100::7194 10.242.113.148,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20201230 15:37:03.859)
      2020/12/30 15:37:03 socat[436] E exactly 2 addresses required (there are 3); use option "-h" for help
      WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20201230 15:37:03.889)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20201230 15:37:03.905)
      Error: "100::7194" does not look like a port.
      Cannot parse dst/src address.
      Error: "100::7194" does not look like a port.
      Cannot parse dst/src address.
      Error: "100::7194" does not look like a port.
      Cannot parse dst/src address.
      Error: "100::7194" does not look like a port.
      Cannot parse dst/src address.

       

       

      Looks like scripts in image are having problems with handling situation where container is obtaining 2 IPs.

      I think similar situation for bitnami/mariaDB-galera image is described here: https://github.com/bitnami/charts/issues/4077

      Main difference is that in this case, even *-0 pod is not comming up (I have also tried "clean" bitnami/mariadb-galera helm chart and I confirm that, same problems as described in link occurs).

      Attaching some logs of failing pods.

       

       

      Show
      SUMMARY Commit [ [COMMON] [MARIADB] Upgrade Mariadb DB galera version (Ib9976227) · Gerrit Code Review (onap.org)|https://gerrit.onap.org/r/c/oom/+/115482] introduced change that causes mariaDB galera to fail at startup on k8s dual stack environment. OS / ENVIRONMENT Kubernetes version: 1.18.8 Helm version: v2.16.6 Kubernetes mode of installation: manual installation CNI Used for Kubernetes: calico OOM VERSION master STEPS TO REPRODUCE Run helm deploy on dual stack k8s environment EXPECTED RESULTS All pods are up & running. ACTUAL RESULTS Many pods that are using new mariaDB galera image, are in a state that pod *-0 is up and pod *-1 is in a CrashLoop, e.g. dev-appc-db-0 2/2 Running 0 18h dev-appc-db-1 1/2 CrashLoopBackOff 206 18h dev-dmaap-dr-db-0 2/2 Running 0 18h dev-dmaap-dr-db-1 1/2 CrashLoopBackOff 205 18h dev-mariadb-galera-0 2/2 Running 0 18h dev-mariadb-galera-1 1/2 CrashLoopBackOff 206 18h dev-vfc-mariadb-0 2/2 Running 0 17h dev-vfc-mariadb-1 1/2 CrashLoopBackOff 201 17h   In logs of failing pods we can see: 2020-12-30 15:37:02 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address ' fd00:100::7194 10.242.113.148 ' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --parent '1' --binlog 'mysql-bin' --mysqld-args --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid' WSREP_SST: [INFO] Streaming with xbstream (20201230 15:37:03.665) WSREP_SST: [INFO] Using socat as streamer (20201230 15:37:03.677) WSREP_SST: [INFO] Stale sst_in_progress file: /bitnami/mariadb/data//sst_in_progress (20201230 15:37:03.694) Error: "100::7194" does not look like a port. Cannot parse dst/src address. WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:100::7194 10.242.113.148,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS [@] } ) (20201230 15:37:03.859) 2020/12/30 15:37:03 socat [436] E exactly 2 addresses required (there are 3); use option "-h" for help WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20201230 15:37:03.889) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20201230 15:37:03.905) Error: "100::7194" does not look like a port. Cannot parse dst/src address. Error: "100::7194" does not look like a port. Cannot parse dst/src address. Error: "100::7194" does not look like a port. Cannot parse dst/src address. Error: "100::7194" does not look like a port. Cannot parse dst/src address.     Looks like scripts in image are having problems with handling situation where container is obtaining 2 IPs. I think similar situation for bitnami/mariaDB-galera image is described here: https://github.com/bitnami/charts/issues/4077 Main difference is that in this case, even *-0 pod is not comming up (I have also tried "clean" bitnami/mariadb-galera helm chart and I confirm that, same problems as described in link occurs). Attaching some logs of failing pods.    

          sdesbure sdesbure
          kosewski kosewski
          Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

            Created:
            Updated:
            Resolved: