Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-2442

Rolling upgrade of MariaDB-Galera doesn't work if there is no actual upgrade in charts

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Highest Highest
    • None
    • Frankfurt Release, Guilin Release
    • None
    • None
    • Hide

      SUMMARY

      Rolling upgrade of MariaDB-Galera (https://jira.onap.org/browse/OOM-2316) doesn't work if there is no actual upgrade in charts.

      Basically if I do next upgrade of ONAP components by using helm deploy and there are no changes in mariadb-galera charts pre-install script scales down maradb-galera stateful set to 0 replicas, then as there are no changes in charts helm doesn't do anything to deploy mariadb-galera cluster and after post-upgrade script removes temporary pod  and that is causing lose of whole galera cluster and data.

      OS / ENVIRONMENT

      • Kubernetes version:
        1.15.6
      • Helm version:
        2.16.6

      OOM VERSION

      frankfurt

      CONFIGURATION

      mariadb-galera with 3 replicas

      STEPS TO REPRODUCE

      1. deploy with command: helm deploy onap6 local/onap --namespace onap6 -f onap/resources/overrides/myoverride.yaml --verbose
      2. upgrade with command: helm deploy onap6 local/onap --namespace onap6 -f onap/resources/overrides/myoverride.yaml --verbose

      EXPECTED RESULTS

      needed components upgraded. ones without changes still running

      ACTUAL RESULTS

      mariadb-galera clusters gets scaled down by pre-upgrade hook job and as there are no changes no deployment is done. and when post-upgrade script kicks in it deletes additional galera node and all data is lost.

      SUGGESTED FIX.

      add additional logic to post-upgrade hook to scale up galera cluster back if it is size of 0. (will do test in my local environment)

      ideal would be to see if any upgrade is going to happen on pre-upgrade hook, but after quick look couldn't find such option.

      Show
      SUMMARY Rolling upgrade of MariaDB-Galera ( https://jira.onap.org/browse/OOM-2316 ) doesn't work if there is no actual upgrade in charts. Basically if I do next upgrade of ONAP components by using helm deploy and there are no changes in mariadb-galera charts pre-install script scales down maradb-galera stateful set to 0 replicas, then as there are no changes in charts helm doesn't do anything to deploy mariadb-galera cluster and after post-upgrade script removes temporary pod  and that is causing lose of whole galera cluster and data. OS / ENVIRONMENT Kubernetes version: 1.15.6 Helm version: 2.16.6 OOM VERSION frankfurt CONFIGURATION mariadb-galera with 3 replicas STEPS TO REPRODUCE deploy with command: helm deploy onap6 local/onap --namespace onap6 -f onap/resources/overrides/myoverride.yaml --verbose upgrade with command: helm deploy onap6 local/onap --namespace onap6 -f onap/resources/overrides/myoverride.yaml --verbose EXPECTED RESULTS needed components upgraded. ones without changes still running ACTUAL RESULTS mariadb-galera clusters gets scaled down by pre-upgrade hook job and as there are no changes no deployment is done. and when post-upgrade script kicks in it deletes additional galera node and all data is lost. SUGGESTED FIX. add additional logic to post-upgrade hook to scale up galera cluster back if it is size of 0. (will do test in my local environment) ideal would be to see if any upgrade is going to happen on pre-upgrade hook, but after quick look couldn't find such option.

          sdesbure sdesbure
          maratsal maratsal
          Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

            Created:
            Updated:
            Resolved: