Uploaded image for project: 'Configuration Persistence Service'
  1. Configuration Persistence Service
  2. CPS-438

Make DB Schema Updates & Data Population More Robust for Kubernetes Environments

    XMLWordPrintable

Details

    • Story
    • Status: Closed
    • Medium
    • Resolution: Done
    • None
    • Kohn Release
    • CPS-Core

    Description

      CPS db schema migration script is managed by liquibase db migration tool.

      If not up to date, the database schema is automatically updated when CPS application is starting. This is quite convenient for dev standalone environments but not suitable for production and K8s environments.

      When the application is deployed in Kubernetes, if ever any database update script is not completed in few seconds (for example index creation, data migration, ... that can take some more time) it is likely that this additional time makes the application liveness check failing, causing the pod to be stopped and a new one to be created. The new container will not have more chance to start and even worse it won't be able to start the db update script as liquibase configuration is being kept locked by the previously killed container. This leads to a lock situation.

      To avoid this situation, OOM deployment needs to be made more robust. The proposition is to decouple the database update from the application startup by running the liquibase script from an init container of CPS pod. By doing that, the main container of the pod (CPS application) is going to start only after the db update init container is completed. Then, CPS liveness check is not dependant on db update anymore.

      See Liquibase Docker image that could help to run liquibase script from K8s init container.

       We have opted instead to use Kubernetes startup probe. See confluence document for details.

       

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              lukegleeson Luke Gleeson
              brusak Bruno Sakoto
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: