Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-1825

Postgresql pod in COMMON project of OOM failing with error message as "Wed Apr 24 06:35:15 UTC 2019 WARN: Signal trap triggered, beginning shutdown.." in CASABLANCA release

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Medium Medium
    • Frankfurt Release
    • Casablanca Release
    • None

      The below pod is the postgresql db and it gets restarted continuously after successfully showing running state.

      dev1-dbc-pg-0                                    0/1       Running            158        14h

       

      On investigating the pod I found the below logs:

      Successfully running container logs:

      ubuntu@k8s-114:~$ kubectl logs dev1-dbc-pg-0 -n onap -c dbc-pg

      Tue Apr 23 08:40:49 UTC 2019

      Tue Apr 23 08:40:49 UTC 2019 INFO: Setting PGROOT to /usr/pgsql-10.

      Tue Apr 23 08:40:49 UTC 2019 INFO: Cleaning up the old postmaster.pid file..

      Tue Apr 23 08:40:49 UTC 2019 INFO: User ID is set to uid=26(postgres) gid=26(postgres) groups=26(postgres).

      Tue Apr 23 08:40:49 UTC 2019 INFO: Ordinal is set to 0.

      pod "dev1-dbc-pg-0" not labeled

      pod "dev1-dbc-pg-0" not labeled

      Tue Apr 23 08:40:50 UTC 2019 INFO: Setting PG_MODE to primary.

      Tue Apr 23 08:40:51 UTC 2019 INFO: Working on primary..

      Tue Apr 23 08:40:51 UTC 2019 INFO: Initializing the primary database..

      Tue Apr 23 08:40:51 UTC 2019 INFO: PGDATA already contains a database.

      Tue Apr 23 08:40:51 UTC 2019 INFO: Starting PostgreSQL..

      2019-04-23 08:40:51.131 UTC [67] LOG:  listening on IPv6 address "::1", port 5432

      2019-04-23 08:40:51.131 UTC [67] LOG:  listening on IPv4 address "127.0.0.1", port 5432

      2019-04-23 08:40:51.204 UTC [67] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

      2019-04-23 08:40:51.254 UTC [67] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"

      2019-04-23 08:40:51.427 UTC [67] LOG:  redirecting log output to logging collector process

      2019-04-23 08:40:51.427 UTC [67] HINT:  Future log output will appear in directory "log".

      ubuntu@k8s-114:~$ 

       

      Failed previous container logs:

      ubuntu@k8s-114:~$ 

      ubuntu@k8s-114:~$ kubectl logs --previous  dev1-dbc-pg-0 -n onap -c dbc-pg

      Tue Apr 23 08:35:19 UTC 2019

      Tue Apr 23 08:35:19 UTC 2019 INFO: Setting PGROOT to /usr/pgsql-10.

      Tue Apr 23 08:35:19 UTC 2019 INFO: Cleaning up the old postmaster.pid file..

      Tue Apr 23 08:35:19 UTC 2019 INFO: User ID is set to uid=26(postgres) gid=26(postgres) groups=26(postgres).

      Tue Apr 23 08:35:19 UTC 2019 INFO: Ordinal is set to 0.

      pod "dev1-dbc-pg-0" not labeled

      pod "dev1-dbc-pg-0" not labeled

      Tue Apr 23 08:35:20 UTC 2019 INFO: Setting PG_MODE to primary.

      Tue Apr 23 08:35:20 UTC 2019 INFO: Working on primary..

      Tue Apr 23 08:35:20 UTC 2019 INFO: Initializing the primary database..

      Tue Apr 23 08:35:20 UTC 2019 INFO: PGDATA already contains a database.

      Tue Apr 23 08:35:20 UTC 2019 INFO: Starting PostgreSQL..

      2019-04-23 08:35:21.289 UTC [66] LOG:  listening on IPv6 address "::1", port 5432

      2019-04-23 08:35:21.289 UTC [66] LOG:  listening on IPv4 address "127.0.0.1", port 5432

      2019-04-23 08:35:21.383 UTC [66] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

      2019-04-23 08:35:21.701 UTC [66] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"

      2019-04-23 08:35:22.400 UTC [66] LOG:  redirecting log output to logging collector process

      2019-04-23 08:35:22.400 UTC [66] HINT:  Future log output will appear in directory "log".

      Tue Apr 23 08:40:45 UTC 2019 WARN: Signal trap triggered, beginning shutdown..

      Tue Apr 23 08:40:45 UTC 2019 INFO: Cleanly shutting down PostgreSQL in force fast mode..

      waiting for server to shut down.... done

      server stopped

      Tue Apr 23 08:40:45 UTC 2019 INFO: PostgreSQL is shutting down. Exiting..

      ubuntu@k8s-114:~$ 

       

      Inside the POD environment log:

      2019-04-23 10:08:45.749 UTC [68] LOG:  database system is shut down

      2019-04-23 10:08:52.269 UTC [69] LOG:  database system was shut down at 2019-04-23 10:08:45 UTC

      2019-04-23 10:08:52.336 UTC [67] LOG:  database system is ready to accept connections

      2019-04-23 10:14:15.440 UTC [67] LOG:  received fast shutdown request

      2019-04-23 10:14:15.555 UTC [67] LOG:  aborting any active transactions

      2019-04-23 10:14:15.578 UTC [67] LOG:  worker process: logical replication launcher (PID 75) exited with exit code 1

      2019-04-23 10:14:15.578 UTC [70] LOG:  shutting down

      2019-04-23 10:14:15.692 UTC [67] LOG:  database system is shut down

      2019-04-23 10:14:21.344 UTC [68] LOG:  database system was shut down at 2019-04-23 10:14:15 UTC

      2019-04-23 10:14:21.488 UTC [66] LOG:  database system is ready to accept connections

      2019-04-23 10:19:45.445 UTC [66] LOG:  received fast shutdown request

      2019-04-23 10:19:45.600 UTC [66] LOG:  aborting any active transactions

      2019-04-23 10:19:45.604 UTC [66] LOG:  worker process: logical replication launcher (PID 74) exited with exit code 1

      2019-04-23 10:19:45.605 UTC [69] LOG:  shutting down

      2019-04-23 10:19:45.694 UTC [66] LOG:  database system is shut down

      2019-04-23 10:19:53.141 UTC [70] LOG:  database system was shut down at 2019-04-23 10:19:45 UTC

      2019-04-23 10:19:53.229 UTC [68] LOG:  database system is ready to accept connections

      2019-04-23 10:25:15.441 UTC [68] LOG:  received fast shutdown request

      2019-04-23 10:25:15.553 UTC [68] LOG:  aborting any active transactions

      2019-04-23 10:25:15.558 UTC [68] LOG:  worker process: logical replication launcher (PID 76) exited with exit code 1

      2019-04-23 10:25:15.558 UTC [71] LOG:  shutting down

      2019-04-23 10:25:15.655 UTC [68] LOG:  database system is shut down

      2019-04-23 10:25:21.397 UTC [70] LOG:  database system was shut down at 2019-04-23 10:25:15 UTC

      2019-04-23 10:25:21.483 UTC [68] LOG:  database system is ready to accept connections

      P.S.:

      This pod runs as part of dmaap-bus-controller where the DMAAP refers to the COMMON project's postgresql pod

      dependencies:

      • name: common
        version: ~3.0.0
        repository: '@local'
      • name: postgres
        version: ~3.0.0
        repository: '@local'

            melliott melliott
            janani janani
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: