Loading...

Type: Bug
Resolution: Done
Priority: High
Fix Version/s: Dublin Release
Affects Version/s: Casablanca Release
Labels:
None

Hi,

I'am installing Casablanca Release to get a quick try. After deploy ONAP, I found POD dev-vnfsdk-vnfsdk-postgres-0 can't be up and keeps in CrashLoopBackOff state. After some debugging, I found this pod is in running state in the first start but get restarted quickly due to postgres no reponse, and then keeping in CrashLoopBackOff due to postpres can't be up anymore.

Any suggestion what I am wrong here?

root@oom-rancher:~/oom# kubectl -n onap get pods | grep vnfsdk
dev-vnfsdk-vnfsdk-f448c7957-frhqs                            0/1       Init:0/1           1589       11d
dev-vnfsdk-vnfsdk-init-postgres-28wtq                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-5vqnc                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-62np7                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-6pcp2                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-7rzjp                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-9sbkt                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-bgdrp                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-h5fgj                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-jm296                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-lb77c                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-pvdqb                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-qrxkh                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-s644k                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-shjsx                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-ttjr4                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-wcfxq                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-xhpm7                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-init-postgres-xqbd6                        0/1       Init:Error         0          12d
dev-vnfsdk-vnfsdk-pgpool-d8974bc75-5fd4b                     1/1       Running            1          13d
dev-vnfsdk-vnfsdk-pgpool-d8974bc75-6bq2v                     1/1       Running            1          13d
dev-vnfsdk-vnfsdk-postgres-0                                 0/1       CrashLoopBackOff   1          37s

Here is some logs:

Container first run:

root@oom-rancher:~/oom# kubectl -n onap logs pod/dev-vnfsdk-vnfsdk-postgres-0
Thu Feb 14 02:15:54 UTC 2019
Thu Feb 14 02:15:54 UTC 2019 INFO: Setting PGROOT to /usr/pgsql-10.
Thu Feb 14 02:15:55 UTC 2019 INFO: Cleaning up the old postmaster.pid file..
Thu Feb 14 02:15:55 UTC 2019 INFO: User ID is set to uid=26(postgres) gid=26(postgres) groups=26(postgres).
Thu Feb 14 02:15:55 UTC 2019 INFO: Ordinal is set to 0.
pod "dev-vnfsdk-vnfsdk-postgres-0" not labeled
pod "dev-vnfsdk-vnfsdk-postgres-0" not labeled
Thu Feb 14 02:15:58 UTC 2019 INFO: Setting PG_MODE to primary.
Thu Feb 14 02:15:58 UTC 2019 INFO: Working on primary..
Thu Feb 14 02:15:58 UTC 2019 INFO: Initializing the primary database..
Thu Feb 14 02:15:58 UTC 2019 INFO: PGDATA already contains a database.
Thu Feb 14 02:15:58 UTC 2019 INFO: Starting PostgreSQL..
2019-02-14 02:15:59.055 GMT [67] LOG:  skipping missing configuration file "/pgdata/dev-vnfsdk-vnfsdk-postgres-0/postgresql.auto.conf"
2019-02-13 21:15:59 EST [67]: [2-1] user=,db=,app=,client=FATAL:  "/pgdata/dev-vnfsdk-vnfsdk-postgres-0" is not a valid data directory
2019-02-13 21:15:59 EST [67]: [3-1] user=,db=,app=,client=DETAIL:  File "/pgdata/dev-vnfsdk-vnfsdk-postgres-0/PG_VERSION" is missing.
Thu Feb 14 02:15:59 UTC 2019 INFO: PostgreSQL is shutting down. Exiting..
Thu Feb 14 02:20:18 UTC 2019
Thu Feb 14 02:20:19 UTC 2019 INFO: Setting PGROOT to /usr/pgsql-10.
Thu Feb 14 02:20:19 UTC 2019 INFO: Cleaning up the old postmaster.pid file..
Thu Feb 14 02:20:19 UTC 2019 INFO: User ID is set to uid=26(postgres) gid=26(postgres) groups=26(postgres).
Thu Feb 14 02:20:19 UTC 2019 INFO: Ordinal is set to 0.
pod "dev-vnfsdk-vnfsdk-postgres-0" not labeled
Thu Feb 14 02:20:19 UTC 2019 WARN: Signal trap triggered, beginning shutdown..
Thu Feb 14 02:20:19 UTC 2019 INFO: Cleanly shutting down PostgreSQL in force fast mode..
pg_ctl: directory "/pgdata/dev-vnfsdk-vnfsdk-postgres-0" is not a database cluster directory
pod "dev-vnfsdk-vnfsdk-postgres-0" not labeled
Thu Feb 14 02:20:20 UTC 2019 INFO: Setting PG_MODE to primary.
Thu Feb 14 02:20:21 UTC 2019 INFO: Working on primary..
Thu Feb 14 02:20:21 UTC 2019 INFO: Initializing the primary database..
Thu Feb 14 02:20:21 UTC 2019 INFO: PGDATA is empty. ID is uid=26(postgres) gid=26(postgres) groups=26(postgres). Creating the PGDATA directory..
Thu Feb 14 02:20:21 UTC 2019 INFO: Checking for restore..
total 0
Thu Feb 14 02:20:21 UTC 2019 INFO: No backup file found.
Thu Feb 14 02:20:21 UTC 2019 INFO: Starting initdb..
initdb -D /pgdata/dev-vnfsdk-vnfsdk-postgres-0 > /tmp/initdb.log &> /tmp/initdb.err
Thu Feb 14 02:20:21 UTC 2019 INFO: Overlaying PostgreSQL's default configuration with customized settings..
Thu Feb 14 02:20:21 UTC 2019 INFO: Checking for PITR WAL files to recover with..
Thu Feb 14 02:20:21 UTC 2019 INFO: Temporarily starting database to run setup.sql..
2019-02-14 02:20:21.389 GMT [86] LOG:  skipping missing configuration file "/pgdata/dev-vnfsdk-vnfsdk-postgres-0/postgresql.auto.conf"
pg_ctl: directory "/pgdata/dev-vnfsdk-vnfsdk-postgres-0" is not a database cluster directory
Thu Feb 14 02:20:21 UTC 2019 INFO: Waiting for PostgreSQL to start..
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response
dev-vnfsdk-vnfsdk-postgres-0:5432 - no response

Container logs after first run (Keeps restart):

root@oom-rancher:~/oom# kubectl -n onap logs pod/dev-vnfsdk-vnfsdk-postgres-0
Thu Feb 14 02:39:36 UTC 2019
Thu Feb 14 02:39:36 UTC 2019 INFO: Setting PGROOT to /usr/pgsql-10.
Thu Feb 14 02:39:36 UTC 2019 INFO: Cleaning up the old postmaster.pid file..
Thu Feb 14 02:39:36 UTC 2019 INFO: User ID is set to uid=26(postgres) gid=26(postgres) groups=26(postgres).
Thu Feb 14 02:39:36 UTC 2019 INFO: Ordinal is set to 0.
pod "dev-vnfsdk-vnfsdk-postgres-0" not labeled
pod "dev-vnfsdk-vnfsdk-postgres-0" not labeled
Thu Feb 14 02:39:38 UTC 2019 INFO: Setting PG_MODE to primary.
Thu Feb 14 02:39:38 UTC 2019 INFO: Working on primary..
Thu Feb 14 02:39:38 UTC 2019 INFO: Initializing the primary database..
Thu Feb 14 02:39:38 UTC 2019 INFO: PGDATA already contains a database.
Thu Feb 14 02:39:38 UTC 2019 INFO: Starting PostgreSQL..
2019-02-14 02:39:38.235 GMT [66] LOG:  skipping missing configuration file "/pgdata/dev-vnfsdk-vnfsdk-postgres-0/postgresql.auto.conf"
2019-02-13 21:39:38 EST [66]: [2-1] user=,db=,app=,client=FATAL:  "/pgdata/dev-vnfsdk-vnfsdk-postgres-0" is not a valid data directory
2019-02-13 21:39:38 EST [66]: [3-1] user=,db=,app=,client=DETAIL:  File "/pgdata/dev-vnfsdk-vnfsdk-postgres-0/PG_VERSION" is missing.
Thu Feb 14 02:39:38 UTC 2019 INFO: PostgreSQL is shutting down. Exiting..

relates to

OOM-1825 Postgresql pod in COMMON project of OOM failing with error message as "Wed Apr 24 06:35:15 UTC 2019 WARN: Signal trap triggered, beginning shutdown.." in CASABLANCA release

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates