We have been trying to deploy OOM(ONAP Operations Manager) UI which is a portal SDK based web application in ONAP k8s3-k8s dev container environment. This is the first time, this application is being integrated  into the ONAP env. During the deployment process, we find that the container process is up but the DB Connection is not established. We cannot resolve the Postgres DB connection error in this app. I am trying to identify the root cause of this error. Need help.

 

Postgres DB cluster is also deployed into k8s PODs in this env.

 

Postgres POD Info:

ubuntu@k8s-rancher:~$ kubectl get pods -n onap | grep dcae-db

dev-dcaegen2-dcae-db-0                                      1/1       Running    0          20d

dev-dcaegen2-dcae-db-1                                      1/1       Running    0          20d

 

ubuntu@k8s-rancher:~$ kubectl exec -it dev-dcaegen2-dcae-db-0 -n onap /bin/bash

bash-4.2$ hostname -f

dev-dcaegen2-dcae-db-0.dcae-postgres.onap.svc.cluster.local

 

Postgres host name used for psql conn and jdbc URL: dcae-pg-primary.onap.svc.cluster.local

 

We think this host name is correct, it is accessible from the app container. I tried with different postgres host names but it is always the same error.

 

Psql works fine from within the container but DataSource initialization and connection pool is failing. The issue that I am reporting is almost similar to the one mentioned in this stackoverflow: https://stackoverflow.com/questions/2166460/c3p0-pool-cannot-establish-a-connection-how-to-debug-this

 

I see these error messages in the application / error logs during application startup/initialization and also whenever a request is sent to the dispatcher servlet. Can you please help to identify the root cause of this failure?

 

 

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.Map]: Factory method 'dataSourceMap' threw exception; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database!

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)

        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)

 

Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!

        at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)

        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)

        at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)

        at org.onap.portalsdk.core.conf.HibernateConfiguration.dataSourceMap(HibernateConfiguration.java:102)

 

Factory method 'dataSourceMap' threw exception; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database!

 

Here are the DB connection properties defined in system.properties that are used by Hibernate ORM to make DB connections:

 

db.driver = org.postgresql.Driver

db.connectionURL = jdbc:postgresql://dcae-pg-primary.onap.svc.cluster.local:5432/dashboard_pg

db.userName=dashboard_pg_admin

db.password=2552f9b46edf923f1a17c3f9b79218671eee28e7a272a059f16b99e6ed72464f

 

–

 

Additional Info:

 

ubuntu@k8s-rancher:~$ kubectl get pods -n onap | grep dcae-db

dev-dcaegen2-dcae-db-0                                      1/1       Running    0          20d

dev-dcaegen2-dcae-db-1                                      1/1       Running    0          20d

 

ubuntu@k8s-rancher:~$ kubectl exec -it dev-dcaegen2-dcae-db-0 -n onap /bin/bash

bash-4.2$ hostname

dev-dcaegen2-dcae-db-0

 

bash-4.2$ hostname -f

dev-dcaegen2-dcae-db-0.dcae-postgres.onap.svc.cluster.local