Since the 12th of January 2021, the CI/CD daily guilin chain shows bad results.

Most of the test errors seem to be linked to the SO.

When looking at the SO, all the pods look up&running

but there are exceptions in so-bpmn (detected with the healthcheck test as well)

example on the run from the 14th of January

global results: https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/01-14-2021_04-01/

Healthcheck core: https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/01-14-2021_04-01/xtesting-healthcheck/core/core/log.html#s1-t10

basic_vm: https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/01-14-2021_04-01/smoke-usecases/basic_vm/pythonsdk.debug.log.gz we can see that the steps before SO are OK, an exception occurs when calling SO (A la carte)

k8s cluste audit seems to indicate that everything is OK on SO side: https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/01-14-2021_04-01/infrastructure-healthcheck/k8s/kubernetes-status/index.html

However when having a look at teh log of so-bpmn we can see errors:

https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/01-14-2021_04-01/infrastructure-healthcheck/k8s/kubernetes-status/pod-onap-so-bpmn-infra-54f6db44f8-gfgts-so-bpmn-infra-logs.html

{{2021-01-14T04:17:10.590Z||main|||||ERROR|500||Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'CXFConfiguration': Unsatisfied dependency expressed through field 'wmr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'WorkflowMessageResource': Unsatisfied dependency expressed through field 'callback'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'CallbackHandlerService' defined in URL [jar:file:/app/app.jar!/BOOT-INF/classes!/org/onap/so/bpmn/common/workflow/service/CallbackHandlerService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.camunda.bpm.engine.spring.SpringProcessEngineServicesConfiguration': Unsatisfied dependency expressed through field 'processEngine'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.camunda.bpm.spring.boot.starter.CamundaBpmAutoConfiguration$ProcessEngineConfigurationImplDependingConfiguration': Unsatisfied dependency expressed through field 'processEngineConfigurationImpl'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'processEngineConfigurationImpl' defined in class path resource [org/camunda/bpm/spring/boot/starter/CamundaBpmConfiguration.class]: Unsatisfied dependency expressed through method 'processEngineConfigurationImpl' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'camundaDatasourceConfiguration': Unsatisfied dependency expressed through field 'transactionManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'transactionManager' defined in class path resource [org/onap/so/bpmn/infrastructure/CamundaDBConfig.class]: Unsatisfied dependency expressed through method 'transactionManager' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/onap/so/bpmn/infrastructure/CamundaDBConfig.class]: Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: unmatched placeholder start [j&)bWt,D7LOQMCPS)[$\{(VL#]
2021-01-14T04:17:10.623Z||main|||||INFO|500||Stopping service [Tomcat]
2021-01-14T04:17:10.629Z||main|||||WARN|500||The web application [ROOT] appears to have started a thread named [bpmn-pool housekeeper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.8/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.8/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.8/java.lang.Thread.run(Unknown Source)
2021-01-14T04:17:10.629Z||main|||||WARN|500||The web application [ROOT] appears to have started a thread named [bpmn-pool connection adder] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.8/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.8/java.lang.Thread.run(Unknown Source)
2021-01-14T04:17:10.631Z||main|||||WARN|500||Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
2021-01-14T04:17:10.631Z||main|||||INFO|500||bpmn-pool - Shutdown initiated...
2021-01-14T04:17:10.654Z||main|||||INFO|500||bpmn-pool - Shutdown completed.
2021-01-14T04:17:10.687Z||main|||||ERROR|500||Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat }}

{{}}

as a consequences all tests dealing with SO are no more working