On Daily Master CI chain and Gatign (based on Master) we got errors on OOF

• pods do  not start properly
• associated healthcheck tests are obviously failing

These problems are reproductive for more than a week

See on last Master CI daily:

• List of Failed Pods: [

• onap-oof-cmso-optimizer-7d55bfd7df-674q4
• onap-oof-cmso-service-5688c8645f-hf4ft
  {{ - onap-vfc-ems-driver-68d5d975fb-tmmwq]}}
   
  logs are available here: https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/jobs/427699462/artifacts/download
  if you browse to results/infrastructure-healthcheck/k8s/onap-k8s.log and go at the end you will see the describe of the pods + the logs (but apparently no logs in stdout from your pods)
   
  Warning Failed 40m (x7 over 41m) kubelet, compute10-onap-master Error: secret "onap-cmso-db" not found
  ...
  2020-02-06 06:23:36,289 - INFO - cmso-db is ready!
  Error from server (BadRequest): container "db-init" in pod "onap-oof-cmso-service-5688c8645f-hf4ft" is waiting to start: CreateContainerConfigError
  {{}}
  Similar behavior in a recent gating (100971-1):

aaf seems ok but 3 oof pods not running

https://orange-opensource.gitlab.io/lfn/onap/xtesting-onap-view/results/100971-1/infrastructure-healthcheck/k8s/onap-k8s.log

see at the end of the file, we can see the describe of the faulty pods:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 58m default-scheduler Successfully assigned onap/onap-oof-oof-cmso-optimizer-577ccbd874-kc28r to worker10-onap-gating-1
Normal Pulling 58m kubelet, worker10-onap-gating-1 Pulling image "oomk8s/readiness-check:2.0.0"
Normal Pulled 58m kubelet, worker10-onap-gating-1 Successfully pulled image "oomk8s/readiness-check:2.0.0"
Normal Created 58m kubelet, worker10-onap-gating-1 Created container oof-cmso-optimizer-readiness
Normal Started 58m kubelet, worker10-onap-gating-1 Started container oof-cmso-optimizer-readiness
Normal Pulled 56m (x7 over 57m) kubelet, worker10-onap-gating-1 Successfully pulled image "nexus3.onap.org:10001/onap/optf-cmso-dbinit:2.1.1"
Warning Failed 56m (x7 over 57m) kubelet, worker10-onap-gating-1 Error: secret "onap-oof-cmso-db" not found
Normal Pulling 3m42s (x228 over 57m) kubelet, worker10-onap-gating-1 Pulling image "nexus3.onap.org:10001/onap/optf-cmso-dbinit:2.1.1"
2020-01-31 11:46:01,694 - INFO - Checking if cmso-db is ready
2020-01-31 11:46:04,007 - INFO - cmso-db is not ready.
2020-01-31 11:46:09,026 - INFO - Checking if cmso-db is ready
2020-01-31 11:46:16,753 - INFO - cmso-db is not ready.
2020-01-31 11:46:21,766 - INFO - Checking if cmso-db is ready
2020-01-31 11:46:23,994 - INFO - cmso-db is not ready.
2020-01-31 11:46:29,005 - INFO - Checking if cmso-db is ready
2020-01-31 11:46:33,276 - INFO - cmso-db is not ready.
2020-01-31 11:46:38,290 - INFO - Checking if cmso-db is ready
2020-01-31 11:46:42,698 - INFO - cmso-db is not ready.
2020-01-31 11:46:47,716 - INFO - Checking if cmso-db is ready
2020-01-31 11:46:50,563 - INFO - cmso-db is not ready.
2020-01-31 11:46:55,612 - INFO - Checking if cmso-db is ready
2020-01-31 11:47:01,048 - INFO - cmso-db is not ready.
2020-01-31 11:47:06,086 - INFO - Checking if cmso-db is ready
2020-01-31 11:47:09,332 - INFO - cmso-db is ready!
Error from server (BadRequest): container "db-init" in pod "onap-oof-oof-cmso-optimizer-577ccbd874-kc28r" is waiting to start: CreateContainerConfigError

....