the onap-sdnc-cds-sdc-listener is the last remaining non Running pod on Master/Dublin

the describe of the pod:

Name: onap-sdnc-cds-sdc-listener-6bffbdddb-rhv2v
Namespace: onap
Node: compute08-onap-gating-2/10.253.0.9
Start Time: Fri, 17 May 2019 09:53:16 +0000
Labels: app=cds-sdc-listener
pod-template-hash=6bffbdddb
release=onap-sdnc
Annotations: <none>
Status: Running
IP: 10.233.107.18
Controllers: <none>
Init Containers:
cds-sdc-listener-readiness:
Container ID: docker://9b2f1d7cafdad3e65d8cada7a786fce8c541d120a052ea8f85a6e166448c9191
Image: oomk8s/readiness-check:2.0.2
Image ID: docker-pullable://oomk8s/readiness-check@sha256:875814cc853d7d9b3227165740584c347884bc8c0d831b4148cb5f7952e4cbff
Port:
Command:
/root/ready.py
Args:
--container-name
sdc-be
--container-name
message-router
--container-name
cds-blueprints-processor
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 17 May 2019 09:54:18 +0000
Finished: Fri, 17 May 2019 10:02:07 +0000
Ready: True
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bpgsp (ro)
Containers:
cds-sdc-listener:
Container ID: docker://9d294445088b1f839c04cefab2d01e4e464f77aead80eb459386ea579efa3130
Image: nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.4.2
Image ID: docker-pullable://nexus3.onap.org:10001/onap/ccsdk-sdclistener@sha256:0c70ad66f53b2a5d491c68b2584fcf43276c33664626597a1fcf6f15e7498042
Port: 8080/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 17 May 2019 10:25:04 +0000
Finished: Fri, 17 May 2019 10:25:11 +0000
Ready: False
Restart Count: 9
Liveness: http-get http://:8080/api/v1/sdclistener/healthcheck delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8080/api/v1/sdclistener/healthcheck delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
APP_CONFIG_HOME: /opt/app/onap/config
Mounts:
/opt/app/onap/config/application.yaml from onap-sdnc-cds-sdc-listener-config (rw)
/opt/app/onap/config/logback.xml from onap-sdnc-cds-sdc-listener-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bpgsp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
onap-sdnc-cds-sdc-listener-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: onap-sdnc-cds-sdc-listener-configmap
Optional: false
default-token-bpgsp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bpgsp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready=:Exists:NoExecute for 300s
node.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
35m 35m 1 default-scheduler Normal Scheduled Successfully assigned onap/onap-sdnc-cds-sdc-listener-6bffbdddb-rhv2v to compute08-onap-gating-2
35m 35m 1 kubelet, compute08-onap-gating-2 spec.initContainers{cds-sdc-listener-readiness} Normal Pulling pulling image "oomk8s/readiness-check:2.0.2"
34m 34m 1 kubelet, compute08-onap-gating-2 spec.initContainers{cds-sdc-listener-readiness} Normal Pulled Successfully pulled image "oomk8s/readiness-check:2.0.2"
34m 34m 1 kubelet, compute08-onap-gating-2 spec.initContainers{cds-sdc-listener-readiness} Normal Created Created container
34m 34m 1 kubelet, compute08-onap-gating-2 spec.initContainers{cds-sdc-listener-readiness} Normal Started Started container
26m 25m 4 kubelet, compute08-onap-gating-2 spec.containers{cds-sdc-listener} Normal Pulling pulling image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.4.2"
26m 25m 4 kubelet, compute08-onap-gating-2 spec.containers{cds-sdc-listener} Normal Pulled Successfully pulled image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.4.2"
26m 25m 4 kubelet, compute08-onap-gating-2 spec.containers{cds-sdc-listener} Normal Created Created container
26m 25m 4 kubelet, compute08-onap-gating-2 spec.containers{cds-sdc-listener} Normal Started Started container
26m 38s 117 kubelet, compute08-onap-gating-2 spec.containers{cds-sdc-listener} Warning BackOff Back-off restarting failed container

the error log on the pod is
APP Config HOME : /opt/app/onap/config
08:54:17,805 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
08:54:17,805 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
08:54:17,805 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.4.2.jar!/logback.xml]
08:54:17,807 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
08:54:17,807 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.4.2.jar!/logback.xml]
08:54:17,807 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at file:/opt/app/onap/config/logback.xml
08:54:17,828 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@dfd3711 - URL [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.4.2.jar!/logback.xml] is not of type file
08:54:17,947 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
08:54:17,949 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
08:54:17,961 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
08:54:17,978 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
08:54:18,107 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
08:54:18,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
08:54:18,109 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework] to INFO
08:54:18,109 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework.web] to INFO
08:54:18,109 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.onap.ccsdk.cds] to INFO
08:54:18,109 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
08:54:18,113 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@42d3bd8b - Registering current configuration as safe fallback point
 
 
  .   ___                      __ _ _
 /
/ __' __ _ ()_ _  _ _ \ \ \ \
( ( )___ | '_ | '| | ' \/ _` | \ \ \ \
 
/  __)| |)| | | | | || (_| |  ) ) ) )
  '  |___| .|| ||| |_, | / / / /
 =========||==============|_/=////
 :: Spring Boot ::        (v2.1.3.RELEASE)
 
2019-05-17 08:54:19,407|||main||||INFO||||||| Starting SdcListenerApplication on onap-sdnc-cds-sdc-listener-6bffbdddb-lfk4q with PID 8 (/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.4.2.jar started by root in /)
2019-05-17 08:54:19,415|||main||||INFO||||||| No active profile set, falling back to default profiles: default
2019-05-17 08:54:21,672|||main||||INFO||||||| Tomcat initialized with port(s): 9000 (http)
2019-05-17 08:54:21,913|||main||||INFO||||||| Root WebApplicationContext: initialization completed in 2308 ms
2019-05-17 08:54:22,971|||main||||INFO||||||| Initializing ExecutorService 'applicationTaskExecutor'
2019-05-17 08:54:23,428|||main||||INFO||||||| Tomcat started on port(s): 9000 (http) with context path ''
2019-05-17 08:54:23,433|||main||||INFO||||||| Started SdcListenerApplication in 5.055 seconds (JVM running for 5.976)
2019-05-17 08:54:23,435|||main||||INFO||||||| Initialize the SDC distribution client
2019-05-17 08:54:24,214|||main||||ERROR||||||| failed to connect to url: /sdc/v1/artifactTypes
org.apache.http.conn.HttpHostConnectException: Connect to localhost:8443 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.onap.sdc.http.HttpAsdcClient.getRequest(HttpAsdcClient.java:277)
at org.onap.sdc.http.SdcConnectorClient.performAsdcServerRequest(SdcConnectorClient.java:127)
at org.onap.sdc.http.SdcConnectorClient.getValidArtifactTypesList(SdcConnectorClient.java:95)
at org.onap.sdc.impl.DistributionClientImpl.validateArtifactTypesWithAsdcServer(DistributionClientImpl.java:340)
at org.onap.sdc.impl.DistributionClientImpl.init(DistributionClientImpl.java:286)
at org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient(SdcListenerClient.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:259)
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:179)
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:142)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:402)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:359)
at org.springframework.boot.context.event.EventPublishingRunListener.running(EventPublishingRunListener.java:105)
at org.springframework.boot.SpringApplicationRunListeners.running(SpringApplicationRunListeners.java:78)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:332)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at org.onap.ccsdk.cds.sdclistener.SdcListenerApplication.main(SdcListenerApplication.java:26)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:348)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 33 common frames omitted
2019-05-17 08:54:24,219|||main||||ERROR||||||| status from ASDC is org.onap.sdc.http.HttpAsdcResponse@6b2e46af
2019-05-17 08:54:24,219|||main||||ERROR||||||| DistributionClientResultImpl [responseStatus=ASDC_CONNECTION_FAILED, responseMessage=ASDC server problem]
2019-05-17 08:54:24,231|||main||||ERROR||||||| Application run failed
java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method
HandlerMethod details: 
Bean [org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient]
Method [public void org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient() throws org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException]
Resolved arguments: 
 
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:276)
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:179)
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:142)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:402)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:359)
at org.springframework.boot.context.event.EventPublishingRunListener.running(EventPublishingRunListener.java:105)
at org.springframework.boot.SpringApplicationRunListeners.running(SpringApplicationRunListeners.java:78)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:332)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at org.onap.ccsdk.cds.sdclistener.SdcListenerApplication.main(SdcListenerApplication.java:26)
Caused by: org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException: SDC distribution client init failed with reason:ASDC server problem
at org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.startSdcClientBasedOnTheResult(SdcListenerClient.java:73)
at org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient(SdcListenerClient.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:259)
... 13 common frames omitted
2019-05-17 08:54:24,235|||main||||INFO||||||| Shutting down ExecutorService 'applicationTaskExecutor'
2019-05-17 08:54:24,235|||main||||INFO||||||| Stopping GRPC connection to CDS backend