-
Bug
-
Resolution: Done
-
High
-
Dublin Release
-
None
-
SB-02
We have failed many times when deploying released version of SDC in Windriver lab recently. A few pods/jobs fail to come up or complete. Here is the pod list:
root@sb02-rancher:~# kubectl -n onap get pod | grep sdc- dev-sdc-sdc-be-5b8bdcbc8b-ktf5h 0/2 Init:0/2 1 19m dev-sdc-sdc-be-config-backend-4lrlc 0/1 Init:Error 0 91m dev-sdc-sdc-be-config-backend-4mrsx 0/1 Init:Error 0 123m dev-sdc-sdc-be-config-backend-7dhwr 0/1 Init:Error 0 26m dev-sdc-sdc-be-config-backend-9vckt 0/1 Init:Error 0 42m dev-sdc-sdc-be-config-backend-gjnjw 0/1 Init:Error 0 3h4m dev-sdc-sdc-be-config-backend-grmqr 0/1 Init:Error 0 107m dev-sdc-sdc-be-config-backend-hcv8w 0/1 Init:Error 0 152m dev-sdc-sdc-be-config-backend-s69ft 0/1 Init:Error 0 3h25m dev-sdc-sdc-be-config-backend-stktl 0/1 Init:Error 0 3h15m dev-sdc-sdc-be-config-backend-tm9lr 0/1 Init:Error 0 139m dev-sdc-sdc-be-config-backend-vlrfh 0/1 Init:Error 0 74m dev-sdc-sdc-be-config-backend-vpwjh 0/1 Init:Error 0 58m dev-sdc-sdc-be-config-backend-w2gsv 0/1 Init:Error 0 163m dev-sdc-sdc-be-config-backend-z729n 0/1 Init:Error 0 174m dev-sdc-sdc-be-config-backend-zxmkm 0/1 Init:Error 0 10m dev-sdc-sdc-cs-config-cassandra-g7cw9 0/1 Error 0 3h25m dev-sdc-sdc-cs-config-cassandra-jnmfz 0/1 Completed 0 3h22m dev-sdc-sdc-dcae-be-6dd4df6cf6-hs4rz 0/2 Init:0/2 20 3h25m dev-sdc-sdc-dcae-be-tools-5v9fw 0/1 Init:Error 0 76m dev-sdc-sdc-dcae-be-tools-6xwlh 0/1 Init:Error 0 108m dev-sdc-sdc-dcae-be-tools-7r2s9 0/1 Init:Error 0 92m dev-sdc-sdc-dcae-be-tools-84mkb 0/1 Init:Error 0 3h5m dev-sdc-sdc-dcae-be-tools-8j9ns 0/1 Init:Error 0 59m dev-sdc-sdc-dcae-be-tools-8p4tk 0/1 Init:Error 0 3h25m dev-sdc-sdc-dcae-be-tools-b495n 0/1 Init:Error 0 124m dev-sdc-sdc-dcae-be-tools-bbznd 0/1 Init:Error 0 11m dev-sdc-sdc-dcae-be-tools-czmfx 0/1 Init:Error 0 152m dev-sdc-sdc-dcae-be-tools-dgxmv 0/1 Init:Error 0 27m dev-sdc-sdc-dcae-be-tools-j96jb 0/1 Init:Error 0 3h15m dev-sdc-sdc-dcae-be-tools-jzp4w 0/1 Init:Error 0 139m dev-sdc-sdc-dcae-be-tools-k75p6 0/1 Init:Error 0 163m dev-sdc-sdc-dcae-be-tools-pw4rq 0/1 Init:Error 0 174m dev-sdc-sdc-dcae-be-tools-v2wjt 0/1 Init:Error 0 43m dev-sdc-sdc-dcae-dt-6f669b4d4c-vct6p 0/2 Init:0/2 20 3h25m dev-sdc-sdc-dcae-fe-684f4495b7-m2575 0/2 Init:0/2 20 3h25m dev-sdc-sdc-dcae-tosca-lab-74cf68f5ff-xwclc 0/2 Init:0/1 20 3h25m dev-sdc-sdc-es-76c4b9f57b-mjnsw 1/1 Running 0 3h25m dev-sdc-sdc-es-config-elasticsearch-sbnwx 0/1 Completed 0 3h25m dev-sdc-sdc-fe-6577f8fb4f-xdx5t 0/2 Init:1/2 20 3h25m dev-sdc-sdc-kb-559976fd47-56kfw 1/1 Running 0 3h25m dev-sdc-sdc-onboarding-be-84cc8bb944-gkm94 1/2 Running 39 3h25m dev-sdc-sdc-onboarding-be-cassandra-init-ggqqr 0/1 Completed 0 3h25m dev-sdc-sdc-wfd-be-56c7747ffc-4dkg2 1/1 Running 0 3h25m dev-sdc-sdc-wfd-be-workflow-init-9spc9 0/1 Completed 0 3h25m dev-sdc-sdc-wfd-fe-c487c8b75-sjpq9 2/2 Running 0 3h25m
Attached two console log files (failed.job.log and complete.job.log) for job sdc-cs-config-cassandra:
dev-sdc-sdc-cs-config-cassandra-g7cw9 0/1 Error 0 3h26m dev-sdc-sdc-cs-config-cassandra-jnmfz 0/1 Completed 0 3h23m
The job container failed and returned error in the first try, then the job was tried 2nd time. The 2nd time apparently did not succeed either because later on sdc-onboarding-be-cassandra-init found missing keyspace, however the job was complete. The job should not complete in this case because the keyspace is still missing in Cassandra.
Added console log in onboarding-be-cassandra-init.log for pod sdc-onboarding-be-cassandra-init. This pod has a lot of error in console log, it should try instead of complete.