Hi David McBride,

As requested in the onap community discussion:https://lists.onap.org/g/onap-discuss/message/24358, herewith please find the initial issues and suggested action implemented.

Initial Problem:

We have brought up new ONAP with Jakarta version in our environment. While doing post configuration, we are facing issue during Netconf deployment.
As part of Netconf deployment, we are re-installing dev-sdnc. When trigger deploy sdnc command, we are facing strange issues that dev-cassandra, dev-mariadb-galera and dev-postgres are getting uninstalled.

root@onap04-helper-vm:~/oom/kubernetes# helm deploy dev-sdnc local/onap --namespace onap --verbose
v3.8.2
Use cache dir: /root/.local/share/helm/plugins/deploy/cache
9
0
0
0
fetching local/onap
history.go:56: [debug] getting history for release dev
upgrade.go:142: [debug] preparing upgrade for dev
upgrade.go:521: [debug] copying values from dev (v1) to new release.
upgrade.go:150: [debug] performing update for dev
upgrade.go:313: [debug] dry run for dev
waiting for dev-strimzi-entity-operator to be deployed
dev-strimzi-entity-operator found. Waiting for pod intialisation
release "dev-cassandra" uninstalled
W0311 09:42:47.425110 1794336 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
release "dev-mariadb-galera" uninstalled
release "dev-postgres" uninstalled
Release "dev-sdnc" does not exist. Installing it now.
NAME: dev-sdnc
LAST DEPLOYED: Sat Mar 11 09:42:48 2023
NAMESPACE: onap
STATUS: deployed
REVISION: 1
TEST SUITE: None
release "dev-strimzi" uninstalled
root@onap04-helper-vm:~/oom/kubernetes#

 

Steps Followed for re-installation of SDNC:

1) Status check:

root@onap04-helper-vm:~# kubectl get pods -n onap | grep sdnc
dev-sdnc-0                                                        1/1     Running                      0                  11d
dev-sdnc-ansible-server-7c56c56c65-5jvmq                          1/1     Running                      0                  11d
dev-sdnc-dbinit-job-clh78                                         0/1     Completed                    0                  11d
dev-sdnc-dgbuilder-7b5bdb7d5f-44rst                               1/1     Running                      0                  11d
dev-sdnc-dmaap-listener-9b8fb65f8-8fdds                           1/1     Running                      0                  11d
dev-sdnc-sdnrdb-init-job-dh4cs                                    0/1     Completed                    0                  11d
dev-sdnc-ueb-listener-6fc94fb6f7-mflkq                            1/1     Running                      0                  11d
dev-sdnc-web-86c55569f6-zvmrt                                     1/1     Running                      0                  11d
dev-so-sdnc-adapter-5b46f7f574-ndgqp                              1/1     Running                      0                  11d

2) Uninstall using below command
helm uninstall dev-sdnc -n onap

3) Remove the directory from nfs server
sudo rm -rf /dockerdata-nfs/dev/sdnc

4) Secret deletion
kubectl delete secret dev-sdnc-secret-0 -n onap

5) dev-sdnc deploy again
helm deploy dev-sdnc local/onap --namespace onap --verbose
Steps changed as per Fiachra Corcoran from ONAP discussion:

Step 5) from the above steps has requeted to change as below

helm install dev-sdnc local/sdnc --debug --namespace onap --create-namespace --set global.masterPassword=test --set global.aafEnabled=false

Current Issue:

After suggested above command used, dev-sdnc is re-deployed(without uninstalling the other pods mentioned in the initial issue).

But looks like secret is not created again as we deleted in the 4th step above. Due to which, after netconf deployement, pod is not coming up.

Cert status:
**
root@onap04-helper-vm:/home/helm/helm# kubectl get cert -n onap
NAME                         READY   SECRET                               AGE
cert-service-client-cert     True    oom-cert-service-client-tls-secret   11d
cert-service-server-cert     True    oom-cert-service-server-tls-secret   11d
dcae-ves-collector-5g-cert   True    dcae-ves-collector-5g-secret         33h
netconfclient-cert           True    netconfclient-secret                 7m57s
netconfserver-cert           True    netconfserver-secret                 7m57s
selfsigned-cert              True    cmpv2-ca-key-pair                    11d

Netconf status:

root@onap04-helper-vm:/home/helm/helm# kubectl -n onap get pod | grep netconf
netconf-server-956b6dbcc-258s6                                    0/1     ContainerCreating            0                 14m

Describe o/p:

  Normal   Scheduled    14m                    default-scheduler  Successfully assigned onap/netconf-server-956b6dbcc-258s6 to onap04-k8s-3
  Warning  FailedMount  14m (x3 over 14m)      kubelet            MountVolume.SetUp failed for volume "certstore" : [secret "dev-sdnc-secret-0" not found, references non-existent secret key: tls.key]
  Warning  FailedMount  12m                    kubelet            Unable to attach or mount volumes: unmounted volumes=[certstore], unattached volumes=[kube-api-access-jn4gh certstore prep-certs prep-models]: timed out waiting for the condition
  Warning  FailedMount  10m                    kubelet            Unable to attach or mount volumes: unmounted volumes=[certstore], unattached volumes=[certstore prep-certs prep-models kube-api-access-jn4gh]: timed out waiting for the condition
  Warning  FailedMount  3m28s (x2 over 5m43s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[certstore], unattached volumes=[prep-models kube-api-access-jn4gh certstore prep-certs]: timed out waiting for the condition
  Warning  FailedMount  70s (x2 over 7m57s)    kubelet            Unable to attach or mount volumes: unmounted volumes=[certstore], unattached volumes=[prep-certs prep-models kube-api-access-jn4gh certstore]: timed out waiting for the condition
  Warning  FailedMount  15s (x12 over 14m)     kubelet            MountVolume.SetUp failed for volume "certstore" : secret "dev-sdnc-secret-0" not found

 

Please help to resolve this current issue or kindly check our initial issue, is any alternate procedure we can follow?

Regards,

Sundar