Uploaded image for project: 'ONAP Operations Manager'
  1. ONAP Operations Manager
  2. OOM-3041

Bootstrap broker dev-strimzi-kafka-bootstrap:9092 (id: -1 rack: null) disconnected

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • Montreal Release
    • Kohn Release
    • None
    • None
    • Any kubernetes environment with OOM installation, Strimzi, AAF and Policy enabled in values.yaml

    • Hide

      There is an issue observed with continuous logs statements stating "Bootstrap broker dev-strimzi-kafka-bootstrap:9092 (id: -1 rack: null) disconnected" with useStrimziKafka as true in oom policy.

      How to reproduce:

      -> Build OOM with useStrimziKafka as true

      -> Set kafka properties with security.protocol: SASL_PLAINTEXT, and jaas.config set to JAASLOGIN secret generated, example as below

           topicSources:
              -
                topic: policy-acruntime-participant
                servers:
                  - dev-strimzi-kafka-bootstrap:9092
                topicCommInfrastructure: kafka
                fetchTimeout: 15000
                useHttps: true
                additionalProps:
                  group-id: policy-group
                  key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
                  value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
                  partition.assignment.strategy: org.apache.kafka.clients.consumer.RoundRobinAssignor
                  enable.auto.commit: false
                  auto.offset.reset: earliest
                  security.protocol: SASL_PLAINTEXT
                  properties.sasl.mechanism: SCRAM-SHA-512
                  properties.sasl.jaas.config: ${JAASLOGIN}

            topicSinks:
              -
                topic: policy-acruntime-participant
                servers:
                  - dev-strimzi-kafka-bootstrap:9092
                topicCommInfrastructure: kafka
                useHttps: true
                additionalProps:
                  key.serializer: org.apache.kafka.common.serialization.StringSerializer
                  value.serializer: org.apache.kafka.common.serialization.StringSerializer
                  acks: 1
                  retry.backoff.ms: 150
                  retries: 3
                  security.protocol: SASL_PLAINTEXT
                  properties.sasl.mechanism: SCRAM-SHA-512
                  properties.sasl.jaas.config: ${JAASLOGIN}

      -> Bring up any policy component with above config, so that communcation will be over Kafka instead of on Dmaap

      -> Kafka class initialization will be good, but while consuming or publishing to the topic, continues errors are observed with bootstrap server disconnected.

      Info from log: 

      [2022-09-27T16:51:42.904+00:00|INFO|network|pool-2-thread-1] [OUT|KAFKA|policy-acruntime-participant]
      {"state":"UNKNOWN","healthStatus":"UNKNOWN","participantStatistics":{"participantId":

      {"name":"org.onap.PM_Policy","version":"1.0.0"}

      ,"timeStamp":"2022-09-27T16:51:42.901870WN","healthStatus":"UNKNOWN","eventCount":0,"lastExecutionTime":0,"averageExecutionTime":0.0,"upTime":0,"lastEnterTime":0,"lastStart":0},"participantDefinitionUpdates":[],"tionInfoList":[],"messageType":"PARTICIPANT_STATUS","messageId":"f4f27649-1dc2-41ef-a57d-e70dd761a6d7","timestamp":"2022-09-27T16:51:42.902235Z","participantType":{"name":"lamp.acm.PolicyParticipant","version":"2.3.1"},"participantId":{"name":"org.onap.PM_Policy","version":"1.0.0"}}
      [2022-09-27T16:51:42.987+00:00|WARN|NetworkClient|KAFKA-source-policy-acruntime-participant] [Consumer clientId=consumer-ca495ee9-47ad-40a4-9615-64eeebc414e1-2, groupId=ca4615-64eeebc414e1] Bootstrap broker dev-strimzi-kafka-bootstrap:9092 (id: -1 rack: null) disconnected
      [2022-09-27T16:51:43.208+00:00|WARN|NetworkClient|kafka-producer-network-thread | producer-1] [Producer clientId=producer-1] Bootstrap broker dev-strimzi-kafka-bootstrap:90null) disconnected
      [2022-09-27T16:51:43.390+00:00|WARN|NetworkClient|KAFKA-source-policy-acruntime-participant] [Consumer clientId=consumer-ca495ee9-47ad-40a4-9615-64eeebc414e1-2, groupId=ca4615-64eeebc414e1] Bootstrap broker dev-strimzi-kafka-bootstrap:9092 (id: -1 rack: null) disconnected

       

      PODs in the namespace:

      root@xerces-master-control-loop-02-nfs:~# kubectl get po -n onap
      NAME                                             READY   STATUS             RESTARTS   AGE
      dev-aaf-cass-7c4d46d696-9w7bs                    1/1     Running            0          23h
      dev-aaf-cm-8465b877c5-mx7m8                      1/1     Running            0          23h
      dev-aaf-fs-586f88c9c-k82g6                       1/1     Running            0          23h
      dev-aaf-gui-7bc5848fb9-d8grx                     1/1     Running            0          23h
      dev-aaf-locate-f858868cc-s5mwd                   1/1     Running            0          23h
      dev-aaf-oauth-7fd855799c-9mq7b                   1/1     Running            0          23h
      dev-aaf-service-6dc6fcc676-ldpjn                 1/1     Running            0          23h
      dev-aaf-sms-64b97f545f-tvhgv                     1/1     Running            0          23h
      dev-aaf-sms-preload-8gpdp                        0/1     Completed          0          23h
      dev-aaf-sms-quorumclient-0                       1/1     Running            0          23h
      dev-aaf-sms-quorumclient-1                       1/1     Running            0          23h
      dev-aaf-sms-quorumclient-2                       1/1     Running            0          23h
      dev-aaf-sms-vault-0                              2/2     Running            0          23h
      dev-policy-apex-pdp-0                            1/1     Running            0          20h
      dev-policy-api-8575c5d54c-z9zk2                  1/1     Running            0          20h
      dev-policy-clamp-ac-http-ppnt-7b45db6878-fkh92   1/1     Running            0          20h
      dev-policy-clamp-ac-k8s-ppnt-6fbd99fb6c-bh6mx    1/1     Running            0          20h
      dev-policy-clamp-ac-pf-ppnt-7776fdc857-hqzl5     1/1     Running            1          20h
      dev-policy-clamp-be-d45949c55-kpdc2              2/2     Running            0          20h
      dev-policy-clamp-galera-config-qcf7z             0/1     Completed          0          20h
      dev-policy-clamp-runtime-acm-758fd6565c-vbtr8    0/1     CrashLoopBackOff   325        20h
      dev-policy-distribution-7887c686c6-976m4         1/1     Running            0          20h
      dev-policy-drools-pdp-0                          1/1     Running            0          20h
      dev-policy-galera-config-2wfhb                   0/1     Completed          0          20h
      dev-policy-gui-6b7dbf97b-g5wwc                   2/2     Running            0          20h
      dev-policy-mariadb-0                             1/1     Running            0          20h
      dev-policy-pap-575bff74d9-r67hr                  1/1     Running            0          20h
      dev-policy-xacml-pdp-678b66b997-kmxn4            1/1     Running            0          20h
      dev-strimzi-entity-operator-79dd84b4ff-dwqqt     3/3     Running            0          23h
      dev-strimzi-kafka-0                              1/1     Running            0          23h
      dev-strimzi-kafka-1                              1/1     Running            0          23h
      dev-strimzi-kafka-2                              1/1     Running            0          23h
      dev-strimzi-zookeeper-0                          1/1     Running            0          23h
      dev-strimzi-zookeeper-1                          1/1     Running            0          23h
      dev-strimzi-zookeeper-2                          1/1     Running            0          23h

      Show
      There is an issue observed with continuous logs statements stating "Bootstrap broker dev-strimzi-kafka-bootstrap:9092 (id: -1 rack: null) disconnected" with useStrimziKafka as true in oom policy. How to reproduce: -> Build OOM with useStrimziKafka as true -> Set kafka properties with security.protocol: SASL_PLAINTEXT, and jaas.config set to JAASLOGIN secret generated, example as below      topicSources:         -           topic: policy-acruntime-participant           servers:             - dev-strimzi-kafka-bootstrap:9092           topicCommInfrastructure: kafka           fetchTimeout: 15000           useHttps: true           additionalProps:             group-id: policy-group             key.deserializer: org.apache.kafka.common.serialization.StringDeserializer             value.deserializer: org.apache.kafka.common.serialization.StringDeserializer             partition.assignment.strategy: org.apache.kafka.clients.consumer.RoundRobinAssignor             enable.auto.commit: false             auto.offset.reset: earliest             security.protocol: SASL_PLAINTEXT             properties.sasl.mechanism: SCRAM-SHA-512             properties.sasl.jaas.config: ${JAASLOGIN}       topicSinks:         -           topic: policy-acruntime-participant           servers:             - dev-strimzi-kafka-bootstrap:9092           topicCommInfrastructure: kafka           useHttps: true           additionalProps:             key.serializer: org.apache.kafka.common.serialization.StringSerializer             value.serializer: org.apache.kafka.common.serialization.StringSerializer             acks: 1             retry.backoff.ms: 150             retries: 3             security.protocol: SASL_PLAINTEXT             properties.sasl.mechanism: SCRAM-SHA-512             properties.sasl.jaas.config: ${JAASLOGIN} -> Bring up any policy component with above config, so that communcation will be over Kafka instead of on Dmaap -> Kafka class initialization will be good, but while consuming or publishing to the topic, continues errors are observed with bootstrap server disconnected. Info from log:   [2022-09-27T16:51:42.904+00:00|INFO|network|pool-2-thread-1] [OUT|KAFKA|policy-acruntime-participant] {"state":"UNKNOWN","healthStatus":"UNKNOWN","participantStatistics":{"participantId": {"name":"org.onap.PM_Policy","version":"1.0.0"} ,"timeStamp":"2022-09-27T16:51:42.901870WN","healthStatus":"UNKNOWN","eventCount":0,"lastExecutionTime":0,"averageExecutionTime":0.0,"upTime":0,"lastEnterTime":0,"lastStart":0},"participantDefinitionUpdates":[],"tionInfoList":[],"messageType":"PARTICIPANT_STATUS","messageId":"f4f27649-1dc2-41ef-a57d-e70dd761a6d7","timestamp":"2022-09-27T16:51:42.902235Z","participantType":{"name":"lamp.acm.PolicyParticipant","version":"2.3.1"},"participantId":{"name":"org.onap.PM_Policy","version":"1.0.0"}} [2022-09-27T16:51:42.987+00:00|WARN|NetworkClient|KAFKA-source-policy-acruntime-participant] [Consumer clientId=consumer-ca495ee9-47ad-40a4-9615-64eeebc414e1-2, groupId=ca4615-64eeebc414e1] Bootstrap broker dev-strimzi-kafka-bootstrap:9092 (id: -1 rack: null) disconnected [2022-09-27T16:51:43.208+00:00|WARN|NetworkClient|kafka-producer-network-thread | producer-1] [Producer clientId=producer-1] Bootstrap broker dev-strimzi-kafka-bootstrap:90null) disconnected [2022-09-27T16:51:43.390+00:00|WARN|NetworkClient|KAFKA-source-policy-acruntime-participant] [Consumer clientId=consumer-ca495ee9-47ad-40a4-9615-64eeebc414e1-2, groupId=ca4615-64eeebc414e1] Bootstrap broker dev-strimzi-kafka-bootstrap:9092 (id: -1 rack: null) disconnected   PODs in the namespace: root@xerces-master-control-loop-02-nfs:~# kubectl get po -n onap NAME                                             READY   STATUS             RESTARTS   AGE dev-aaf-cass-7c4d46d696-9w7bs                    1/1     Running            0          23h dev-aaf-cm-8465b877c5-mx7m8                      1/1     Running            0          23h dev-aaf-fs-586f88c9c-k82g6                       1/1     Running            0          23h dev-aaf-gui-7bc5848fb9-d8grx                     1/1     Running            0          23h dev-aaf-locate-f858868cc-s5mwd                   1/1     Running            0          23h dev-aaf-oauth-7fd855799c-9mq7b                   1/1     Running            0          23h dev-aaf-service-6dc6fcc676-ldpjn                 1/1     Running            0          23h dev-aaf-sms-64b97f545f-tvhgv                     1/1     Running            0          23h dev-aaf-sms-preload-8gpdp                        0/1     Completed          0          23h dev-aaf-sms-quorumclient-0                       1/1     Running            0          23h dev-aaf-sms-quorumclient-1                       1/1     Running            0          23h dev-aaf-sms-quorumclient-2                       1/1     Running            0          23h dev-aaf-sms-vault-0                              2/2     Running            0          23h dev-policy-apex-pdp-0                            1/1     Running            0          20h dev-policy-api-8575c5d54c-z9zk2                  1/1     Running            0          20h dev-policy-clamp-ac-http-ppnt-7b45db6878-fkh92   1/1     Running            0          20h dev-policy-clamp-ac-k8s-ppnt-6fbd99fb6c-bh6mx    1/1     Running            0          20h dev-policy-clamp-ac-pf-ppnt-7776fdc857-hqzl5     1/1     Running            1          20h dev-policy-clamp-be-d45949c55-kpdc2              2/2     Running            0          20h dev-policy-clamp-galera-config-qcf7z             0/1     Completed          0          20h dev-policy-clamp-runtime-acm-758fd6565c-vbtr8    0/1     CrashLoopBackOff   325        20h dev-policy-distribution-7887c686c6-976m4         1/1     Running            0          20h dev-policy-drools-pdp-0                          1/1     Running            0          20h dev-policy-galera-config-2wfhb                   0/1     Completed          0          20h dev-policy-gui-6b7dbf97b-g5wwc                   2/2     Running            0          20h dev-policy-mariadb-0                             1/1     Running            0          20h dev-policy-pap-575bff74d9-r67hr                  1/1     Running            0          20h dev-policy-xacml-pdp-678b66b997-kmxn4            1/1     Running            0          20h dev-strimzi-entity-operator-79dd84b4ff-dwqqt     3/3     Running            0          23h dev-strimzi-kafka-0                              1/1     Running            0          23h dev-strimzi-kafka-1                              1/1     Running            0          23h dev-strimzi-kafka-2                              1/1     Running            0          23h dev-strimzi-zookeeper-0                          1/1     Running            0          23h dev-strimzi-zookeeper-1                          1/1     Running            0          23h dev-strimzi-zookeeper-2                          1/1     Running            0          23h

          rameshiyer27 Ramesh Murugan Iyer
          liamfallon liamfallon
          Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

            Created:
            Updated:
            Resolved: