Uploaded image for project: 'Service Design and Creation'
  1. Service Design and Creation
  2. SDC-2374

SDC appears to lose connectivity to Cassandra and Titan intermittently

XMLWordPrintable

      During integration stability test, we experience test failure due to SDC losing connectivity to Cassandra and Titan intermittently. The issue can happen even when there is no ping loss detected from SDC-BE to Cassandra.

      You can find error messages in SDC-BE log: "Components Availability Filter Failed - ES/Cassandra is DOWN", "o.o.s.be.dao.titan.TitanGraphClient RequestId=null ErrorCategory=ERROR ServiceName=titanHealthCheck ErrorCode=200 TargetEntity=titanHealthCheck Titan Health Check Failed. java.util.concurrent.TimeoutException"

       

      Timed out to Cassandra DB error can be also found in sdc-onboarding-be error log: 

      2019-06-19T00:03:15.265Z [qtp215145189-91] INFO o.o.s.h.r.services.HealthCheckImpl ResponseCode=200 RequestId=9b532ed9-d47d-4d4e-9172-a6bed88e0182 Server=dev-sdc-sdc-onboarding-be-84cc8bb944-cs6tq InstanceId=18d4dd1f-23ca-4b93-8e68-f2b6e3344e60 ElapsedTime=7424 EndTimestamp=2019-06-19T00:03:15.265Z PartnerName=cs0008 StatusCode=COMPLETE ServerIpAddress=10.42.14.14 ServiceName=GET: /onboarding-api/v1.0/healthcheck ClientIpAddress=127.0.0.1 ResponseDescription=OK BeginTimestamp=2019-06-19T00:03:07.841Z
2019-06-19T00:03:29.862Z [qtp215145189-83] ERROR c.a.z.c.l.i.ZusammenSLF4JLoggerFactoryImpl RequestId=69a6d8d7-2ccd-4279-ae95-d58986a876b4 Server=dev-sdc-sdc-onboarding-be-84cc8bb944-cs6tq InstanceId=18d4dd1f-23ca-4b93-8e68-f2b6e3344e60 PartnerName=cs0008 ServerIpAddress=10.42.14.14 ServiceName=GET: /onboarding-api/v1.0/healthcheck [cassandra.onap/10.42.4.3:9042] Timed out waiting for server response
com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.42.4.3:9042] Timed out waiting for server response
 at com.datastax.driver.core.exceptions.OperationTimedOutException.copy(OperationTimedOutException.java:44)
 at com.datastax.driver.core.exceptions.OperationTimedOutException.copy(OperationTimedOutException.java:26)
 at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)

       

      What we ask for here is SDC can retry several times when DB server times out. 

       

        1. sdc-be.error.log
          7.96 MB
        2. sdc-onboardng-be.error.log
          9.48 MB

            bdfreeman1421 bdfreeman1421
            xuyang11 xuyang11
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: