Uploaded image for project: 'Music'
  1. Music
  2. MUSIC-147

Cassandra-job is failing due to cqlsh timeout

XMLWordPrintable

      1. cassandra-job keeps creating new pods and fails due to connection timeout. I tested by sleeping the job and manually running runcql.sh within the pod.
      2. Another issue is that cassandra-job starts running before the cassandra cluster is up fully. Since we are using hostname to connect cqlsh, the dns resolution might return NOT READY replica IPs and will timeout in that case. 
      3. Tomcat starts up before cassandra-job finishes. So the start up is throwing error. https://gist.github.com/dileepranganathan/68aab343f6e9a09f31bfaa4884d1cf80

      Logs for Issue 1

       

      Connection error: ('Unable to connect to any servers', {'10.244.1.86': AuthenticationFailed('Failed to authenticate to 10.244.1.86: Error from server: code=0100 [Bad credentials] message="Provided username cassandra and/or password are incorrect"',), '10.244.1.102': AuthenticationFailed('Failed to authenticate to 10.244.1.102: Error from server: code=0100 [Bad credentials] message="Provided username cassandra and/or password are incorrect"',), '10.244.1.100': AuthenticationFailed('Failed to authenticate to 10.244.1.100: Error from server: code=0100 [Bad credentials] message="Provided username cassandra and/or password are incorrect"',)})

            arthurdent3 arthurdent3
            dileep.ranganathan dileep.ranganathan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: