Uploaded image for project: 'Active and Available Inventory'
  1. Active and Available Inventory
  2. AAI-2079

aai-traversal and aai container failure to deploy issues in casablanca 3.0.0-ONAP

    XMLWordPrintable

Details

    Description

      known (PTL 20190114) issues with
      onap onap-aai-aai-59686b87c7-zzddq 0/1 Init:0/1 0 15d
      onap onap-aai-aai-traversal-5b496d986c-vb876 1/2 CrashLoopBackOff 2821 15d
      onap onap-aai-aai-traversal-update-query-data-fcn4h 0/1 Init:0/1 1834 15d

      Was meaning on raising this for the last couple weeks - if there is a duplicate - close/dup it.

      https://lists.onap.org/g/onap-discuss/topic/onap_deployment_issues/28985087?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28985087

      in all systems prerequisites are up (aaf for example)

2 week old 3.0.0-ONAP system - azure 1 x 256g node
onap          onap-aai-aai-59686b87c7-zzddq                                 0/1       Init:0/1           0          15d
onap          onap-aai-aai-babel-5bc54bfdf5-7bd27                           2/2       Running            0          15d
onap          onap-aai-aai-cassandra-0                                      1/1       Running            2          15d
onap          onap-aai-aai-champ-7f8c7cfffd-472fv                           1/2       Running            0          15d
onap          onap-aai-aai-data-router-54bfcf8c94-djdgd                     2/2       Running            0          15d
onap          onap-aai-aai-elasticsearch-6d7f96f9b8-9kzpv                   1/1       Running            0          15d
onap          onap-aai-aai-gizmo-79dd84956c-kfc94                           2/2       Running            0          15d
onap          onap-aai-aai-graphadmin-57fc47c48c-6v6wd                      2/2       Running            0          15d
onap          onap-aai-aai-graphadmin-create-db-schema-sgmvt                0/1       Completed          0          15d
onap          onap-aai-aai-modelloader-7784d7d649-w44nd                     2/2       Running            0          15d
onap          onap-aai-aai-resources-cbfc9546d-vb5vj                        2/2       Running            0          15d
onap          onap-aai-aai-search-data-65ff94556f-6pgwb                     2/2       Running            0          15d
onap          onap-aai-aai-sparky-be-6d489d4dc9-fwcq4                       0/2       Init:0/1           0          15d
onap          onap-aai-aai-spike-5dfcdc9cb8-zgcwp                           2/2       Running            0          15d
onap          onap-aai-aai-traversal-5b496d986c-vb876                       1/2       CrashLoopBackOff   2821       15d
onap          onap-aai-aai-traversal-update-query-data-fcn4h                0/1       Init:0/1           1834       15d

3 day old - 13 node openlab cluster
onap          onap-aai-aai-babel-696d964845-c8q8m                            2/2       Running            0          3d
onap          onap-aai-aai-cassandra-0                                       1/1       Running            0          3d
onap          onap-aai-aai-champ-5f7f94bb84-kjchk                            1/2       Running            0          3d
onap          onap-aai-aai-d74b75758-pzx56                                   0/1       Init:0/1           0          3d
onap          onap-aai-aai-data-router-7f6d895dc4-jph4h                      2/2       Running            4          3d
onap          onap-aai-aai-elasticsearch-6d7f96f9b8-dn8rr                    1/1       Running            6          3d
onap          onap-aai-aai-gizmo-5d9c884b9-cdv89                             2/2       Running            9          3d
onap          onap-aai-aai-graphadmin-99648f689-nlr46                        2/2       Running            0          3d
onap          onap-aai-aai-graphadmin-create-db-schema-lkdsv                 0/1       Completed          0          3d
onap          onap-aai-aai-modelloader-8b659f647-6hzz6                       2/2       Running            0          3d
onap          onap-aai-aai-resources-6dc4766986-hgl5z                        2/2       Running            0          3d
onap          onap-aai-aai-search-data-7dbb99df5f-xpfdr                      2/2       Running            0          3d
onap          onap-aai-aai-sparky-be-86db4b4fdf-4qfjf                        0/2       Init:0/1           0          3d
onap          onap-aai-aai-spike-7bb658d79b-dqmzr                            2/2       Running            0          3d
onap          onap-aai-aai-traversal-6ffbcb7dbf-zkc22                        1/2       CrashLoopBackOff   586        3d
onap          onap-aai-aai-traversal-update-query-data-8kj7w                 0/1       Init:0/1           493        3d

yesterday 3.0.0-ONAP - 4 x 32g openlab cluster
onap          onap-aai-aai-58c94d46c7-2lj8r                           0/1       Init:0/1           0          38m
onap          onap-aai-aai-babel-696d964845-cdhrl                     2/2       Running            0          5h
onap          onap-aai-aai-cassandra-0                                1/1       Running            0          5h
onap          onap-aai-aai-champ-5f7f94bb84-jkzvt                     1/2       Running            0          5h
onap          onap-aai-aai-data-router-7f6d895dc4-j95fz               2/2       Running            16         5h
onap          onap-aai-aai-elasticsearch-6d7f96f9b8-sn44j             1/1       Running            0          5h
onap          onap-aai-aai-gizmo-5d9c884b9-p2cxr                      2/2       Running            0          5h
onap          onap-aai-aai-graphadmin-99648f689-v88cs                 2/2       Running            0          5h
onap          onap-aai-aai-graphadmin-create-db-schema-m6sz8          0/1       Completed          0          5h
onap          onap-aai-aai-modelloader-8b659f647-p4f89                2/2       Running            0          5h
onap          onap-aai-aai-resources-6dc4766986-m6gvs                 2/2       Running            0          5h
onap          onap-aai-aai-search-data-7dbb99df5f-x6fll               2/2       Running            0          5h
onap          onap-aai-aai-sparky-be-86db4b4fdf-9fznn                 0/2       Init:0/1           0          5h
onap          onap-aai-aai-spike-7bb658d79b-cbhbs                     2/2       Running            0          5h
onap          onap-aai-aai-traversal-6ffbcb7dbf-rbnf6                 1/2       Running            34         5h
onap          onap-aai-aai-traversal-update-query-data-qlhxp          0/1       Init:0/1           32         5h

      quoting onap-discuss list email
      The deployment script just sequences the deploys in order – if a pod does not come up it is now due to the pod itself – nothing to due with the deployment/script itself.

      We discussed this in the last PTL meet on Monday

      https://wiki.onap.org/display/DW/PTL+2019-01-14

      AAI traversal is blocking the main aai pod – for the past 3-5 weeks at least – I would bring it up with the AAI team and add any triage you have.

      I would raise a jira myself on this one like the others I have in the past – I have just not gone through the details yet (too many other parallel issues right now) – the integration team is aware of this as well and have flagged it.

      Also is your AAF up – as there is a conditional check on aaf-locate if aaf is enabled – the nice thing is if you only bring up aai – the check will not block you - a good pattern.

      https://jira.onap.org/browse/LOG-924

      https://git.onap.org/oom/tree/kubernetes/aai/charts/aai-traversal/templates/job.yaml#n42

      spec: initContainers:
      {{ if eq .Values.global.aafEnabled true }}

      • --container-name
      • aaf-locate
        {{ end }}

      Attachments

        Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-898 ONAP Deployment Resiliency changes to deploy without POD failures

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TSC-92 Casablanca RM branch stability before 3.0.1-ONAP tagging - helm deploy/hc before merging

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. OOM-1556 OOM AAI fails to deploy in Dublin

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Task - A task that needs to be done. AAI-2068 Upversion for Casablanca Maintenance Release in OOM

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. AAI-2076 A&AI healthcheck timeout

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed
          links to

          Web Link https://lists.onap.org/g/onap-discuss/topic/onap_deployment_issues/28985087?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28985087

          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jimmydot James Forsyth
              michaelobrien Michael O'Brien
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: