Uploaded image for project: 'ONAP TSC'
  1. ONAP TSC
  2. TSC-25

Task Force to implement CD (Continuous Deployment)

    XMLWordPrintable

Details

    Description

      Meeting Videos: https://wiki.onap.org/display/DW/CD+-+Continuous+Deployment
      20181026: LF Ticket 62287 is tracking this request

      Meets 1230 EDT (GMT-5) Thu https://zoom.us/j/7939937123
      https://lists.onap.org/g/onap-discuss/topic/cd_task_force_tsc_25_meetings/29416249?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,29416249

        POCs
      Integration team: 2018
      gwu - full onap system 13 vms - freq: master once a day
      http://onapci.org/grafana/d/8cGRqBOmz/daily-summary?orgId=1
      https://github.com/garyiwu/onap-lab-ci

      Logging/OOM Team: 201710
      michaelobrien - partial onap system 1 vm - freq: hourly for 3 pod - 6 hours for full onap deploy 256G vm
      http://kibana.onap.info:5601/app/kibana#/dashboard/AWAtvpS63NTXK5mX2kuS
      http://jenkins.onap.info/job/oom-cd-master2-aws/
      http://jenkins.onap.info/job/oom-cd-master/
      https://git.onap.org/logging-analytics/tree/deploy
      GITLAB OOM mirror for 2nd CD pipeline
      https://gitlab.com/obriensystems/oom

      Orange Labs 201901
      https://gitlab.com/Orange-OpenSource/lfn/onap/onap_oom_automatic_installation/pipelines/44398686
      https://gitlab.com/Orange-OpenSource/lfn
      Working E2E demo: https://wiki.onap.org/display/DW/CD+-+Continuous+Deployment#CD-ContinuousDeployment-20190131-OrangeCDdemofromSylvainDesbureaux
      Orange demo from last week on https://wiki.onap.org/display/DW/CD+-+Continuous+Deployment#CD-ContinuousDeployment-20190131-OrangeCDdemofromSylvainDesbureaux

      example on https://gerrit.onap.org/r/#/c/77660/

      • flow:
      Existing flow
- gerrit commit on the oom repo for a particular component like so, aai - keyed by Issue-ID Jira
- helm-verify jjb jenkins job currently runs - reports +1/-1
- review is merged
- helm verify runs again on master
- no helm deploy

Proposed flow 1
- gerrit commit on the oom repo for a particular component like so, aai - keyed by Issue-ID Jira
- helm-verify jjb jenkins job currently runs - reports +1/-1
- manual magic word "run-helm-deploy" will kick in a helm-deploy jjb job that deploys robot and the particular pod to a 16-32g VM (preconfigured with rancher as a single node) - how? jenkins will run a remote ssh shell to a server using a cached key - a cd.sh script will need to be written - see the 2 pocs below already running 
- reports +1/-1 if healthcheck for that component passes after 20 min - parse the logs from jenkins
- scripts to bring up k8s/helm/docker - see links in comments

Caveats:
  3 types of tests (docker image tag, kubernetes chart/job/config changes) - docker image tag changes will require that the image is in nexus3 already - ideally only oom repo changes will be in phase 1

Proposed flow 2 later
- gerrit commit on the oom repo for a particular component like so, aai - keyed by Issue-ID Jira
- helm-verify jjb jenkins job currently runs - reports +1/-1
- same helm-install jjb job again is automatically triggered and reports back a +1/-1 after 20 min
- based on # of vms - jobs can be parallelized or batched (report -1/+1 to the batch - but only 1+ are at fault

      Task Force: Michael O'Brien, Gildas, Christophe Closset, Jessica, Jeremy,

      https://wiki.onap.org/display/DW/Development+Procedures+and+Policies?focusedCommentId=25437277#DevelopmentProceduresandPolicies-MagicWords
      Linux Foundation Ticket: 62287

      Use labs if they have public non-VPN access - https://wiki.onap.org/display/DW/Physical+Labs

        Use cases

      (essentially at a minimum the CD deploy does)

      kubectl get pods --all-namespaces
and
oom/kubernetes/robot$ sudo ./ete-k8s.sh onap health

      Ultimate Main Goal: prevent to merge code that has not been tested in a CD environment.

      Realistic code: test merged code in a CD environment.

      20181015 notes: glanilis michaelobrien ChrisC

      • when to run: only after a successfull helm verify (before code is merged)
        example https://jenkins.onap.org/job/oom-master-merge-helm/289/
        had a +1
      • Q: if the LF would like a separate CDBuilder job to run after or do as part of the current jobbuilder
        "Verified +1 ONAP Jobbuilder"
      • recommend a single VM deployment for 1 pod - not a full ONAP system yet
        (for example - the commit is under LOG-NNN
        so we run the command with a --set log.enabled=true - the rest of onap is false like 
        sudo helm install local/onap -n onap --namespace onap -f onap/resources/environments/disable-allcharts.yaml --set log.enabled=true

        example http://kibana.onap.info:5601/app/kibana#/dashboard/AWAtvpS63NTXK5mX2kuS

      • Timing should be <20 min for one pod to helm install and run healthcheck
      • Concurrency limit (as a result of multiple gerrit merges within a period) - ask for 4
      • LF: servers # and capacity (13 x 16Gb for full onap, 1x 16Gb for a particular component)
        (vCPU limits are not enforced yet - but in the future a component like pomba with 11 containers using 10G ram will need 2 x 11 for example vCPUs) - current vCPU limit is 2 - a fraction of a vCPU (10% of a core per container) - full system between 32 and 64 cores
      • HD need 40G per VM for K8S system and 10G for /dockerdata_nfs - a full deploy is 90G on master and 50g on each cluster VM's
        Numbers: full 13x16 = 700G total HD , one component = 90G (single VM)
      • access (public not VPN) from jenkins jobs
      • pilot project
      • LF: work on JJB to continue past existing helm-verify
      • LF: to see the outcome of the particular helm install - like we get the +1 in for example https://gerrit.onap.org/r/#/c/70486/
      • optional nice to have gwu kibana view like in http://onapci.org/grafana/d/8cGRqBOmz/daily-summary?orgId=1

      Community
      https://docs.opnfv.org/en/latest/submodules/releng-xci/docs/xci-overview.html#xci-overview
      investigate ansible based https://zuul-ci.org/
      https://jenkins.rook.io/blue/organizations/jenkins/rook%2Frook/detail/master/559/pipeline/

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. TSC-92 Casablanca RM branch stability before 3.0.1-ONAP tagging - helm deploy/hc before merging

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Task - A task that needs to be done. TSC-42 Dublin Pair-Wise Activities Proposal

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. COMMON-27 ONAP Docker images and base images should be ONAP controlled

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Open

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-310 Gitlab CI/CD via OOM repo mirroring

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Task - A task that needs to be done. LOG-331 Oneclick Kubernetes install for ARM processor backed A1 AWS VMs

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OOM-1563 All Charts do no honor global image pull Policy

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TSC-79 LF Nexus3 routing slowdown starting 20181217 - 80-100x slower download times totalling 120+ hours - using nexus3/4.onap.cloud proxy for now

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. DCAEGEN2-1067 Deploying DCAEGEN2 late in a sequenced deployment will cause the secondary cloudify orchestration pods DEP* to fail

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. DCAEGEN2-1068 DCAEGEN2 failed redeployment leaves cloudify orchestrated dev-dcae namespace pods up

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. INT-586 Improve stability of automated tests

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Task - A task that needs to be done. LOG-326 CD: OOM automated deployment script

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-806 upgrade rancher/kubernetes from 1.6.18/1.10 to 1.6.22|23/1.11 to align with INT-586 71375 29 Oct upgrade

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Task - A task that needs to be done. LOG-890 Retrofit cd.sh install with the docker manifest to values.yaml override script

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Sub-task - The sub-task of the issue LOG-924 Kubernetes chart dependencies - make all 105 in 87 files conditional - post yaml for cd

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Sub-task - The sub-task of the issue LOG-925 Adjust deployment dependency list - msb must be before dcaegen2

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OOM-1089 DCAE pods are not getting purged

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OOM-1580 HELM undeploy may hang on Terminating pods that need a --purge

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Task - A task that needs to be done. OOM-1586 Add --force --grace-period=0 delete pod capability to cd.sh script

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TSC-86 Lock down docker image tag name source of truth - oom values.yaml or integration repo manifest - A: both but manifest is the source

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. TSC-101 Dublin kubernetes deployment stability tracking

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed
          is duplicated by

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LOG-300 CD: OOM framework for continuous E2E deploy validation of tagged commit/merge trigger docker snapshots

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-992 RKE 0.16 / Docker 18.06 for ONAP installation - migrate from Rancher for Dublin - script support

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-266 F2F: ONAP CI/CD using OOM Kubernetes

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Task - A task that needs to be done. LOG-323 CD: Oneclick Amazon AWS Cloudformation template for Kubernetes VM install and OOM deployment

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-895 Upgrade Rancher to 1.6.25 to address CVE-2018-1002105 and move to Kubernetes 1.11.5 (server side)

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Task - A task that needs to be done. OOM-1539 CVE-2018-1002105: proxy request handling in kube-apiserver can leave vulnerable TCP connections · Issue #71411 · kubernetes/kubernetes · GitHub

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LOG-300 CD: OOM framework for continuous E2E deploy validation of tagged commit/merge trigger docker snapshots

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Task - A task that needs to be done. TSC-45 Dublin Prioritization Criteria

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Task - A task that needs to be done. TSC-53 TSC "Must Have" Dublin Requirements - E2E Process Automation

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Task - A task that needs to be done. TSC-54 TSC "Must Have" Dublin Requirements - Documentation

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Task - A task that needs to be done. TSC-55 TSC "Must Have" Dublin Requirements - Continuous Integration Approach

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Task - A task that needs to be done. TSC-58 Dublin Toolchain Improvement

          • Highest - Catastrophic defect that causes total failure of the software or unrecoverable data loss with no available workaround. Complete shut-down of the process, nothing can proceed further. Blocks testing of the product / feature. Must be fixed in the next build.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LOG-296 Provide user friendly deployment profiles for all component subtrees of ONAP

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SDNC-481 SDNC fails health check after install

          • High - Severely impaired functionality. Erratic performance and reduced system availability. It may cause major components or features of the system to be unavailable although certain parts of the system remain functional. A workaround may exist but its use is unsatisfactory or at a high time cost (manual intervention required). Must be fixed in any of the upcoming builds and should be included in the current release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. CLAMP-234 CLAMP invalid image name on 20181015:1800 full deployment - clamp-dashboard-kibana:3.0-STAGING-latest

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. LOG-305 Docker images must merge/build on each commit not daily

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Sub-task - The sub-task of the issue OOM-1549 Dashboard OOM deployment

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. OOM-1568 Define configuration overrides for target deployment environment

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SDC-1836 Image pull backoff 20181015 on onap/workflow-frontend:1.3.0-SNAPSHOT

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. TSC-70 ONAP and OPFNV collaboration - specifically ARM based AWS VMs

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. TSC-75 CVE security governance of deployment undercloud (Docker, Kubernetes, Helm, Rancher) - propose new CLM job

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. VVP-130 DEV mode dev.yaml override for all remaining ReplicaSet counts still above 1

          • Medium - Failure of non-critical aspects of the system but results in an inconvenient situation. It causes some undesirable behavior, but the system is still functional. There is a reasonably satisfactory workaround. It is still a valid defect that should be corrected May be fixed after the release or in the next release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. DOC-389 Trivial documentation issues

          • Low - It won't cause any major break-down of the system. It does not result in the termination of services, does not impact usability of the system and the desired results can be easily obtained by a workaround. Fix can be deferred until after more serious defect have been fixed.
          • Closed
          links to

          Web Link http://jenkins.onap.info/job/oom-cd-master/

          Web Link http://kibana.onap.info:5601/app/kibana#/dashboard/AWAtvpS63NTXK5mX2kuS

          Web Link https://docs.opnfv.org/en/latest/submodules/releng-xci/docs/xci-overview.html#xci-overview

          Web Link https://jenkins.onap.org/sandbox/

          Web Link https://lists.onap.org/g/onap-discuss/topic/cd_task_force_tsc_25_meetings/29001640?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,29001640

          Web Link https://lists.onap.org/g/onap-discuss/topic/onap_master_or_onap_master/29668857?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,29668857

          Web Link https://rancher.com/docs/rancher/v1.6/en/hosts/#supported-docker-versions

          Web Link https://zuul-ci.org/

          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Wiki Page

          Wiki Page Loading...

          Wiki Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              michaelobrien Michael O'Brien
              Katel34 Catherine Lefevre
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: