-
Epic
-
Resolution: Done
-
Highest
-
None
-
OOM CD framework for continuous E2E deploy validation of component commits
Issue: we currently build docker images daily - not by gerrit merge per developer change - so we can run CD on each commit
TODO: look at out of the box systems like gitlab, gocd, bamboo...
POC running now that can consume a tagged docker manifest for a commit under test
http://jenkins.onap.info/job/oom-cd/
against http://beijing.onap.info:8880/env/1a7/kubernetes/dashboard
tracked on http://kibana.onap.info:5601/app/kibana#/dashboards?_g=()
Manifests
https://onap.readthedocs.io/en/latest/release/release-manifest.html
existing flow (actual)
- gerrit review commits - partial-CI runs - JobBuilder marks -1 (compile failure, sonar failure)
- gerrit review commits - partial-CI runs - JobBuilder marks +1, committer +2 merges to master, (later daily docker merge build tagged docker blindly)
- no way to know whether that commit degraded ONAP
proposed flow (expected) - two phase JobBuilder +1 process
- gerrit review commits - partial-CI runs - JobBuilder marks -1 (compile failure, sonar failure)
- gerrit review commits - partial-CI runs - JobBuilder marks +1, (what we add below)
kick in docker merge immediately, we tag the docker image, we adjust the manifest for this review
run extra CDBuilder that runs CD deploy of OOM using above tagset manifest, healthcheck, vFW
report failure - marks gerrit review as -1 - no tag set published for this failed commit
or report pass - marks gerrit review as +1
jenkins now retags "latest" to the the docker built above for the review (do not blindly tag "latest" to the last docker build whether it passes/fails CD)
Tagset for that build becomes the latest stable master build of ONAP (as in an OOM deploy will run not from master tip but from a tagged set that omits the last breaking change)
committer +2 merges to master, (docker merge and tagging should not be required as long as master has not moved during the 1 hour CD build)
(todo: developer can destabilize this if the submit order is wrong - ie: dcae will add an appc rest api, then checkin dcae first and appc second)
Note: Align with OOM-460 upcoming config retrofit
Issues: (workarounds for both)
- out of order commits on cross-project commits -
- repo drift during the CD build
We also need a way to do a full deployment of OOM containing the change in a tagged docker image (from the CI build) - OOM will need either a script to retrofit all the values.yaml docker version tags or a way to pull in an automated manifest that has the entire ONAP tag set.
The CD build with heatlhcheck and minimal (vFirewall run - as much as is automated) - is then run on this single commit trigger or set of triggers for the past hour - later the JobBuilder marks the gerrit reviews as passed/failed based on the CD run
A CD poc running on AWS EC2 is currently up that does part of this (jenkins, elk stack , cd server to run OOM) - but it needs to be finished and migrated into the LF infrastructure
-------OOM-393OOM-540
Proposal
1 - developer gerrit merge occurs
2 - CI system saves git hash - INT-359
3 - tag git based on when build starts (not when it finishes) - INT-189
4 - build docker image per merge - not daily
5 - insert artifact into docker image
6 - tag docker image with TBD timestamp standard tag
https://nexus3.onap.org/#browse/browse/components:docker.snapshot
currently we have non-standard tags
aai
| version | 1.2.0-SNAPSHOT-STAGING-20180121T124253 |
ccsdk 0.2.0-SNAPSHOT-STAGING-180117-133202
0.1.1-STAGING-180121-124346
appc
| Version | 1.3.0-STAGING-20180105T120937 |
7 - run CD system - after 1 hour collect results on running vFW for example
8 - tag gerrit review -1 or +1 via JobBuilder for CD results - INT-366
9 - If success CD: mark tag set including new blind tag from earlier gerrit merge per component
9b - tag latest only if succesfull
10 - publish tag set in dynamic manifest
https://gerrit.onap.org/r/gitweb?p=integration.git;a=blob;f=version-manifest/src/main/resources/docker-manifest.csv;h=35e992adc0678ccd98328d33c9a5ffc88dbb4dfa;hb=refs/heads/master
11 - retrofit manifest into all values.yaml files in OOM (replace v1.1.0 below - per component)
https://git.onap.org/oom/tree/kubernetes/aai/values.yaml?h=amsterdam
Acceptance Critieria
- jobbuilder can mark a gerrit review CD pass/fail - just like we do for compile CI jobs
- kibana history is available for CD jobs
- run on any branch or tag
- publicly visible server results
- mitigation procedure on not using unvalidated tags until they pass
- mitigation procedure on high volume commit times - lack of resources (group commits - has its own issues)
Issues to fix
1 - tag timestamps are non-standard
https://nexus3.onap.org/#browse/browse/components:docker.snapshot
v2/onap/aai-resources/manifests/1.2.0-SNAPSHOT-STAGING-20171205T173445
v2/onap/aai/esr-gui/manifests/1.0.1-SNAPSHOT-STAGING-171205-122425
We can pull all the highest version tags in a query (after we fix above) and generate the tag set this way
2- need to update the manifest and use the plugin
https://wiki.onap.org/display/DW/ONAP+Version+Manifest+Maven+Plugin
Notes
http://blog.christianposta.com/deploy/blue-green-deployments-a-b-testing-and-canary-releases/
https://aws.amazon.com/about-aws/whats-new/2016/08/netflix-oss-spinnaker-on-the-aws-cloud-quick-start-reference-deployment/
https://medium.com/@gajus/the-missing-ci-cd-kubernetes-component-helm-package-manager-1fe002aac680
- blocks
-
LOG-265
Migrate OOM CD job to Azure public - run by jenkins.onap.org
-
- Closed
-
-
LOG-370
Kubernetes DevOps Nexus/Jenkins/Gitlab/Kibana/Elasticsearch/ helm chart infrastructure
-
- Closed
-
- duplicates
-
TSC-25 Task Force to implement CD (Continuous Deployment)
-
- Closed
-
- is blocked by
-
-
- Closed
-
-
OOM-752
LF support for Helm build job and repository
-
- Closed
-
-
AAI-614
AAI: Refactor CHEF out or enable $TAG passed into chef branch git pull in traversal/resources to enable CI/CD
-
- Closed
-
-
DCAEGEN2-300 DCAE controller error with neutron_plugin env substitutions in Cloudify centos_vm blueprint
-
- Closed
-
-
-
- Closed
-
-
OOM-476
Parameterize values.yaml docker image repos into global config variables
-
- Closed
-
-
-
- Closed
-
-
CIMAN-132
Docker Standard Tag Format
-
- Done
-
- is duplicated by
-
LOG-332
Deployment CI/CD Integration Jenkins job using OOM-K8S - to validate branch
-
- Closed
-
- relates to
-
INT-189
TAG All builds when Deployment CI passes
-
- Closed
-
-
INT-359
Versioning For Stable ONAP HEAT/Kubernetes Deployment
-
- Closed
-
-
INT-120
Deployment Status Sanity - Daily auto E2E verification status of the build
-
- Closed
-
-
INT-284
Proper Release Management for Deployments
-
- Closed
-
-
INT-366 Jenkins CD trigger build can write -1 or +1 to the gerrit review under test
-
- Closed
-
-
LOG-266
F2F: ONAP CI/CD using OOM Kubernetes
-
- Closed
-
-
LOG-332
Deployment CI/CD Integration Jenkins job using OOM-K8S - to validate branch
-
- Closed
-
-
-
- Closed
-
-
INT-300
CI-CD For ONAP master branch
-
- Closed
-
-
SECCOM-36
•Security of CI/CD of ONAP– signature verification of artifacts + artifacts
-
- Open
-
-
CIMAN-174
Jenkins jobs failed by LF resource timeout - could use log parsing to auto retry the job - recheck
-
- In Progress
-
-
OOM-528
Confirm k8s context with a prompt for deleteAll.bash
-
- Closed
-
-
LOG-338
Deploy CD framework to Microsoft Azure
-
- Closed
-
-
LOG-350
ONS conference prep - ONAP OOM CD on Amazon/Azure
-
- Closed
-
-
LOG-341
Run CD jenkins/kibana from our own Kubernetes deployment - not off docker compose
-
- Closed
-
-
LOG-375
Nexus3.onap.info public proxy for throughput
-
- Closed
-
-
-
- Closed
-
-
OPENLABS-161
TLAB - Create tenant for OOM-deployed ONAP instance
-
- Closed
-
-
-
- Closed
-
-
-
- To Do
-
-
CIMAN-132
Docker Standard Tag Format
-
- Done
-
- links to
- mentioned in
-
Page Loading...
-
-
-
-
-