Yes, good point- we will need some guidelines/numbers around deployment. Orange, Huawei and Amdocs have similar deployment numbers a couple min up to 5h. Depending on the component coming up it will take between 1 min and 80 for a single project (including dependencies like dmaap) or 90-300 min for a full ONAP deploy Issues will be Single VM vs 4-14 node cluster Reuse (2-15 min teardown of prev deployment) or create from scratch the kubernetes cluster (15 min from bare VMs) Prepull dockers script (2-20 min) Integration override yaml, manifest script Single project deployment Defined dependencies (most of these are in the helm charts for cases where healthcheck is non-atomic) https://wiki.onap.org/display/DW/Log+Streaming+Compliance+and+API#LogStreamingComplianceandAPI-DeploymentDependencyTree Undefined dependencies (If DMAAP deployment fails then SDC healthcheck will fail – even though the dependency is not set in the helm chart) Dependency failures – if DMAAP will not deploy – then the SDNC will fail to deploy (even before you can run healthcheck – because it has defined it’s pod dependency in the chart – as a good design pattern) Full ONAP deployment Secondary orchestration Dcaegen2 dep-* namespace pods take between 34 and 70 min to deploy There are some times per component that I have prototyped in the following https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-DeploymentIntegritybasedonPodDependencies https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n228 DEPLOY_ORDER_POD_NAME_ARRAY=('consul msb dmaap dcaegen2 aaf robot aai esr multicloud oof so sdc sdnc vid policy portal log vfc uui vnfsdk appc clamp cli pomba vvp contrib sniro-emulator') DEPLOY_NUMBER_PODS_DESIRED_ARRAY=(4 5 11 11 13 1 15 2 6 17 10 12 11 2 8 6 3 18 2 5 5 5 1 11 11 3 1) Timing example is below (of course timing gets longer as more components use cpu/network/hd resources – and half of these can be done in parallel) Note that some management of problem containers is required, and not every container participates in healthcheck and dependency checking = so healthcheck is actually a <100% check of deployment status, do we wait for the whole cluster or a particular quorum for 3/5/7 ReplicaSets. http://jenkins.onap.info/job/oom-cd-master2-aws/124/consoleFull Minutes: component 1: consul 2: msb 3: dmaap 2: dcaegen2 (part 1) 32-70: dcaegen2 (part 2 cloudify) 4: aaf 4: robot 5: aai 2: esr 2: multicloud 20+: oof (wait for 15/16 is optional) 12: so 20+: sdc (20m timeout waiting on 7/11) 9: sdnc 2: vid 9: policy 12: portal 12: log 8: vfc 5: uui 10: vnfsdk 12: appc 8: clamp 3: cli 8: pomba 8: vvp 6: contrib 6: sniro-emulator Ideally most single pods will come up in under 10 min as per the numbers below – but in order to test for example SDNC – you need to bring up SDC and DMAAP first – the question becomes – if DMAAP is unstable does it block SDNC. https://wiki.onap.org/display/DW/Log+Streaming+Compliance+and+API#LogStreamingComplianceandAPI-DeploymentDependencyTree So turnaround time for a short-regression-CD would be 10-30 min for most components and 90 min for DCAE. For full deployment – it will be 90m to 5 hours – this should be the long regression test – emailed likely. The pattern is usually within an hour the review is marked as a pass/fail, and within 6 hours is marked as breaking the overall build – to be lumped in with all the changes that went into that period. /michael From: Yang Xu (Yang, Fixed Network) Sent: Monday, February 4, 2019 1:45 PM To: Michael O'Brien ; sylvain.desbureaux@orange.com; Mike Elliott ; de Talhouet, Alexis ; Gary Wu ; Jessica Wagantall Cc: DEBEAU Eric TGI/OLN ; Catherine LEFEVRE ; ROBERT René TGI/OLN ; RICHOMME Morgan TGI/OLN ; Phil Robb Subject: RE: OOM Gating with Orange Open Lab Team, To integrate in the build process, it needs to work reliably and fast enough. From the gerrit history, I can’t really tell what the turnaround time is. . -Yang From: Michael O'Brien [mailto:Frank.Obrien@amdocs.com] Sent: Monday, February 04, 2019 12:25 PM To: sylvain.desbureaux@orange.com; Mike Elliott ; de Talhouet, Alexis ; Gary Wu ; Yang Xu (Yang, Fixed Network) ; Jessica Wagantall Cc: DEBEAU Eric TGI/OLN ; Catherine LEFEVRE ; ROBERT René TGI/OLN ; RICHOMME Morgan TGI/OLN ; Phil Robb Subject: RE: OOM Gating with Orange Open Lab Sylvain, Team, The POC by Orange is working and has a POC integration with gerrit in the review below. Based on this working demo, the expertise from Sylvain and Morgan on the Orange CD pipeline – I recommend we go with this approach and integrate the MQTT approach or something similar with the LF asap https://wiki.onap.org/display/DW/CD+-+Continuous+Deployment#CD-ContinuousDeployment-20190131-OrangeCDdemofromSylvainDesbureaux https://gerrit.onap.org/r/#/c/77660/ Because of the known issues with the 3.x release – the -1,0,+1 as -1 or 0 works – I usually need to bake some workarounds into the deployment script to not wait indefinitely for the known pods. Thanks Sylvain /michael From: sylvain.desbureaux@orange.com Sent: Monday, February 4, 2019 10:31 AM To: Mike Elliott ; de Talhouet, Alexis ; Michael O'Brien ; gary.i.wu@huawei.com; yang.xu3@huawei.com; Jessica Wagantall Cc: DEBEAU Eric TGI/OLN ; Catherine LEFEVRE ; ROBERT René TGI/OLN ; RICHOMME Morgan TGI/OLN ; Phil Robb Subject: OOM Gating with Orange Open Lab Hello, As shown on Thursday, I’ve started OOM Gating with Orange Open Lab. In a nutshell: every new patchset will (should ??) be tested with a deployment of a complete ONAP using OOM and with some automated tests after: 1st round: • Core Healthcheck (AAI, DMAAP, Portal, SDC, SDNC, SO) • Small Healthcheck (Holmes, VID, Multicloud-pike/ocata/api, MSB, Logstash, Kibana, Elasticsearch, NBI, CLI, APPC, AAF) • Medium Healthcheck (CLAMP, DCAE, OOF, Policy, UUI) • Full Healthcheck (everything) • Check if all pods are running If Core and Small are OK, we’re going for the second round (for each use case, we’re doing , onboarding, instanciation, check if exists on OpenStack, and delete): • Simple Ubuntu16 deployment (1 VM) • Clearwater IMS (6 VMs for 1 VNF) • Freeradius with NBI • vFW We then score the review. As of today it’s always 0 but goal is to +1 if success (ONAP deployment, Core + Small healthchecks, all use cases OK), -1 if not. On today’s deployment, I always have the current issues: • DCAE2 deployment fails on inventory API • SDC FE is crashing • SDC is not working (healtcheck NOK) • SO is not cleaning when deleting I also have some random issue on DMaaP-dr-prov. That’s why I’m not scoring because I would put -1 all the time... In the detail, We have done: +----------------------+ | | | | | GERRIT LFN | | | | | +--------------+ +-----^----------+-----+ | | | | | | | | | | | +--v--------------+ | | +------------------------------+ | | | | +-----> | | | Gerrit 2 MQTT +---> | | Chained-CI MQTT trigger | | | | | MQTT <-----+ | | +-----------------+ | | +-----------------------+------+ | | | | | +-----------------+ | Broker | +-----v---------+ | | | | | | | +-------+ MQTT 2 Gerrit <---+ | +-----+ Chained CI +--+ | | | | | | | | +-----------------+ | | | +---------------+ | | | | | | | +-------v--------+ +----------v-+ +--------------+ | | | | | Deploy ONAP | | Test ONAP | | | | | +----------------+ +------------+ All the code is open sourced on https://gitlab.com/Orange-OpenSource/lfn: Gerrit 2 MQTT: https://gitlab.com/Orange-OpenSource/lfn/ci_cd/gerrit-to-mqtt/ MQTT 2 Gerrit: https://gitlab.com/Orange-OpenSource/lfn/ci_cd/mqtt-to-gerrit/ Chained-CI trigger: https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-mqtt-trigger Chained-CI parts: https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-roles, https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-vue, https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-tools ONAP Deployment: https://gitlab.com/Orange-OpenSource/lfn/onap/onap_oom_automatic_installation ONAP E2E use case tests: https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests Functest Wrapper around ONAP E2E tests and healthchecks: https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap **Gitlab Promotion** As you may understand, we’re using extensively Gitlab and gitlab ci. For our projects, I’ve added with few lines in the CI YML description files : • docker image creation automation • code quality tests • container scanning for vulnerabilities • dependency scanning, • license management, • security scan It works for Python, Java, Ruby, … so it’s a good catch for ONAP ?? (and better than Github I think as it’s also OpenSource).