Uploaded image for project: 'Data Collection, Analytics, and Events'
  1. Data Collection, Analytics, and Events
  2. DCAEGEN2-352

Cloudify manager reports deployment success after floatingIP allocation failure bringing up CDAP cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: High High
    • None
    • Amsterdam Release
    • None
    • None

      From the dcae bootstrap container's docker container during deployment.
      I was not getting this behaviour before the openlab router deletion issue 2 days ago
      https://jira.onap.org/browse/OPENLABS-175

      ubuntu@dcae-dcae-bootstrap:~$ sudo docker ps -a
      CONTAINER ID        IMAGE                                                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
      59524098086d        nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.bootstrap:v1.1.1   "/bin/sh -c 'exec ..."   33 minutes ago      Up 33 minutes                           boot
      ubuntu@dcae-dcae-bootstrap:~$ sudo docker logs -f boot
      Finished executing workflow install on deployment DockerComponent
      * Run 'cfy events list --include-logs --execution-id b899c896-6353-4efb-ab64-a64041fd1e72' to retrieve the execution's events/logs
      + cfy install -p ./blueprints/cdap/cdapbp7.yaml -b cdapbp7 -d cdap7 -i ../config/cdapinputs.yaml -i location_id=4R4x
      Uploading blueprint ./blueprints/cdap/cdapbp7.yaml...
      Blueprint uploaded. The blueprint's id is cdapbp7
      Processing inputs source: ../config/cdapinputs.yaml
      Processing inputs source: location_id=4R4x
      Creating new deployment from blueprint cdapbp7...
      Deployment created. The deployment's id is cdap7
      Executing workflow install on deployment cdap7 [timeout=900 seconds]
      Deployment environment creation is in progress...
      2018-02-16T04:37:55 CFY <cdap7> Starting 'create_deployment_environment' workflow execution
      2018-02-16T04:37:55 CFY <cdap7> Installing deployment plugins
      2018-02-16T04:37:55 CFY <cdap7> Sending task 'cloudify_agent.operations.install_plugins'
      2018-02-16T04:37:55 CFY <cdap7> Task started 'cloudify_agent.operations.install_plugins'
      2018-02-16T04:37:56 CFY <cdap7> Task succeeded 'cloudify_agent.operations.install_plugins'
      2018-02-16T04:37:56 CFY <cdap7> Skipping starting deployment policy engine core - no policies defined
      2018-02-16T04:37:56 CFY <cdap7> Creating deployment work directory
      2018-02-16T04:37:57 CFY <cdap7> 'create_deployment_environment' workflow execution succeeded
      2018-02-16T04:38:02 CFY <cdap7> Starting 'install' workflow execution
      2018-02-16T04:38:02 CFY <cdap7> [key_pair_df6d4] Creating node
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap04_2cfed] Creating node
      2018-02-16T04:38:02 CFY <cdap7> [private_net_d15fa] Creating node
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap03_4808a] Creating node
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap04_2cfed.create] Sending task 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap06_81dc1] Creating node
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap03_4808a.create] Sending task 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap05_17385] Creating node
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap04_2cfed.create] Task started 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap03_4808a.create] Task started 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap06_81dc1.create] Sending task 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:03 CFY <cdap7> [key_pair_df6d4.create] Sending task 'nova_plugin.keypair.create'
      2018-02-16T04:38:03 CFY <cdap7> [private_net_d15fa.create] Sending task 'neutron_plugin.network.create'
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap00_89e4a] Creating node
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap06_81dc1.create] Task started 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:03 CFY <cdap7> [security_group_3e25d] Creating node
      2018-02-16T04:38:03 CFY <cdap7> [key_pair_df6d4.create] Task started 'nova_plugin.keypair.create'
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap05_17385.create] Sending task 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap01_5452d] Creating node
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap00_89e4a.create] Sending task 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap02_c05d6] Creating node
      2018-02-16T04:38:03 CFY <cdap7> [private_net_d15fa.create] Task started 'neutron_plugin.network.create'
      2018-02-16T04:38:03 CFY <cdap7> [sharedsshkey_cdap_9efda] Creating node
      2018-02-16T04:38:03 CFY <cdap7> [sharedsshkey_cdap_9efda.create] Sending task 'sshkeyshare.keyshare_plugin.generate'
      2018-02-16T04:38:03 CFY <cdap7> [security_group_3e25d.create] Sending task 'neutron_plugin.security_group.create'
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap02_c05d6.create] Sending task 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap01_5452d.create] Sending task 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:05 CFY <cdap7> [key_pair_df6d4.create] Task succeeded 'nova_plugin.keypair.create'
      2018-02-16T04:38:05 CFY <cdap7> [floatingip_cdap05_17385.create] Task started 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:05 CFY <cdap7> [private_net_d15fa.create] Task succeeded 'neutron_plugin.network.create'
      2018-02-16T04:38:05 CFY <cdap7> [floatingip_cdap00_89e4a.create] Task started 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:05 CFY <cdap7> [key_pair_df6d4] Configuring node
      2018-02-16T04:38:05 CFY <cdap7> [floatingip_cdap04_2cfed.create] Task succeeded 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:05 CFY <cdap7> [sharedsshkey_cdap_9efda.create] Task started 'sshkeyshare.keyshare_plugin.generate'
      2018-02-16T04:38:06 CFY <cdap7> [private_net_d15fa] Configuring node
      2018-02-16T04:38:06 CFY <cdap7> [key_pair_df6d4] Starting node
      2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap04_2cfed] Configuring node
      2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap06_81dc1.create] Task succeeded 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:06 CFY <cdap7> [security_group_3e25d.create] Task started 'neutron_plugin.security_group.create'
      2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap03_4808a.create] Task succeeded 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap02_c05d6.create] Task started 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap06_81dc1] Configuring node
      2018-02-16T04:38:06 CFY <cdap7> [private_net_d15fa] Starting node
      2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap04_2cfed] Starting node
      2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap03_4808a] Configuring node
      2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap06_81dc1] Starting node
      2018-02-16T04:38:07 CFY <cdap7> [sharedsshkey_cdap_9efda.create] Task succeeded 'sshkeyshare.keyshare_plugin.generate'
      2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap01_5452d.create] Task started 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:07 CFY <cdap7> [dns_cdap04_c80a9] Creating node
      2018-02-16T04:38:07 CFY <cdap7> [dns_cdap04_c80a9.create] Sending task 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:07 CFY <cdap7> [fixedip_cdap00_2934e] Creating node
      2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap05_17385.create] Task succeeded 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:07 CFY <cdap7> [dns_cdap04_c80a9.create] Task started 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:07 CFY <cdap7> [fixedip_cdap02_9697b] Creating node
      2018-02-16T04:38:07 CFY <cdap7> [fixedip_cdap01_7f78b] Creating node
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap04_42278] Creating node
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap05_433fb] Creating node
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap01_7f78b.create] Sending task 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap03_4808a] Starting node
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap02_9697b.create] Sending task 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap00_89e4a.create] Task succeeded 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap01_7f78b.create] Task started 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap03_1c4c2] Creating node
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap00_2934e.create] Sending task 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap04_42278.create] Sending task 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap05_433fb.create] Sending task 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap06_02e83] Creating node
      2018-02-16T04:38:08 CFY <cdap7> [security_group_3e25d.create] Task succeeded 'neutron_plugin.security_group.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap02_9697b.create] Task started 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap03_1c4c2.create] Sending task 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [sharedsshkey_cdap_9efda] Configuring node
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap06_02e83.create] Sending task 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap05_17385] Configuring node
      2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap00_89e4a] Configuring node
      2018-02-16T04:38:08 CFY <cdap7> [dns_cdap06_2d407] Creating node
      2018-02-16T04:38:08 CFY <cdap7> [security_group_3e25d] Configuring node
      2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap02_c05d6.create] Task succeeded 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap00_2934e.create] Task started 'neutron_plugin.port.create'
      2018-02-16T04:38:08 CFY <cdap7> [dns_cdap06_2d407.create] Sending task 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:09 CFY <cdap7> [sharedsshkey_cdap_9efda] Starting node
      2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap00_89e4a] Starting node
      2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap05_17385] Starting node
      2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap02_c05d6] Configuring node
      2018-02-16T04:38:09 CFY <cdap7> [dns_cdap03_f3c7d] Creating node
      2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap01_5452d.create] Task succeeded 'neutron_plugin.floatingip.create'
      2018-02-16T04:38:09 CFY <cdap7> [dns_cdap03_f3c7d.create] Sending task 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:09 CFY <cdap7> [security_group_3e25d] Starting node
      2018-02-16T04:38:09 CFY <cdap7> [fixedip_cdap04_42278.create] Task started 'neutron_plugin.port.create'
      2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap01_5452d] Configuring node
      2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap02_c05d6] Starting node
      2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap01_7f78b.create] Task succeeded 'neutron_plugin.port.create'
      2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap05_433fb.create] Task started 'neutron_plugin.port.create'
      2018-02-16T04:38:10 CFY <cdap7> [dns_cdap00_e22b0.create] Sending task 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:10 CFY <cdap7> [dns_cdap00_e22b0] Creating node
      2018-02-16T04:38:10 CFY <cdap7> [dns_cdap05_9f71a] Creating node
      2018-02-16T04:38:10 CFY <cdap7> [floatingip_cdap01_5452d] Starting node
      2018-02-16T04:38:10 CFY <cdap7> [dns_cdap04_c80a9.create] Task succeeded 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap03_1c4c2.create] Task started 'neutron_plugin.port.create'
      2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap06_02e83.create] Task started 'neutron_plugin.port.create'
      2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap02_9697b.create] Task succeeded 'neutron_plugin.port.create'
      2018-02-16T04:38:10 CFY <cdap7> [dns_cdap05_9f71a.create] Sending task 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap01_7f78b] Configuring node
      2018-02-16T04:38:10 CFY <cdap7> [dns_cdap06_2d407.create] Task started 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap00_2934e.create] Task succeeded 'neutron_plugin.port.create'
      2018-02-16T04:38:10 CFY <cdap7> [dns_cdap02_a5bef] Creating node
      2018-02-16T04:38:11 CFY <cdap7> [dns_cdap04_c80a9] Configuring node
      2018-02-16T04:38:11 CFY <cdap7> [dns_cdap02_a5bef.create] Sending task 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap02_9697b] Configuring node
      2018-02-16T04:38:11 CFY <cdap7> [dns_cdap01_65c42] Creating node
      2018-02-16T04:38:11 CFY <cdap7> [dns_cdap03_f3c7d.create] Task started 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap04_42278.create] Task succeeded 'neutron_plugin.port.create'
      2018-02-16T04:38:11 CFY <cdap7> [dns_cdap01_65c42.create] Sending task 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap01_7f78b] Starting node
      2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap00_2934e] Configuring node
      2018-02-16T04:38:11 CFY <cdap7> [dns_cdap04_c80a9] Starting node
      2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap02_9697b] Starting node
      2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap04_42278] Configuring node
      2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap05_433fb.create] Task succeeded 'neutron_plugin.port.create'
      2018-02-16T04:38:12 CFY <cdap7> [dns_cdap00_e22b0.create] Task started 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap00_2934e] Starting node
      2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap04_42278] Starting node
      2018-02-16T04:38:12 CFY <cdap7> [dns_cdap02_a5bef.create] Task started 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:12 CFY <cdap7> [dns_cdap05_9f71a.create] Task started 'dnsdesig.dns_plugin.aneeded'
      2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap06_02e83.create] Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409]
      2018-02-16T04:38:12 CFY <cdap7> 'install' workflow execution failed: RuntimeError: Workflow failed: Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409]
      Execution of workflow install for deployment cdap7 failed. [error=Traceback (most recent call last):
        File "/tmp/pip-build-U0y0vb/cloudify-plugins-common/cloudify/dispatch.py", line 472, in _remote_workflow_child_thread
        File "/tmp/pip-build-U0y0vb/cloudify-plugins-common/cloudify/dispatch.py", line 504, in _execute_workflow_function
        File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/workflows.py", line 27, in install
          node_instances=set(ctx.node_instances))
        File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/lifecycle.py", line 28, in install_node_instances
          processor.install()
        File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/lifecycle.py", line 83, in install
          graph_finisher_func=self._finish_install)
        File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/lifecycle.py", line 103, in _process_node_instances
          self.graph.execute()
        File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 133, in execute
          self._handle_terminated_task(task)
        File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 207, in _handle_terminated_task
          raise RuntimeError(message)
      RuntimeError: Workflow failed: Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409]
      ]
      * Run 'cfy events list --include-logs --execution-id 7d34383e-4558-4475-9ddf-e3aeabb5a665' to retrieve the execution's events/logs
      + cfy install -p ./blueprints/cbs/config_binding_service.yaml -b config_binding_service -d config_binding_service -i location_id=4R4x
      Uploading blueprint ./blueprints/cbs/config_binding_service.yaml...
      Blueprint uploaded. The blueprint's id is config_binding_service
      Processing inputs source: location_id=4R4x
      Creating new deployment from blueprint config_binding_service...
      Deployment created. The deployment's id is config_binding_service
      Executing workflow install on deployment config_binding_service [timeout=900 seconds]
      Deployment environment creation is in progress...
      2018-02-16T04:38:22 CFY <config_binding_service> Starting 'create_deployment_environment' workflow execution
      2018-02-16T04:38:22 CFY <config_binding_service> Installing deployment plugins
      2018-02-16T04:38:22 CFY <config_binding_service> Sending task 'cloudify_agent.operations.install_plugins'
      2018-02-16T04:38:22 CFY <config_binding_service> Task started 'cloudify_agent.operations.install_plugins'
      2018-02-16T04:38:23 CFY <config_binding_service> Task succeeded 'cloudify_agent.operations.install_plugins'
      2018-02-16T04:38:23 CFY <config_binding_service> Skipping starting deployment policy engine core - no policies defined
      2018-02-16T04:38:23 CFY <config_binding_service> Creating deployment work directory
      2018-02-16T04:38:24 CFY <config_binding_service> 'create_deployment_environment' workflow execution succeeded
      
      

            wrider wrider
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: