[9.2][contrail] deployment has failed with error "Cluster[]: Loop detected"

[Ruslan Khozinov]

Fuel 9.2 #628
Contrail v5.0

Trace:
Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[generate_keys/master], Task[copy_keys/1], Task[pre_deployment_end/virtual_sync_node], Task[deploy_start/1], Task[contrail-config-primary/1], Task[contrail-config-provision-primary/1], Task[contrail-config-all/2], Task[openstack-controller-provision/8], Task[top-role-compute/9], Task[openstack-network-start/9], Task[openstack-network-common-config/9], Task[openstack-network-server-nova/8], Task[openstack-network-end/8], Task[openstack-controller-provision/8].

bvt_test has failed

steps to reproduce
build environment:

FUEL_RELEASE_VERSION = 9.2
UPDATE_MASTER = true
ENABLE_PROPOSED = true
ISO_FILE = MirantisOpenStack-9.0.iso
SNAPSHOTS_ID = lastSuccessfulBuild
MIRROR_HOST = mirror.seed-cz1.fuel-infra.org
PLUGIN_VERSION = 5.0.0
ADMIN_NODE_CPU=4;
ADMIN_NODE_MEMORY=4096;
ALWAYS_CREATE_DIAGNOSTIC_SNAPSHOT=true;
CONV_SNAP_PATH=plugin_test/fuel-qa/utils/jenkins/conv_snapshot_file.py;
DEBUG_MODE=true;
DESTROY_ENV_AFTER=0;
DRIVER_ENABLE_ACPI=true;
ESXI_PASSWORD=swordfish;
ESXI_USER=root;
FORCE_DISABLE_UPDATES=false;
FUEL_STATS_ENABLED=false;
GIT_CURL_VERBOSE=false;
MAKE_SNAPSHOT=true;
NODES_COUNT=10;
OPENSTACK_RELEASE=Ubuntu;
SLAVE_NODE_CPU=4;
SLAVE_NODE_MEMORY=4096;
SNAPSHOT_ARTIFACTS_FILE=snapshots.params;
SNAPSHOT_OUTPUT_FILE=extra_repos.sh;
WORKSTATION_PASSWORD=VMware01;
WORKSTATION_USERNAME=vmware;
BONDING=False;
CONTRAIL_DISTRIBUTION=;
CONTRAIL_PLUGIN_PACK_UB_PATH=;
CONTRAIL_PLUGIN_PATH=;
CONTRAIL_VERSION=5.0;
DISABLE_SSL=false;
FORCE_VSRX_COPY=False;
KVM_USE=True;
MIRROR=http://mirror.seed-cz1.fuel-infra.org;
MULTIPLE_NETWORKS=;
MULTIPLE_NETWORKS_TEMPLATE=;
NODE_VOLUME_SIZE=512;
SPECIFIC_VC=False;
VSRX_CONFIG_PATH=;
VSRX_ORIGINAL_IMAGE_PATH=/storage/contrail/vSRX.400.img;
VSRX_TARGET_IMAGE_PATH=/var/lib/libvirt/images/vSRX.400.img;
VSRX_TEMPLATE_PATH=/storage/contrail/vSRX_400.template;
=======================================================
watch the state of job
-------------
link to the job template to build - http://jenkins-tpi.bud.mirantis.net:8080/job/9.x.contrail.5.0.0.bvt/build?delay=0sec

[Ruslan Khozinov (rkhozinov)]

[14717] Error running RPC method task_deploy: Cluster[]: Loop detected! Path: Task[generate_keys/master], Task[copy_keys/1], Task[pre_deployment_end/virtual_sync_node], Task[deploy_start/1], Task[contrail-config-primary/1], Task[contrail-config-provision-primary/1], Task[contrail-config-all/2], Task[openstack-controller-provision/8], Task[top-role-compute/9], Task[openstack-network-start/9], Task[openstack-network-common-config/9], Task[openstack-network-server-nova/8], Task[openstack-network-end/8], Task[openstack-controller-provision/8], trace:
["/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:241:in `visit'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:254:in `block in visit'",
 "/usr/share/ruby/set.rb:232:in `each_key'",
 "/usr/share/ruby/set.rb:232:in `each'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/task.rb:266:in `each_forward_dependency'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:253:in `visit'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:254:in `block in visit'",
 "/usr/share/ruby/set.rb:232:in `each_key'",
 "/usr/share/ruby/set.rb:232:in `each'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/task.rb:266:in `each_forward_dependency'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:253:in `visit'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:254:in `block in visit'",
 "/usr/share/ruby/set.rb:232:in `each_key'",
 "/usr/share/ruby/set.rb:232:in `each'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/task.rb:266:in `each_forward_dependency'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:253:in `visit'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:254:in `block in visit'",
 "/usr/share/ruby/set.rb:232:in `each_key'",
 "/usr/share/ruby/set.rb:232:in `each'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/task.rb:266:in `each_forward_dependency'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:253:in `visit'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:254:in `block in visit'",
 "/usr/share/ruby/set.rb:232:in `each_key'",
 "/usr/share/ruby/set.rb:232:in `each'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/task.rb:266:in `each_forward_dependency'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:253:in `visit'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:254:in `block in visit'",
 "/usr/share/ruby/set.rb:232:in `each_key'",
 "/usr/share/ruby/set.rb:232:in `each'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/task.rb:266:in `each_forward_dependency'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:253:in `visit'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:254:in `block in visit'",
 "/usr/share/ruby/set.rb:232:in `each_key'",
 "/usr/share/ruby/set.rb:232:in `each'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/task.rb:266:in `each_forward_dependency'",
 "/usr/share/gems/gems/astute-9.0.0/lib/fuel_deployment/cluster.rb:253:in `visit'",
 "/usr/share/gems/gems/astut...

[Ruslan Khozinov (rkhozinov)]

The issue has been reproduced on the 681 snapshot_id

[Dmitry Belyaninov (dbelyaninov)]

seems duplicate of https://bugs.launchpad.net/fuel/+bug/1651066

[Illia Polliul (ipolliul)]

I've removed tasks that caused cycles, and it helped to start deployment.
However it still fails, because plugin roles got broken with recent updates.

There's a role `contrail-controller` with `has_primary: true`, but in the env with one such node it turned to be non-primary:

root@node-1:~# hiera roles
["contrail-controller"]

This breaks tasks execution sequence.

[Illia Polliul (ipolliul)]

Fix committed in https://review.openstack.org/#/c/420613/

[okosse (okosse)]

Guys.
How should this fix be delivered to customers? What will happen, if customers make a decision to update 9.1 an environment with 5.0.0 contrail plugin? Correct me, but the update will be failed. So, I guess, it is fuel-team responsibility to provide correct update process.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421844

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/421881

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421844
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=64d8c93d96f9ce41105b6edb35b2bc49bd65a7e5
Submitter: Jenkins
Branch: stable/mitaka

commit 64d8c93d96f9ce41105b6edb35b2bc49bd65a7e5
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed Jan 18 14:31:11 2017 +0300

    Clarify regexp for top-role-compute

    Add start and end anchors to the regexp to avoid intersection with
    other tasks.

    Change-Id: Ib18894d81d79e6ce52154bd5544226ded74c2acd
    Closes-Bug: #1649889

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/422707

[Ekaterina Khomyakova (ekhomyakova)]

Verified on Fuel 9.2 #773

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30297

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30297
Submitter: Mariia Zlatkova <email address hidden>
Branch: master

Commit: 325ae9c9a9517bfd03c46e5040669aaaa545e2e9
Author: Evgeny Konstantinov <email address hidden>
Date: Thu Feb 2 14:20:10 2017

Add contrail DPDK known issue to 9.2 relnotes

Change-Id: I1917954bfde1cbc7c92b000f9a2bc949e5dd88d4
Related-Bug: #1649889
Closes-Bug: #1661174

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: stable/9.2
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30426

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30426
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.2

Commit: 59645425f23cb2afafbcfb4c817c410a1cb295b7
Author: Evgeny Konstantinov <email address hidden>
Date: Thu Feb 2 14:24:23 2017

Add contrail DPDK known issue to 9.2 relnotes

Change-Id: I1917954bfde1cbc7c92b000f9a2bc949e5dd88d4
Related-Bug: #1649889
Closes-Bug: #1661174
(cherry picked from commit 325ae9c9a9517bfd03c46e5040669aaaa545e2e9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421881
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e6b49683838d03c9f0f5021480b914a952b85298
Submitter: Jenkins
Branch: master

commit e6b49683838d03c9f0f5021480b914a952b85298
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed Jan 18 14:31:11 2017 +0300

    Clarify regexp for top-role-compute

    Add start and end anchors to the regexp to avoid intersection with
    other tasks.

    Change-Id: Ib18894d81d79e6ce52154bd5544226ded74c2acd
    Related-Bug: #1649889

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422707
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8b2b03b77d544519ee678e6db8d75a7bcafb25e1
Submitter: Jenkins
Branch: stable/newton

commit 8b2b03b77d544519ee678e6db8d75a7bcafb25e1
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed Jan 18 14:31:11 2017 +0300

    Clarify regexp for top-role-compute

    Add start and end anchors to the regexp to avoid intersection with
    other tasks.

    Change-Id: Ib18894d81d79e6ce52154bd5544226ded74c2acd
    Related-Bug: #1649889