Deploy of detached rabbit plugin failed with error: Loop detected!

[Nastya Urlapova]

Deployment with separate rabbitmq nodes and ceph for all

    Scenario:
        1. Install the plugin on the master node
        2. Create Ubuntu, Neutron VXLAN, ceph for all storages cluster
        3. Change ceph replication factor to 2
        4. Add 3 nodes with controller role
        5. Add 3 nodes with rabbitmq role
        6. Add 1 compute node
        7. Add 2 ceph node
        8. Run network verification
        >>> 9. Deploy changes <<< failed here
        10. Run network verification
        11. Run OSTF tests

Deployment failed with error:
Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[generate_keys/master], Task[copy_keys/1], Task[pre_deployment_end/virtual_sync_node], Task[deploy_start/3], Task[hiera/3], Task[globals/3], Task[rabbitmq-hiera-override/3], Task[logging/3], Task[tools/3], Task[netconfig/3], Task[firewall/3], Task[primary-cluster/3], Task[cluster/1], Task[virtual_ips/1], Task[netconfig/3].

Version9.2 snapshot #638

[Dmitry Belyaninov (dbelyaninov)]

failed job
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.detach-rabbitmq-ci-group/110/testReport/(root)/separate_rabbit_service/separate_rabbit_service/

similar bug
https://bugs.launchpad.net/fuel/+bug/1649889

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.plugins.thread_2_separate_services/161/testReport/(root)/separate_rabbit_service/separate_rabbit_service/

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.plugins.thread_separate_services_ceph/30/testReport/(root)/separate_db_ceph_service/separate_db_ceph_service/

[Nastya Urlapova (aurlapova)]

So issue affects 5 test cases, so it is a swarm-blocker.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418852

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/419543

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/420434

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418852
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ceb77edfe06acc3ce703ff704054074aab4a00e4
Submitter: Jenkins
Branch: master

commit ceb77edfe06acc3ce703ff704054074aab4a00e4
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

    Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

    Also, that CR makes change in rabbitmq networking trigger OpenStack
    tasks. That fixes the old assumption that RabbitMQ always lives on
    controllers. Now changes in networking of nodes with RabbitMQ role
    will also trigger modification of configs for OpenStack services on
    other nodes, updating rabbit_hosts parameter.

    For the same reason (changes in rabbitmq should trigger OpenStack
    tasks) all OpenStack tasks now cross-depend on rabbitmq.

    Finally the CR fixes typo in openstack-cinder/tasks.yaml:
    cross-depened-by -> cross-dependend-by

    Closes-Bug: #1651066

    Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419543
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ab1c4dd8260d3d66d6d9a6f13a8a7ea3f9acd934
Submitter: Jenkins
Branch: stable/mitaka

commit ab1c4dd8260d3d66d6d9a6f13a8a7ea3f9acd934
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

    Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

    Also, that CR makes change in rabbitmq networking trigger OpenStack
    tasks. That fixes the old assumption that RabbitMQ always lives on
    controllers. Now changes in networking of nodes with RabbitMQ role
    will also trigger modification of configs for OpenStack services on
    other nodes, updating rabbit_hosts parameter.

    For the same reason (changes in rabbitmq should trigger OpenStack
    tasks) all OpenStack tasks now cross-depend on rabbitmq.

    Finally the CR fixes typo in openstack-cinder/tasks.yaml:
    cross-depened-by -> cross-dependend-by

    Closes-Bug: #1651066

    Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420434
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=036fc274b29bdd6e03bf52f5a7e5b4513476ee6d
Submitter: Jenkins
Branch: stable/newton

commit 036fc274b29bdd6e03bf52f5a7e5b4513476ee6d
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

    Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

    Also, that CR makes change in rabbitmq networking trigger OpenStack
    tasks. That fixes the old assumption that RabbitMQ always lives on
    controllers. Now changes in networking of nodes with RabbitMQ role
    will also trigger modification of configs for OpenStack services on
    other nodes, updating rabbit_hosts parameter.

    For the same reason (changes in rabbitmq should trigger OpenStack
    tasks) all OpenStack tasks now cross-depend on rabbitmq.

    Finally the CR fixes typo in openstack-cinder/tasks.yaml:
    cross-depened-by -> cross-dependend-by

    Closes-Bug: #1651066

    Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

[Ekaterina Shutova (eshutova)]

Verified on 9.2 snapshot #788. Passed run: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.detach-rabbitmq-ci-group/135/testReport/
Test is sometimes failed on step: run OSTF tests, but it's another issue.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.