Deploy of detached rabbit plugin failed with error: Loop detected!

Bug #1651066 reported by Nastya Urlapova on 2016-12-19
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Dmitry Mescheryakov
Mitaka
High
Dmitry Mescheryakov
Newton
High
Dmitry Ilyin

Bug Description

Deployment with separate rabbitmq nodes and ceph for all

    Scenario:
        1. Install the plugin on the master node
        2. Create Ubuntu, Neutron VXLAN, ceph for all storages cluster
        3. Change ceph replication factor to 2
        4. Add 3 nodes with controller role
        5. Add 3 nodes with rabbitmq role
        6. Add 1 compute node
        7. Add 2 ceph node
        8. Run network verification
        >>> 9. Deploy changes <<< failed here
        10. Run network verification
        11. Run OSTF tests

Deployment failed with error:
Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[generate_keys/master], Task[copy_keys/1], Task[pre_deployment_end/virtual_sync_node], Task[deploy_start/3], Task[hiera/3], Task[globals/3], Task[rabbitmq-hiera-override/3], Task[logging/3], Task[tools/3], Task[netconfig/3], Task[firewall/3], Task[primary-cluster/3], Task[cluster/1], Task[virtual_ips/1], Task[netconfig/3].

Version9.2 snapshot #638

Nastya Urlapova (aurlapova) wrote :
Changed in fuel:
milestone: none → 9.2
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
tags: added: swarm-fail
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Dmitry Mescheryakov (dmitrymex)
Changed in fuel:
status: New → Confirmed
tags: added: area-plugins
tags: added: swarm-blocker
tags: added: blocker-for-qa
tags: removed: swarm-blocker
Nastya Urlapova (aurlapova) wrote :

So issue affects 5 test cases, so it is a swarm-blocker.

tags: added: swarm-blocker
removed: swarm-fail
Changed in fuel:
milestone: 9.2 → 11.0

Fix proposed to branch: master
Review: https://review.openstack.org/418852

Changed in fuel:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/418852
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ceb77edfe06acc3ce703ff704054074aab4a00e4
Submitter: Jenkins
Branch: master

commit ceb77edfe06acc3ce703ff704054074aab4a00e4
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

    Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

    Also, that CR makes change in rabbitmq networking trigger OpenStack
    tasks. That fixes the old assumption that RabbitMQ always lives on
    controllers. Now changes in networking of nodes with RabbitMQ role
    will also trigger modification of configs for OpenStack services on
    other nodes, updating rabbit_hosts parameter.

    For the same reason (changes in rabbitmq should trigger OpenStack
    tasks) all OpenStack tasks now cross-depend on rabbitmq.

    Finally the CR fixes typo in openstack-cinder/tasks.yaml:
    cross-depened-by -> cross-dependend-by

    Closes-Bug: #1651066

    Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/419543
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ab1c4dd8260d3d66d6d9a6f13a8a7ea3f9acd934
Submitter: Jenkins
Branch: stable/mitaka

commit ab1c4dd8260d3d66d6d9a6f13a8a7ea3f9acd934
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

    Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

    Also, that CR makes change in rabbitmq networking trigger OpenStack
    tasks. That fixes the old assumption that RabbitMQ always lives on
    controllers. Now changes in networking of nodes with RabbitMQ role
    will also trigger modification of configs for OpenStack services on
    other nodes, updating rabbit_hosts parameter.

    For the same reason (changes in rabbitmq should trigger OpenStack
    tasks) all OpenStack tasks now cross-depend on rabbitmq.

    Finally the CR fixes typo in openstack-cinder/tasks.yaml:
    cross-depened-by -> cross-dependend-by

    Closes-Bug: #1651066

    Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

tags: added: on-verification

Reviewed: https://review.openstack.org/420434
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=036fc274b29bdd6e03bf52f5a7e5b4513476ee6d
Submitter: Jenkins
Branch: stable/newton

commit 036fc274b29bdd6e03bf52f5a7e5b4513476ee6d
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 11 13:50:30 2017 +0400

    Limit cluster task cross-dependency on primary-cluster

    Cluster task should cross-depend on primary-cluster only if current
    node has primary-something role. Otherwise a loop listed in the
    referenced bug occurs and deployment breaks.

    Also, that CR makes change in rabbitmq networking trigger OpenStack
    tasks. That fixes the old assumption that RabbitMQ always lives on
    controllers. Now changes in networking of nodes with RabbitMQ role
    will also trigger modification of configs for OpenStack services on
    other nodes, updating rabbit_hosts parameter.

    For the same reason (changes in rabbitmq should trigger OpenStack
    tasks) all OpenStack tasks now cross-depend on rabbitmq.

    Finally the CR fixes typo in openstack-cinder/tasks.yaml:
    cross-depened-by -> cross-dependend-by

    Closes-Bug: #1651066

    Change-Id: I8b262ef73c9c62ce5a71f0a5eee001b61ba9ecb1

Ekaterina Shutova (eshutova) wrote :

Verified on 9.2 snapshot #788. Passed run: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.detach-rabbitmq-ci-group/135/testReport/
Test is sometimes failed on step: run OSTF tests, but it's another issue.

tags: removed: on-verification

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers