[regression] Test deploy_ceph_net_tmpl failed with "Loop detected" error

[ElenaRossokhina]

Detailed bug description:
Test deploy_ceph_net_tmpl failed with following message
Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[netconfig/1], Task[hosts/1], Task[primary-cluster/1], Task[virtual_ips/1], Task[netconfig/1].
Steps to reproduce:
run 'deploy_ceph_net_tmpl' group
Expected results:
pass
Actual result:
deployment fails due to loop detection

fuel-9.0-335-2016-05-19_08-00-59.iso
here is diagnostic snapshot https://drive.google.com/open?id=0B2ag_Bf-ShtTQkhCRVNFWFk1d2s

[Vladimir Kuklin (vkuklin)]

Regression was introduced by https://github.com/openstack/fuel-library/commit/7903d3d152503da746d2610bb54a4ab5e57ad6bf

role combination creates a loop. netconfig and netconfig-controller tasks should be exclusive

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/318763

[Oleksiy Butenko (obutenko)]

This bug blocker for tempest ci

[Timur Nurlygayanov (tnurlygayanov)]

The deployment of all 4 popular OpenStack configuration are blocked because of the issue, priority changed to Critical.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/318763
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7c106ae13ee38bb5ad91f30ef3a7d58e7c8c2227
Submitter: Jenkins
Branch: master

commit 7c106ae13ee38bb5ad91f30ef3a7d58e7c8c2227
Author: Aleksandr Didenko <email address hidden>
Date: Thu May 19 18:16:11 2016 +0200

    Fix netconfig task loop for multiroles

    Add strict condition to yaql_exp for netconfig task to not run it
    on controllers.

    Change-Id: I6faf8ff7cf97353d9c6b79265d5f9fb16afb57b3
    Closes-bug: #1583629

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/319389

[Sergey Novikov (snovikov)]

it seems same for https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.error_node_reinstallation/116/
from astute.log:
http://paste.openstack.org/show/498039/

[Aleksandr Didenko (adidenko)]

Btw, maybe we should add multirole case to Fuel-CI? Considering that now we have complex task-based orchestration this could be a good addition to our CI coverage. Any role + controller would be sufficient. For instance (copy-paste from the test group that detected this regression):

            3. Add 3 controller + ceph nodes
            4. Add 2 compute + ceph nodes

[Anastasia Kuznetsova (akuznetsova)]

The same issue has occurred on ISO 9.0-mos 388 during following test case:
Scenario:
            1. Create cluster
            2. Add 3 node with controller role and mongo
            3. Add 1 nodes with compute role
            4. Set Ceilometer install option
            5. Deploy the cluster
            6. Verify Heat and Ceilometer services
            7. Run OSTF platform tests
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha/115/testReport/(root)/deploy_heat_ha/deploy_heat_ha/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/319389
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d850ed5808a783fc6cb447ea2144fb4798ddd75d
Submitter: Jenkins
Branch: stable/mitaka

commit d850ed5808a783fc6cb447ea2144fb4798ddd75d
Author: Aleksandr Didenko <email address hidden>
Date: Thu May 19 18:16:11 2016 +0200

    Fix netconfig task loop for multiroles

    Add strict condition to yaql_exp for netconfig task to not run it
    on controllers.

    Change-Id: I6faf8ff7cf97353d9c6b79265d5f9fb16afb57b3
    Closes-bug: #1583629
    (cherry picked from commit 7c106ae13ee38bb5ad91f30ef3a7d58e7c8c2227)

[Kyrylo Romanenko (kromanenko)]

This bug occurs also on environment without ceph:
ironic_deploy_swift
        Scenario:
            1. Create cluster
            2. Add 1 node with controller role
            3. Add 2 node with controller+ironic role
            4. Add 1 node with compute role
            5. Add 1 nodes with ironic role
            6. Deploy the cluster
...
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[netconfig/1], Task[hosts/1], Task[primary-cluster/1], Task[virtual_ips/1], Task[netconfig/1].
Inspect Astute logs for the details'

[Ivan Lozgachev (ilozgachev)]

Has been reproduced on MOS 9.0 build 390 with configuration:

1 controller + mongo
2 compute
1 cinder

[Alexandr Kostrikov (akostrikov-mirantis)]

Also, reproduce on swarm:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_db_separate_services/116/testReport/(root)/separate_db_service/separate_db_service/

[Alexandr Kostrikov (akostrikov-mirantis)]

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_2_separate_services/117/testReport/(root)/separate_rabbit_service/separate_rabbit_service/
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_db_separate_services/116/testReport/(root)/separate_db_service/separate_db_service/

[Alexandr Kostrikov (akostrikov-mirantis)]

Possible, fix has not been on iso at the time swarm started

[Sergey Novikov (snovikov)]

Reproduced for https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.error_node_reinstallation/117/

[Sergey Novikov (snovikov)]

same for https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.cic_maintenance_mode/119

[Artem Minasyan (aminasyan)]

Also , reproduce here: https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha.ceilometer/116/testReport/(root)/deploy_ceilometer_ha_multirole/deploy_ceilometer_ha_multirole/

[Kyrylo Romanenko (kromanenko)]

Bug is still reproduced on today`s ISO 394 (2016-05-24).

Cluster:
1 Controller+Ironic
1 Compute

Error:
Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[netconfig/1], Task[hosts/1], Task[primary-cluster/1], Task[virtual_ips/1], Task[netconfig/1].

[Alexander Gubanov (ogubanov)]

Bug again reproduced, this is affected Tempest CI with merged fix https://review.openstack.org/#/c/318763/

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 394
cat /etc/fuel_build_number:
 394
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0

details http://pastebin.com/GMnvukQt
Snapshot is attached

[Aleksandr Didenko (adidenko)]

Yep, I confirm the bug is still there. yaql_exp does not remove task from the graph, it adds it into the graph with 'type=>skipped'. So task won't be executed, but the loop in the graph is still there.

[Artem Minasyan (aminasyan)]

This bug reproduced on 394 ISO with enabled ceilometer , with 3 controllers and compute with cinder LVM.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/320530

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/320807

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/320530
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=08b7a0b50aa654696fade182f10808354452b778
Submitter: Jenkins
Branch: master

commit 08b7a0b50aa654696fade182f10808354452b778
Author: Aleksandr Didenko <email address hidden>
Date: Tue May 24 15:19:15 2016 +0200

    Revert all network idempotency related fixes

    Revert "Fix netconfig for envs without controllers/vips"
    This reverts commit 0d009b7facea66a1b88c9a428c8d4250e4790bba.

    Revert "Fix netconfig task loop for multiroles"
    This reverts commit 7c106ae13ee38bb5ad91f30ef3a7d58e7c8c2227.

    Revert "Make network configuration idempotent"
    This reverts commit 7903d3d152503da746d2610bb54a4ab5e57ad6bf.

    Change-Id: I3bfd3f31f53fa7fa99414bea956802371de41340
    Closes-bug: #1583912
    Closes-bug: #1583629
    Closes-bug: #1584002

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/320807
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f5879c940f584f9f94e6fac7fc95b43c06d4a3ab
Submitter: Jenkins
Branch: stable/mitaka

commit f5879c940f584f9f94e6fac7fc95b43c06d4a3ab
Author: Aleksandr Didenko <email address hidden>
Date: Wed May 25 09:13:19 2016 +0200

    Revert all network idempotency related fixes

    Revert "Fix netconfig task loop for multiroles"
    This reverts commit d850ed5808a783fc6cb447ea2144fb4798ddd75d.

    Revert "Make network configuration idempotent"
    This reverts commit fdb541f57de7d8769441ce92c2526807d5694017.

    Change-Id: I472341efdd2e38cc355d64a46030ffcf17dfac96
    Closes-bug: #1583912
    Closes-bug: #1583629
    Closes-bug: #1584002

[Jay Pipes (jaypipes)]

Hi all, what's the status on this? Looks like the stable/mitaka revert was merged. Can this be marked Fix Committed?

[Alexandr Kostrikov (akostrikov-mirantis)]

Hi, Jay. During today swarm there were no reproduces. I am closing it as 'Fix Released'

[Maksym Strukov (unbelll)]

Reproduced manually:

Steps:
1. Install detach-rabbit plugin on master node (https://github.com/openstack/fuel-plugin-detach-rabbitmq.git)
2. Create next cluster - Ubuntu, all default values, separate_rabbit enabled, 3 controllers, 3 rabbitmq_nodes, 1 compute, 1 cinder
3. Deploy cluster

Actual:
Deployment has failed. Method task_deploy. Cluster[]: Loop detected! Path: Task[generate_keys/master], Task[copy_keys/1], Task[pre_deployment_end/virtual_sync_node], Task[deploy_start/4], Task[hiera/4], Task[setup_repositories/4], Task[fuel_pkgs/4], Task[globals/4], Task[rabbitmq-hiera-override/4], Task[logging/4], Task[tools/4], Task[netconfig/4], Task[firewall/4], Task[primary-cluster/4], Task[cluster/3], Task[virtual_ips/3], Task[netconfig/4].
Inspect Astute logs for the details

Env:
9.0-mos-452

[Alexandr Kostrikov (akostrikov-mirantis)]

Indeed, there is a regression. Maybe, new bug should be opened.
That is reproducing at swarm and it is a swarm blocker.

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_2_separate_services/135/testReport/(root)/separate_all_service/separate_all_service/
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_db_separate_services/133/testReport/(root)/separate_db_service/separate_db_service/
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/135/testReport/(root)/separate_keystone_service/separate_keystone_service/

[Alexandr Kostrikov (akostrikov-mirantis)]

I have opened new bug https://bugs.launchpad.net/fuel/+bug/1590329