[SWARM][8.0] Provision fail in "Stop reset cluster" scenario on CentOS Bootstrap

[Vladimir Jigulin]

Steps to reproduce:
1. Choose CentOS bootstrap on master node
2. Bootstrap slaves
3. Verify bootstrap on slaves
4. Create cluster in Ha mode with 1 controller
5. Add 1 node with controller role
6. Add 1 node with compute role
7. Verify network
8. Deploy cluster
9. Stop deployment
10. Verify bootstrap on slaves
11. Add 1 node with cinder role
12. Re-deploy cluster
13. Reset cluster
14. Verify bootstrap on slaves
15. Re-deploy cluster

or use deploy_reset_on_ready_centos_bootstrap test from fuel-qa
test fail at line: https://github.com/openstack/fuel-qa/blob/stable/8.0/fuelweb_test/tests/test_centos_bootstrap.py#L180

Expected results: cluster deployment is successfull

Actual result: Task 'deploy' has incorrect status. error != ready

Traceback: http://paste.openstack.org/show/537397/

Reproducibility: 6 times in a row on ci: https://patching-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.bvt_centos_bootstrap/25/

[Anton Chevychalov (achevychalov)]

Bug confirmed on test environment.

Reasons:
1. Message shows because there is no answer over RPC (rabbitmq) from nodes. (UI show information about first affected node only).
2. We have no answer because mcollective on affected nodes is not running.
3. Mcollective is not up because nailgun-agent did not shoot command "service mcollective restart" (http://git.openstack.org/cgit/openstack/fuel-nailgun-agent/tree/agent?h=stable/8.0#n121)
5. Start is blocked by hostname!=bootstrap. That used by nailgun-agent as indicator of "provisioned" state (http://git.openstack.org/cgit/openstack/fuel-nailgun-agent/tree/agent?h=stable/8.0#n846 http://git.openstack.org/cgit/openstack/fuel-nailgun-agent/tree/agent?h=stable/8.0#n892

It is not clear in current moment what is the reason for node to be in that state.

[Anton Chevychalov (achevychalov)]

It's regression of https://bugs.launchpad.net/fuel/+bug/1455489

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/361132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/361132
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=7e44f553f2d5bb5fefa29c22e03867573e57f10d
Submitter: Jenkins
Branch: stable/8.0

commit 7e44f553f2d5bb5fefa29c22e03867573e57f10d
Author: Anton Chevychalov <email address hidden>
Date: Fri Aug 26 14:41:02 2016 +0300

    Fix regression after I0c6f3720943ad21e22899368832e451bc906b098

    It force centos to use static hostname (from /etc/sysconfig/network)
    only instead of using information from resolve back zone ip over DNS.
    After that commit we will have same behavior in Centos and Ubuntu images
    Additional information can be found in:
    https://bugs.launchpad.net/fuel/+bug/1455489
    and https://bugs.launchpad.net/fuel/+bug/1585671.
    Actual changes over upstream image is line 84 in ifup-post

    Closes-Bug: #1604365
    Change-Id: Idfcfbc6d99784b39818d169132dfe59c8b19fcfd

[Dmitry (dtsapikov)]

Verified on 8.0+mu3
but ci is blocked by https://bugs.launchpad.net/fuel/+bug/1552349