[network_template] Admin network goes down on compute after netconfig task if OVS bridges are used

[Artem Panchenko]

Fuel version info (9.0 mos build #458): http://paste.openstack.org/show/509227/

Deployment fails if network template with OVS as provider for bridges is used:

Error
Deployment has failed. All nodes are finished. Failed tasks: Task[openstack-haproxy-cinder/1], Task[openstack-haproxy-glance/3], Task[openstack-haproxy-stats/2], Task[netconfig/5] Stopping the deployment process!

Network template: https://github.com/openstack/fuel-qa/blob/stable/mitaka/fuelweb_test/network_templates/default_ovs.yaml

Test scenario:
            1. Revert snapshot with 5 slaves
            2. Create cluster (HA) with Neutron VLAN
            3. Add 3 controller and 1 compute + cinder nodes
            4. Upload 'default_ovs' network template
            5. Create custom network groups basing
               on template endpoints assignments
            6. Run network verification
FAIL >>>> 7. Deploy cluster and run basic health checks

Expected result: deployment is successful

Actual result: deployment fails

After 'netconfig' task node-5 (compute) went down, because admin network wasn't running (see vnc_screen.png). After `ifup br-fw-admin` admin network woked fine.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Artem Panchenko (apanchenko-8)]

Also here is diagnostic snapshot generated after I manually started admin network on node-5.

[Sergey Vasilenko (xenolog)]

Network-scheme: http://paste.openstack.org/show/NY3XUMICgRWWWGvVTonW/

Result:

IP and native linux bridges assignment:
http://paste.openstack.org/show/0X239G7HO8E6CDUtGyW5/

OVS configuration:
http://paste.openstack.org/show/jYD1tp0SAxqX8SxGqR8o/

I can't reproduce this bug on virtual ENV multiple time. Please check, that all NICs on you env is e1000. If you use virtio or realtec drivers -- please fix it. Only e1000 provide stable behavior for use OVS-based bridges on virtual-based ENVs.

[Andrey Sledzinskiy (asledzinskiy)]

All NICs are e1000, issue is constantly reproduced on SWARM

[slava valyavskiy (slava-val-al)]

Issue was reproduced for network template where 'br-fw-admin' has 'ovs' provider.

[slava valyavskiy (slava-val-al)]

Log of one more unsuccessful run.

[Anton Matveev (amatveev)]

sla1 for MOS9.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/341484

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/341620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/341621

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341484
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=38fe157e573cd4559ed2d07ad9ecea779f63cff2
Submitter: Jenkins
Branch: master

commit 38fe157e573cd4559ed2d07ad9ecea779f63cff2
Author: Sergey Vasilenko <email address hidden>
Date: Wed Jul 13 17:12:27 2016 +0300

    Make more strict order between packet installation and related actions

    like:
     * hotplug disabling
     * create interface config files

    Change-Id: I4008be87926f9fec26332747a765a30b749f8cbc
    Closes-bug: #1590875

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341620
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d7b3fa8989a342ea728b0928c0acb82e39f070a2
Submitter: Jenkins
Branch: stable/mitaka

commit d7b3fa8989a342ea728b0928c0acb82e39f070a2
Author: Sergey Vasilenko <email address hidden>
Date: Wed Jul 13 17:12:27 2016 +0300

    Make more strict order between packet installation and related actions

    like:
     * hotplug disabling
     * create interface config files

    Change-Id: I4008be87926f9fec26332747a765a30b749f8cbc
    Closes-bug: #1590875

[Tatyana Kuterina (tkuterina)]

Verified on 9.1 snapshot #115

FUEL_QA_COMMIT=5279ce17271bc0ac6cefc8c0ac4b9482260531ce
UBUNTU_MIRROR_ID=ubuntu-2016-08-03-174238
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-08-09-160321
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-08-09-170321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341621
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=644bf8782ac7cd2b69f6691204439f4bbb97028e
Submitter: Jenkins
Branch: stable/8.0

commit 644bf8782ac7cd2b69f6691204439f4bbb97028e
Author: Sergey Vasilenko <email address hidden>
Date: Wed Jul 13 17:12:27 2016 +0300

    Make more strict order between packet installation and related actions

    like:
     * hotplug disabling
     * create interface config files

    Change-Id: I4008be87926f9fec26332747a765a30b749f8cbc
    Closes-bug: #1590875

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 8.0 + MU3 updates using steps from description.

Bug is not reproducible before updates.

Actual results after updates:
Deployment is successful and basic health checks are passed.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.