Bootstrap timeout for multiracks

[Dmitry Belyaninov]

Some tests are failed with error:

Bootstrap timeout for nodes: [****]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_7/8/testReport/(root)/deploy_neutron_tun_ha_nodegroups/deploy_neutron_tun_ha_nodegroups/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_7/8/testReport/(root)/deploy_controllers_from_custom_nodegroup/deploy_controllers_from_custom_nodegroup/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_7/8/testReport/(root)/deploy_ceph_ha_nodegroups/deploy_ceph_ha_nodegroups/

[Dmitry Belyaninov (dbelyaninov)]

Also:

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.thread_separate_haproxy/9/testReport/(root)/separate_haproxy/separate_haproxy/

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multiracks_2/8/testReport/(root)/two_nodegroups_network_templates/two_nodegroups_network_templates/

[Alexander Kislitsky (akislitsky)]

Bootstrap of admin node is failed. Seems that it is product-ci server issue.

[Alexander Kislitsky (akislitsky)]

http://paste.openstack.org/show/542602/

[Alexander Kislitsky (akislitsky)]

On the problem environment we have only 3 bootstraped nodes. In virsh we can see all 5 nodes as active. After connecting to the problem node we can see, that configuration of network interfaces is failed: see attachment.

[Alexander Kislitsky (akislitsky)]

Seems that we have no network interfaces for the nodes 4, 5

virsh # net-list --all
 Name State Autostart Persistent
----------------------------------------------------------
 10.0.system_test.ubuntu.ha_neutron.104.104_admin active yes yes
 10.0.system_test.ubuntu.ha_neutron.104.104_management active yes yes
 10.0.system_test.ubuntu.ha_neutron.104.104_private active yes yes
 10.0.system_test.ubuntu.ha_neutron.104.104_public active yes yes
 10.0.system_test.ubuntu.ha_neutron.104.104_storage active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_admin active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_admin2 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_admin3 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_management active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_management2 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_management3 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_private active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_private2 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_private3 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_public active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_public2 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_public3 active yes yes
 9.x.system_test.ubuntu.multiracks_2.9_storage active yes yes
 9.x.system_test.ubuntu.repeatable_image_based.9_admin active yes yes
 9.x.system_test.ubuntu.repeatable_image_based.9_management active yes yes
 9.x.system_test.ubuntu.repeatable_image_based.9_private active yes yes
 9.x.system_test.ubuntu.repeatable_image_based.9_public active yes yes
 9.x.system_test.ubuntu.repeatable_image_based.9_storage active yes yes

[Alexander Kislitsky (akislitsky)]

But we have networks in the configuration for vm:
virsh dumpxml 9.x.system_test.ubuntu.multiracks_2.9_slave-05

Moreover configurations for successfully bootstarped and problem nodes are identical.

[Alexander Kislitsky (akislitsky)]

For fix this bug we need PXE expert. Bug is stable reproduced by job https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_7/

[Georgy Kibardin (gkibardin)]

Guys, were there any changes recently on srv85? Like reinstalling it from scratch, or just libvirt or fuel-devops update? It, actually, was green quite recently.

Also, please ensure that syslinux is installed on the slave where the test is being run.

[Alex Schultz (alex-schultz)]

The success/failure split is when we actually started applying the updates to swarm I believe. Previously the swarm runs weren't getting updated packages. https://review.fuel-infra.org/#/c/23680/ so it's definitely a regression

[Alex Schultz (alex-schultz)]

I tracked this down, so I think it's related to https://review.openstack.org/#/c/343317/. The issue with the environment is that the cobbler dnsmasq is not listening on the additional address for the other network. I manually updated cobbler to include the second rack network and the nodes bootstrapped.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/349721

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/349735

[Alex Schultz (alex-schultz)]

ETA: 8/3

[Dmitry Belyaninov (dbelyaninov)]

Appropriate jobs are passed on snapshot #136.
Mova to FixReleased state

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/349735
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=acbb2ee849bd617ab5eac9164be3fabaf0315260
Submitter: Jenkins
Branch: master

commit acbb2ee849bd617ab5eac9164be3fabaf0315260
Author: Alex Schultz <email address hidden>
Date: Mon Aug 1 16:05:46 2016 -0600

    Update dnsmasq for additional networks

    This change includes an updated listen-address configuration for the
    additional admin networks that can be configured. This change looks
    through the puppet facts to try and identify which interface is
    configured for the extra admin network and uses the ip address that
    exists in the admin network cidr as a listen-address for dnsmasq

    Change-Id: I3322b4e945d3bd016a2c1977dc89c2227735b2e8
    Closes-Bug: #1606914

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[Dmitry Belyaninov (dbelyaninov)]

Please, take a look:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_7/140/console

but may be it is CI issue.

[Dmitry Pyzhov (dpyzhov)]

@Dmitry, it is a bad practice to reopen issue with merged, tested and released fix when you don't have strong evidence that you faced this very issue again. Please create new report.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.