system tests booting of 5 slave nodes somtimes fails

[Vladimir Kuklin]

{"build_id": "2014-05-07_01-00-26", "mirantis": "yes", "build_number": "184", "ostf_sha": "fe718434f88f2ab167779770828a195f06eb29f8", "nailgun_sha": "6b5e2797dae6a3b803b3ac7102ae3d9164a013cf", "production": "docker", "api": "1.0", "fuelmain_sha": "cb99264a1d5c5cd949f95a835f86ea1eb00c766f", "astute_sha": "9c83d3ecec69df03cd94620e2df92249ba4ec786", "release": "5.0", "fuellib_sha": "616a164132a6a195b756236923bc2f4345bc93f0"}

try to run HA test with 5 slave nodes. One of the nodes cannot boot over PXE.
Also, all the container services are configured and work fine. All the other nodes boot just fine. It looks like cobbler/dnsmasq/tftpd server cannot handle 5 simultaneous connections sometimes.

This may be due to highload issues

[Vladimir Kuklin (vkuklin)]

after snapshot revert it seems that this some issue with dhcrelay performance or iPXE emulation of slaves. if I insert sleep for 1 second between slaves starts the problem goes away

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/92657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92657
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=81348bd8c8f7cba312ff76872ec1ec464967acd2
Submitter: Jenkins
Branch: master

commit 81348bd8c8f7cba312ff76872ec1ec464967acd2
Author: Vladimir Kuklin <email address hidden>
Date: Wed May 7 22:45:11 2014 +0400

    Add 2-sec sleep between slave nodes start

    There are some issues either with iPXE or
    dhcrelay which lead to the problem with
    PXE booting of virtual slaves. This
    workaround makes bootstrapping success
    more likely

    Change-Id: I85584f61cd06a7c5cd1f73e6638c9475d0f86ac2
    Partial-Bug: #1317213

[Mike Scherbakov (mihgen)]

Vladimir is visiting OpenStack design summit, so deferring this to 5.1 to verify.

[Nastya Urlapova (aurlapova)]

Added fix for dhcrelay https://github.com/stackforge/fuel-main/blob/master/fuelweb_test/models/environment.py#L398, looks like issue fixed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/99169

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99169
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=947f94c733eade81a6b79c98155ef027d883cc60
Submitter: Jenkins
Branch: stable/4.1

commit 947f94c733eade81a6b79c98155ef027d883cc60
Author: Vladimir Kuklin <email address hidden>
Date: Wed May 7 22:45:11 2014 +0400

    Add 2-sec sleep between slave nodes start

    There are some issues either with iPXE or
    dhcrelay which lead to the problem with
    PXE booting of virtual slaves. This
    workaround makes bootstrapping success
    more likely

    Change-Id: I85584f61cd06a7c5cd1f73e6638c9475d0f86ac2
    Partial-Bug: #1317213
    (cherry picked from commit 81348bd8c8f7cba312ff76872ec1ec464967acd2)