Provisioning as a graph is unstable

[Vladimir Kuklin]

While testing provisioning graph I figured out the following:

1) move_to_bootstrap exits when the node is not yet ready for tasks execution, thus we have sporadic failures of the consequent 'upload_provision_info' task. it may be related to what fix-configs-on-startup script is doing
2) it seems that cobbler does not apply netboot disable/enable flag immediately - we need either to sync it before or introduce some way of waiting for it to sync properly
3) node reboot task does not check system node type and reports that the node has successfully rebooted, while it failed to reboot into the OS actually
4) dnsmasq restart limits for systemd should be raised

[Vladimir Kuklin (vkuklin)]

3) move_to_bootstrap and disable cobbler netboot should happen one by one on the master node
4) move_to_boostrap should fire nailgun agent to generate identity config or we will get into the situation when the node does not have mco id - we need to exit from move_to_bootstrap only when the identity is generated properly, e.g. when nailgun-agent write date to nailgun_uid and fix-configs-on-startup changes the mcollective config and restarts it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/438505

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441695
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6b3d03b1677623ba2a9cc284e3346eec3cbea805
Submitter: Jenkins
Branch: master

commit 6b3d03b1677623ba2a9cc284e3346eec3cbea805
Author: Vladimir Kuklin <email address hidden>
Date: Sun Mar 5 20:10:18 2017 +0300

    Raise start limit burst for dnsmasq

    When we provision nodes, we can
    restart dnsmasq to frequently
    this leads to systemd not starting
    dnsmasq again, thus we have dnsmasq
    stopped and deployment failing.

    Raise limit to 100 starts in 10 seconds
    and also sync cobbler on netboot disable

    Partial-bug: #1667006

    Change-Id: Id14bb2bb162f0a9fc6e0a9a102d98f4f2a6dcf1a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/441957

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/442442

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/442442
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1d68d74cb0fe462ca3050a4d2854b6718e236ec1
Submitter: Jenkins
Branch: stable/ocata

commit 1d68d74cb0fe462ca3050a4d2854b6718e236ec1
Author: Vladimir Kuklin <email address hidden>
Date: Sun Mar 5 20:10:18 2017 +0300

    Raise start limit burst for dnsmasq

    When we provision nodes, we can
    restart dnsmasq to frequently
    this leads to systemd not starting
    dnsmasq again, thus we have dnsmasq
    stopped and deployment failing.

    Raise limit to 100 starts in 10 seconds
    and also sync cobbler on netboot disable

    Partial-bug: #1667006

    Change-Id: Id14bb2bb162f0a9fc6e0a9a102d98f4f2a6dcf1a
    (cherry picked from commit 6b3d03b1677623ba2a9cc284e3346eec3cbea805)

[Vladimir Khlyunev (vkhlyunev)]

We got something similar in several swarm threads: https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.bonding_ha/212/testReport/(root)/deploy_bonding_neutron_tun/deploy_bonding_neutron_tun/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Kozhukalov (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/441957
Reason: this is superseeded by https://review.openstack.org/#/c/435561/

[Dmitry Pyzhov (dpyzhov)]

The bug is about cobbler and we removed cobbler in Newton.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/438505
Reason: This repo is retired now, no further work will get merged.