Potential race between neutron port update and node pxe boot

[Adam Gandelman]

Splitting off a new bug from Bug #1300589 to track progress on one specific issue.

There is a potential race condition in the pxe driver where it will update neutron ports with DHCP data for a soon-to-be-booted ironic node. This data sets, among other things, the tftp next server address. The update is asynchronous on the Neutron side. After the update request is sent, the node is immediately powered on. If using the ssh power driver with fast booting virtual machines, there is a potential race where the node attempts pxe boot before the neutron agents have processed the updates and reconfigured DHCP servers appropriately. Copying from the original bug, Robert describes the specific issues in the driver:

This is the problem:
       _create_token_file(task, node)
        _update_neutron(task, node)
        manager_utils.node_set_boot_device(task, node, 'pxe', persistent=True)
        manager_utils.node_power_action(task, node, states.REBOOT)

There's no synchronisation with neutron (neither poll nor call-back) to know that (all) the dnsmasq processes serving that port have been updated - and a VM that boots quickly may DHCP before the dnsmasq is hupped.

[Adam Gandelman (gandelman-a)]

Temporary workaround merged @ https://review.openstack.org/#/c/91719/

[Armando Migliaccio (armando-migliaccio)]

This bug is > 172 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/327589

[Pavlo Shchelokovskyy (pshchelo)]

in fact, this affects not only (fast) booting virtual BMs on devstack. It could also affect booting real BMs when Neutron is backed by some slow SDN.

as there is already a config option for setting the delay
http://git.openstack.org/cgit/openstack/ironic/tree/ironic/dhcp/neutron.py#n205

operators might set this value to account for the real port update delay in their deployment.

I'd say the bug is fixed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lucas Alvares Gomes (<email address hidden>) on branch: master
Review: https://review.openstack.org/327589
Reason: Thanks anton for pointing it out