dhcp scheduling is racey with first port creation

[Kevin Benton]

By deferring scheduling of networks to dhcp agents until the first VM port is created, there is a possibility to create the VM port and have it issuing DHCP requests before the DHCP agent gets a chance to create a port on the network, launch its dnsmasq process, etc.

By scheduling the agent at subnet creation time instead of the first port creation, we can buy extra time to help avoid this situation (DHCP scheduling notification sent before VM port creation rather than after). Additionally, scripts can watch the DHCP port state after creating the subnet to know when it should be safer to proceed.

Ultimately we will need a way to know that the DHCP agent is truly ready (dnsmasq process running with updated config), but this will require new state to be stored and passed from the agent which will probably require a spec so we can address 'readiness' in a broader fashion.

[YAMAMOTO Takashi (yamamoto)]

have you seen the race in the field?
or just code inspection?

[Kevin Benton (kevinbenton)]

race with a heavily loaded dhcp agent doing scale testing.

The instance usually eventually retries for DHCP but it could take several minutes.

[Akihiro Motoki (amotoki)]

Sounds reasonable. I believe the current behavior is to allow operators to decide agent assignment, but I think it is not an usual case.
Even if a network is scheduled when a subnet is created, operators can move dhcp agents as they want, so I don't see any problems on changing the timing of dhcp agent scheduling.

[Kevin Benton (kevinbenton)]

If operators want to manually schedule, they can also do it after the network is created but before the subnet is created.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/163672
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=05f234481474aa05f59c4af459b4343d21397afc
Submitter: Jenkins
Branch: master

commit 05f234481474aa05f59c4af459b4343d21397afc
Author: Kevin Benton <email address hidden>
Date: Wed Mar 11 18:32:52 2015 -0700

    Schedule net to a DHCP agt on subnet create

    Change the DHCP notifier behavior to schedule a network
    to a DHCP agent when a subnet is created rather than
    waiting for the first port to be created.

    This will reduce the possibility to get a VM port created
    and have it send a DHCP request before the DHCP agent is
    ready. Before, the network would be scheduled to an agent
    as a result of the API call to create the VM port, so the
    DHCP port wouldn't be created until after the VM port.
    After this patch, the network will have been scheduled to
    a DHCP agent before the first VM port is created.

    There is still a possibility that the DHCP agent could be
    responding so slowly that it doesn't create its port and
    activate the dnsmasq instance before the VM sends traffic.
    A proper fix will ensure that the dnsmasq instance is
    truly ready to serve requests for a new port will require
    significantly more code for barriers (either on the subnet
    creation, port creation, or the nova boot process) are too
    complex to add this late in the cycle.

    This patch also eliminates the logic in the n1kv plugin that
    was already doing the same thing.

    Closes-Bug: #1431105
    Change-Id: I1c1caed0fdda6b801375a07f9252a9127058a07e