[RFE] Dynamic DHCP allocation pool

[Harald Jensås]

Neuton currently only support configuring the DHCP agent with static/fixed-address allocations. The DHCP client id (the client MAC address in neutron's case) is mapped to a specific IP address in the dhcp server configuration. No range of addresses are made available for clients without a pre-allocated fixed-address.

When network booting on IPv6 this become an issue, the DHCPv6 specification which mandetes use of the DHCP Unique Identifier (DUID) and Identity Association Identifier (IAID) to identify a lease. When network booting, an instance will move trough a minimum of two DHCP clients, and these rareley end up using identical DUID's and IAID's.

The combination of static/fixed-address allocations in the DHCP server and the changeing DUID/IAID of the clients causes the second DHCP client request to get a ``no address available`` reply from the server, and thus the network boot process errors.

NOTE:
  In some cases just the UEFI PXE6 end up doing two cycles
  of DHCPv6 S.A.R.R (Solicit, Advertise, Request, Reply)
  with different IAID's. Because some UEFI firwmare use's a
  non-RFC compliant random generator for the IAID see bug:
    https://bugzilla.tianocore.org/show_bug.cgi?id=1518.

  While this is a bug in UEFI firmware, it's a common enough
  implementation that is used by various hardware vendors
  that it makes sense to workaround the issue where possible.

This RFE is for adding the possibility to make a subnet with dynamic allocation pool(s).

This would solve the network booting issue with changing IAID's described above. A new lease with a new address will be offered during each step of network booting.

For example an instance deployment via Openstack Baremetal service (ironic) would typically include three DHCP clients during provisioning: UEFI firmware, iPXE, ironic-python-agent ramdisk. So a toatal of 3 leases would be consumed to complete the provisioning.

If this RFE is implemented, the dhcp server (dnsmasq) would configure the dhcp-range for a dynamic subnet (or dynamic allocation pool of a subnet) without the ``mode`` set to ``static``. To ensure that the dhcp server only provide a dynamic allocation for the desired ports the ``ignore`` option is used in a ``dhcp-host`` entry with a wildcard ``*`` host (``dhcp-host="*",ignore``). Ports that require dynamic addressing would get a ``dhcp-host`` entry with ``dhcp-host=<MAC_ADDRESS>`` (whithout the ``ignore``) so that these specific ports get addresses from the dynamic allocation pool.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/698206

[Slawek Kaplonski (slaweq)]

Hi,

Sorry if I just didn't understand something properly but in I want to be sure about few things.
1. In case which You described with Ironic instance, each of those 3 dhcp clients will send dhcp request with different DUID/IAID so we should send them 3 different IP addresses?

2. If we will do that per port, how neutron will actually know which IPs are already taken and what IPs can be allocated to "standard" VMs?

[Harald Jensås (harald-jensas)]

Replying to questions in #2 below. It is clear that we require a spec, and carefully think trough this before merging any implementation.

> 1. In case which You described with Ironic instance,
> each of those 3 dhcp clients will send dhcp request
> with different DUID/IAID so we should send them 3
> different IP addresses?

Yes that is the idea. It would be left to the DHCP server to allocate and address from the pool.

> 2. If we will do that per port, how neutron will
> actually know which IPs are already taken and what
> IPs can be allocated to "standard" VMs?

The idea is that neutron won't know exactly which IPs are already taken, it will only know that pool(s) of addresses is outside it's contol. This pool of addresses should never be allocated to "standard" VMs. The dhcp server would have to be the one keeping track of the leased addresses in the dynamic pool.

In the Ironic example, a dynamic address will be used during instance provisioning. Once Ironic is done provisioning, a port with a static address, i.e a "standard" neutron port with a neutron ipam address allocation etc is bound to the final baremetal instance.

There is a HA challenge here, I did'nt think about this before today. Only one dhcp-agent should configure it's dhcp server instance with the dynamic pool. We don't want two dhcp servers handing out addresses from one dynamic pool, we would end up with conflicts. Unless there is a way to keep leased addresses in sync. I don't think dnsmasq can do that. For IPv6 there is a lot of addresses, so we might be able to split a pool into separate a pool into per-agent sub-pools.

We may be able to sync back the used IP addresses to neutron, the dhcp-agent could read the lease file and report back to neutron. But in the dhcpv6 case it's extra tricky since there is no data in the leasefile to identify the host. For dhcpv4 the MAC address is in the lease file. (Ref docstring: https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L810) Another option would be to use the --dhcp-script option in dnsmasq to do a callback to the neutron dhcp-agent when an address is leased, the environment of the script would include "DNSMASQ_MAC - containing the MAC address of the client, if known."

Other options:
1. This patch to dnsmasq completely solves the problem:
 http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2018q4/012707.html

Several people have asked on the list to consider merging this patch, but to date the dnsmasq maintainer did not respond. The maintainer is not a fan of the idea of ignoring the changing IAID in this mail thread: http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2017q1/011267.html

2. If UEFI firmware bug https://bugzilla.tianocore.org/show_bug.cgi?id=1518 is fixed, we could push for iPXE to either support doing a DHCPv6 RELEASE before chainloading. (Or at least make it possible to pass the lease information to the chainloaded OS so that it can take-over or do a RELEASE. ref: http://lists.ipxe.org/pipermail/ipxe-devel/2019-November/006805.html and http://lists.ipxe.org/pipermail/ipxe-devel/2019-December/006828.html

3. It is possible a different DHCP server be...

[Slawek Kaplonski (slaweq)]

Thx for answer to my questions.
So with Your proposal we would have in subnet "allocation_pools" as we have it now and also "dynamic_allocation_pools" which would be used by DHCP servers, right?
And those pools can't overlap.

I will set this rfe as triaged now and I will bring it up for discussion on one of the next drivers meetings.
But I think we will need to have spec for that to discuss details in it later.

[Slawek Kaplonski (slaweq)]

Hi,

We discussed that proposal on our last drivers meeting: http://eavesdrop.openstack.org/meetings/neutron_drivers/2020/neutron_drivers.2020-01-10-14.00.log.html#l-64

What we would like to see is spec with more details about this. Please also described in details various potential alternative solutions as we would like to really carefully evaluate every potential alternative for that.

[Harald Jensås (harald-jensas)]

Hi,

Thank you for sharing the drivers meeting discussion.

Let's put this on hold.
I hope to be able to work out a solution for the Ironic provisioning issues in dnsmasq.

As haleyb mentioned I am currently working on a solution in dnsmasq, and the maintainer is warming up to the idea. It's another approach, not the one first mentioned in this bug, see: http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2020q1/013671.html. I am discussing the approach with the dnsmasq maintainer, he is open to the idea but wants it implemented in a slightly different way. (NOTE, neutron will need an update to how it writes dhcp configuration for dnsmasq when a port have multiple IP's. Neutron does it wrong today, but we can't fix it until we have an implementation in dnsmasq and the final configuration format defined.)

As I started working on a spec for a dynamic address pool, I realized that it's very problematic with dhcp HA. We'd end up with conflicts because the lease database is not synced between the dhcp servers. My conclusion is that we would have to use a different dhcp server, one that could use a central lease database shared between all the servers.

--
Harald

[Slawek Kaplonski (slaweq)]

@Harald: thx for Your comment. Based on it, I will set this BZ as postponed for now. Feel free to change it or ping me to change it whenever You will want to revive work on it.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Harald Jensås (<email address hidden>) on branch: master
Review: https://review.opendev.org/698206
Reason: Aiming for a different solution to the IPv6 network boot/network provisioning issue. https://bugs.launchpad.net/neutron/+bug/1861032