[2.2] Expire the DHCP lease after commissioning or make the default-lease-time configurable

[Jorge Niedbalski]

[Environment]

MAAS 2.0
Xenial

[Problem description]

It would be desirable for the dynamic IP addresses used during commissioning
to be released upon shutdown instead of waiting for the default lease time to be reached.

The use case example is a user that configured a small dynamic range of 5 IP addresses
then they started 10+ for enlisting, the results are 5 machines will correctly enlist while the other 5 will be stuck waiting for the original enlisting IP's to expire (10 minutes by default).

[Possible Solutions]

- Expire/Release the DHCP leases upon shutdown.
- Expose the default-lease-time configuration or make it configurable via DHCP snippets.

[Bert JW Regeer (bregeer-ctl)]

Update:

It looks like MaaS doesn't release the "dynamic IP" assigned to a node until it is being deployed, so as soon as I deployed the existing 5 machines, the other 5 machines started booting.

To the 5 machines that were enlisted, and are now being deployed, it automatically created a new IP address and set it to "Automatic" in the network list. Thereby removing it from "Observed" which allowed the other nodes to enlist.

That means right now realistically you need an IP address space that is twice as large as the amount of machines in the network if you want to be able to have the machines standing by in the "Ready" state. 1 IP for the Dynamic range, and 1 IP for the host when it goes to "Deployed". This seems like a waste of IP space because if all the machines are deployed 50% of your IP space is no longer used...

[Brendan Donegan (brendan-donegan)]

There might be a technical reason why this is necessary, but I'll let someone else answer on that, however you *can* configure the default-lease-time (and a lot of other things), by using DHCP snippets @ /MAAS/#/settings/dhcp

[Andres Rodriguez (andreserl)]

MAAS 2.1 reduces the time the lease is valid for everything on the dynamic rnage. It wont immediately expire but it would leave long enough for one commissioning cycle.

[Andres Rodriguez (andreserl)]

@Bert, I think there may be a confusion on the MAAS behavior. In MAAS 2.0+, the machine in a subnet has a "dynamic" range. This "Dynamic" range is use /only/ for enlistment and commissioning purposes (and for any other machine that you need to get an IP, without you caring what IP that machine has).

When you deploy a machine, MAAS will pick any machine /outside/ the dynamic range and in the rest of the "available" network. That means that the dynamic range will always be available for the randomness that may occur on your network, but the rest of the subnet is used for the deployed machines.

What you are seeing is correct behavior. What happens is the machine enlists and commissions with a random IP from the /dynamic/ range, and a DNS record is created for that process. However, when you deploy a machine the IP the machine gets is outside of the /dynamic/ range and the DNS record gets updated to reflect that. For example:

192.168.10.0/MAAS

1. 192.168.10.1 -> GW
2. 192.168.10.2 - 192.168.10.102 -> dynamic range
3. 192.168.10.103 - 192.168.10.254 -> rest of the network.

So a machine will commission with an IP inside of (2), a DNS record will be created. When you deploy the machine, MAAS will automatically pick an IP from (3) and update the record if present.

Now, that said, keep in mind that the lease time inside the dynamic range is less than 10 minutes. So for example:

a. enlist/commissioning machine
b. maas creates DNS record for IP inside (2)
c. 10 minutes later, isc-dhcp tells MAAS the lease expired
d. MAAS clears the DNS record

Hope this helps clarify the usage.

[Bert JW Regeer (bregeer-ctl)]

@andres:

The behavior I am seeing is not what you are telling me it should be. It never seems to time out the lease.

Here is what I am seeing:

1. Host boots, gets IP from the dynamic range
2. MaaS sets this IP as Observed
3. Node finishes enlisting
4. Node shuts down
5. IP stays as "Observed" in what seems like perpetuity (I waited 15+ minutes)
6. Other nodes that couldn't get an DHCP address don't PXE boot and don't enlist.

Now, as soon as I commission the nodes, the IP's get removed from Observed and yes, MaaS assigns a "static" IP.

At that point my remaining hosts enlisted without issues.

Unfortunately journalctl doesn't have information back to when I was trying to troubleshoot this, I tried the following:

1. Remove the leases files from /var/lib/maas/dhcp/*.leases*
2. Restart maas-dhcpd
3. Watch node attempt to PXE boot, but it would log that no leases were available

I was discussing this with @roaksoax on IRC, apparently ISC-DHCP is supposed to notify MaaS, so removing the leases file should have cleared the leases that ISC knew about and it should have let the new nodes enlist. Does MaaS somehow notify DHCP about the existing "Observed" entries?

[Victor Tapia (vtapia)]

From my last tests, if we try to use dhclient in the preseed files to release the IPs after an enlistment/commission, the DHCP server will still keep the lease set to 'active' and not 'free' until the 10 min lease time expires. Regarding the "Observed" state: when the lease is released/expired, it's removed from the "Used" list via maas-dhcp-helper. But in the release case, the lease is still active in the DHCP server until the expiration time hits, rendering it unusable for other machines until then.

So, in short, a lease can't be re-used until it expires (10 mins).

Is it possible to shorten this time by reducing the timeout or forcing the server to release the IP?

[Andres Rodriguez (andreserl)]

As per discussion,. the recommend way is to add a DHCP snippet for the subnet with the desired lease time. The lease time added on the snippet will be the preferred one.