maas-dhcp-helper stopped working which gives issues with DNS updates

[Tom-Erik Røberg]

We are using snap version/build 3.3.3-13184-g.3e9972c19.

We have trouble with DNS records in MAAS getting stuck to the initial IP that the machine gets during enlistment. After enlistment we configure the other network interfaces (bond with vlans on top) of the machine to use a network for our OAM traffic which is separate from the PXE network used during enlistment and deployment. The machine does not have an IP address configured in the PXE network after deployment. Only a static IP in the OAM network (plus some other non-routed vlans when needed) is configure on the machine. This worked great with MAAS 3.2.7 but broke after upgrading to version 3.3.0 and 3.3.3.

In MAAS version 3.2.7 the temporary FQDN for the machine would be cleared after enlistment such that it's available for use when configuring the other network interfaces on the machine.

With this configuration we get an A record in DNS from the machine name to the IP it is assigned in the OAM network.

hostname.oam.example.com -> oam ip

In version 3.3.0 this is no longer the case, instead wet get two A records in DNS. One from the machine name to the IP in the PXE network which is not active and an interface dependent name to the IP in the OAM network.

hostname.oam.example.com -> pxe ip
interface-name.hostname.oam.example.com -> oam ip

We have traced it to be maas-dhcp-helper not working anymore such that the DNS record created during enlistment is never removed.

Running maas-dhcp-helper after enlistment removes the DNS record in MAAS version 3.2.7 but not in 3.3.0 beta3 and later.

/snap/maas/current/usr/sbin/maas-dhcp-helper notify --action release --mac "8:0:27:2f:66:f8" --ip-family ipv4 --ip "10.1.2.127" --socket /var/snap/maas/common/maas/dhcpd.sock

In MAAS version 3.3.0 beta3 updates for DHCP leases was changed from single update to bulk in commit ffa1fd3c4fe2a32c0e2352fef08f58784bcb2dbd. By reverting this commit we get maas-dhcp-helper to work again on MAAS 3.3.0 beta3 and later.

https://git.launchpad.net/maas/commit/?id=ffa1fd3c4fe2a32c0e2352fef08f58784bcb2dbd

Steps to reproduce:

1. PXE boot a machine.

2. Check DNS in web interface that a DNS record exist for the machine.

3. Run maas-dhcp-helper to remove the lease.

4. Check DNS in web interface to see if DNS record was removed or not.

[Alberto Donato (ack)]

This might be related to LP:2025159

[Bernard Stafford (bernard010)]

LXD in VM is not created in this bug report. May not be related to this bug.
"MAAS getting stuck to the initial IP that the machine gets during enlistment"
Is what this bug report is about not LXD.

[Christian Grabowski (cgrabowski)]

That was just observed on a LXD instance, it's possible it's the same. In this case, does the commissioning IP and a new IP assigned at deployment exist as DNS records or is it just the commissioning IP?

[Tom-Erik Røberg (tom-erik-roberg)]

There exists DNS records for both the enlistment/commissioning IP and the new IP assigned at deployment. Our issue is that the FQDN record is pointing to the enlistment/commissioning IP and not the one we assigned in MAAS before deploying the machine, rendering the machine unreachable on the FQDN. The new IP is automatically assigned an interface name dependent DNS record.

Maybe more a feature then a bug but it would be great to have some way to tell MAAS which interface is consider to be the main interface for the machine which the FQDN should point to.

[Matus Kosut (matuskosut)]

@alexsander-souza good work on the fixes. Do you know when this could be released? I guess fixing this is part of larger DNS changes going on in MAAS so I understand if it's hard to estimate, but any info would be appreciated since we are currently stuck on 3.3.3.