MAAS 3.4 RC (Aug 2nd 2023) breaks DNS

[Kevin Reeuwijk]

After the MAAS snap in the 3.4/beta channel auto-upgraded to the 3.4.0~rc1-14297-g.b27476324 build, DNS no longer functions properly. New entries in the DNS zones in MAAS do not show up in their respective zone files and DNS queries for these new records fail.

Restarting or reloading the MAAS snap will populate the zone files and get queries for the new records to resolve, but this action causes in-flight PXE deployments to crash as all of MAAS restarts. This solution also isn't practical for us since we deploy Kubernetes clusters with MAAS and need DNS to work immediately, otherwise the cluster fails to build.

As I suspected that maybe this had something to do with AppArmor, I tried disabling it and setting the profiles to complain mode, but none of these had any positive effect. So this was rolled back to the original AppArmor configuration.

Regiond logs are attached.

[Christian Grabowski (cgrabowski)]

seems there's a DB query outside of a database thread.

[Anton Troyanov (troyanov)]

Hi Kevin,

Thanks for the bug report. We are planning to land a fix for 3.4 asap but it will take some time for us to make it to 3.4/beta channel.

Meanwhile maybe you can try 3.4/edge (after a new version is published) to check if that actually solves the issue?

[Kevin Reeuwijk (kreeuwijk)]

Hi Anton,

yes I can easily try a 3.4/edge build when you've made it available. Just ping me and I'll get right on it.

[Anton Troyanov (troyanov)]

Kevin, 3.4.0~rc1-14298-g.8b414218b is now available in 3.4/edge

[Kevin Reeuwijk (kreeuwijk)]

Upgrade to 3.4.0~rc1-14298 succeeded, testing now

[Kevin Reeuwijk (kreeuwijk)]

Anton,

I can confirm the edge release works, deployments are working again. Thanks!
I have one lingering issue, as you can see from the attached screenshot.

Deleting a DNS record causes a ghost record to appear in both the MAAS database as well as the actual zonefile. The ghost record has the form of A-B-C-D for an IP address of a.b.c.d.
It is impossible to delete the record from the GUI.

This happens with both 3.4.0~rc1-14297-g.b27476324 and the new 3.4.0~rc1-14298-g.8b414218b version.
While not a huge problem, it would be appreciated if this can be fixed in the next drop.

-Kevin

[Anton Troyanov (troyanov)]

Kevin,

This happens because after DNS record is removed, MAAS still tracks IP address marked as USER_RESERVED.
You can find it in the database table `maasserver_staticipaddress`

I agree that it is not very clear from the UX perspective, but as a workaround I can suggest you to use MAAS CLI to remove unwanted record: `maas $LOGIN ipaddresses release ip=a.b.c.d`

I've checked 3.2 and 3.3 and it behaves exactly the same way.

But if you consider this to be a bug, may I ask you to open a new bug in LP, so we can track it separately.

Thanks in advance!

[Kevin Reeuwijk (kreeuwijk)]

Hi Anton,

thank you for the workaround to clean them up, I will try that.

It also happens when I manually create a DNS record (for a random IP) in the MAAS GUI and then delete it. So I don't think this is intended to work that way, it didn't in prior releases at least. So I'll raise a bug for it.

-Kevin