DNS entries for MAAS servers change to secondary IPs

[Arif Ali]

There is an issue with MAAS that, when you have multiple IP addresses on the same interface, the DNS name for the FQDN of the host can change.

In the scenario were you have MAAS + postgres + HA, you can have 2 IP addresses that are controlled via corosync, and based on where the IP address is, MAAS will automatically create DNS entries, we can see the host resolution[1] and he PTR records [2] as shown. Now the VIPs for maas and postgres are 192.168.210.2 and 192.168.210.3, and hence the FQDN shouldn't really be used for these IP addresses, as ultimately they are secondary.

In a juju deployed environment, in some scenarios we have LMA cross model relations, and the 3 MAAS servers also host telegraf, and then direct their data towards prometheus. This then grabs the IP address of the host as per the relational data (primary interface IP), and grabs the DNS record. In this particular scenario, rather than giving infra1.maas, it will give broam.infra1.maas., which is not the required result.

For a simple reproducer, all you need is a simple MAAS server with single IP address. Then add IP address to the same machine in the same network, and we will see the DNS name of the server fluctuate, below is some output of my reproduced issue on a single MAAS server, noting that each time I am expecting 192.168.1.22 to be my MAAS FQDN

ip a a 192.168.1.8/24 dev eth0

log message in regiond.log

2022-12-15 19:18:05 maasserver.region_controller: [info] Reloaded DNS configuration; ip 192.168.1.8 connected to maas on eth0

~~~
root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa
<SNIP>
8 30 IN PTR maas.maas.
<SNIP>
22 30 IN PTR eth0.maas.maas.
~~~

ip a a 192.168.1.7/24 dev eth0

~~~
root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa
<SNIP>
22 30 IN PTR maas.maas.
<SNIP>
7 30 IN PTR eth0.maas.maas.
8 30 IN PTR eth0.maas.maas.
~~~

ip a a 192.168.1.6/24 dev eth0

~~~
root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa
<SNIP
8 30 IN PTR maas.maas.
<SNIP
6 30 IN PTR eth0.maas.maas.
22 30 IN PTR eth0.maas.maas.
7 30 IN PTR eth0.maas.maas.
~~~

As we can see from the above output, we can see the undesired output.

My expectation is that .22 is always maas.maas. in my scenario, as that should be consistent, no matter what the secondary IP address is set

This has been tested in 2.9.2, as well as my lab with 3.2.6

[1] https://paste.ubuntu.com/p/CZ2wfQgnkx/
[2] https://paste.ubuntu.com/p/7WmYFwb2nT/

### ADDED

As this patch https://bugs.launchpad.net/maas/+bug/1600259, it takes IPs reversely and generate zone file.

https://github.com/maas/maas/blob/0c069f0aafbb650928f6c8536debd0de79eb7484/src/maasserver/dns/zonegenerator.py#L399

e.g

assume there is one nic which has 172.16.116.131 as a primary IP.

after adding one more ip to the same nic, zone file refer to secondary IP instead of primary IP.

{<Domain: name=maas>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), <Subnet: 172.16.116.0/24:172.16.116.0/24(vid=0)>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), 'reverse': defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)})}

I printed info from code, xtrusia.maas had 172.16.116.131 in the beginning, but after I added 172.16.116.254, it is changed to later.

[Alberto Donato (ack)]

Does the boot interface for the host change at any time?

[Seyeong Kim (seyeongkim)]

I've been analyzing this, and I checked that maas picked it as reverse order as

https://bugs.launchpad.net/maas/+bug/1600259

and

https://github.com/maas/maas/blob/master/src/maasserver/dns/zonegenerator.py#L399

[Seyeong Kim (seyeongkim)]

I can test this in virtual machine easily.

[Seyeong Kim (seyeongkim)]

Added description but comment as well

As this patch https://bugs.launchpad.net/maas/+bug/1600259, it takes IPs reversely and generate zone file.

https://github.com/maas/maas/blob/0c069f0aafbb650928f6c8536debd0de79eb7484/src/maasserver/dns/zonegenerator.py#L399

e.g

assume there is one nic which has 172.16.116.131 as a primary IP.

after adding one more ip to the same nic, zone file refer to secondary IP instead of primary IP.

{<Domain: name=maas>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), <Subnet: 172.16.116.0/24:172.16.116.0/24(vid=0)>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), 'reverse': defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)})}

I printed info from code, xtrusia.maas had 172.16.116.131 in the beginning, but after I added 172.16.116.254, it is changed to later.

[Seyeong Kim (seyeongkim)]

reproduction step further.

0. deploy machine for maas. ( in my case, virtual machine )
1. deploy maas
- sudo snap install maas
- setup maas-test-db
- setup admin

2. ssh into the machine
- ip a
ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    ...
    inet 10.0.0.55/24 brd 10.0.0.255 scope global ens4f0
       valid_lft forever preferred_lft forever
- cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
...
@ 30 IN NS maas.
55 30 IN PTR node-20.maas.

3. add additional ip to ens4f0
- sudo ip a a 10.0.0.100 dev ens4f0
- ip a
ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:97:25:20 brd ff:ff:ff:ff:ff:ff
    altname enp0s4f0
    inet 10.0.0.55/24 brd 10.0.0.255 scope global ens4f0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global ens4f0
       valid_lft forever preferred_lft forever

4. few mins later
- cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
...
100 30 IN PTR node-20.maas.
55 30 IN PTR ens4f0.node-20.maas.

[Jerzy Husakowski (jhusakowski)]

Seyeong, is this reproducible with the latest edge MAAS (3.4.0-beta3)? We've been investigating and fixing a similar issue and the fix recently landed there.

[Seyeong Kim (seyeongkim)]

Hello @jhusakowski

I tested it with 3.4.0-beta3 snap in edge. but it has same behavior

ubuntu@node-20:~$ cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
; Zone file modified: 2023-06-09 00:07:55.343496.
$TTL 30

...

@ 30 IN NS maas.
51 30 IN PTR node-20.maas.

After

ubuntu@node-20:~$ cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
; Zone file modified: 2023-06-09 00:10:09.763022.
$TTL 30

...

@ 30 IN NS maas.
100 30 IN PTR node-20.maas.
51 30 IN PTR ens4f0.node-20.maas.

[Seyeong Kim (seyeongkim)]

I analyzed a bit more, reverse is not the thing.

actually mappings Subnet has already modified values.

keep researching..

{
  <Domain: name=maas>:
    defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>,
    {
      'node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.100'}, 4, None, None),
      'ens4f0.node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.51'}, 4, None, 1)
    }),
  <Subnet: 10.0.0.0/24:10.0.0.0/24(vid=0)>:
    defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>,
    {
      'node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.100'}, 4, None, None),
      'ens4f0.node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.51'}, 4, None, 1)
    }
  ),
  'reverse':
    defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>,
    {
      'node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.100'}, 4, None, None),
      'ens4f0.node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.51'}, 4, None, 1)
    }
  )
}

[Seyeong Kim (seyeongkim)]

I checked code a bit more.

https://pastebin.ubuntu.com/p/jYhdxqKQfR/

one more left outer join make order changed

i think this is the point.

Could you please give me any advices?

Thanks.

[Seyeong Kim (seyeongkim)]

I proposed merge request

[Jerzy Husakowski (jhusakowski)]

There's a newer fix in 3.4.0, it'd be interesting to know if you can reproduce the issue using that fix. Could you test it on MAAS from this ppa? https://launchpad.net/~cgrabowski/+archive/ubuntu/maas-next

[Seyeong Kim (seyeongkim)]

@jhusakowski

I can't install that ppa I see build is failed.

I'll test after fixing it.

Thanks in advance.!

[Arif Ali (arif-ali)]

@Jerzy, I am running the 3.4/edge snap

root@maas:~# snap list maas
Name Version Rev Tracking Publisher Notes
maas 3.4.0~rc1-14285-g.358798bbf 29143 3.4/edge canonical✓ -

I am presuming this should have the aforementioned fix, if so, I was able to reproduce this in my lab

root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa | grep maas.maas
6 30 IN PTR maas.maas.
22 30 IN PTR eth0.maas.maas.

root@maas:~# grep -E "192.168.1.6|192.168.1.22" /var/snap/maas/current/bind/*
grep: /var/snap/maas/current/bind/cache: Is a directory
/var/snap/maas/current/bind/zone.maas:maas 30 IN A 192.168.1.6
/var/snap/maas/current/bind/zone.maas:eth0.maas 30 IN A 192.168.1.22
/var/snap/maas/current/bind/zone.maas-internal:192-168-1-0--24 15 IN A 192.168.1.6
/var/snap/maas/current/bind/zone.maas-internal:192-168-1-0--24 15 IN A 192.168.1.22

[Mauricio Faria de Oliveira (mfo)]

MAAS bug-council,

Can this fix be included/targeted to series 3.3 and 3.2 as well?

If so, SEG will proceed with older series afterward.

Thanks,
Mauricio

[Jerzy Husakowski (jhusakowski)]

The change proposed by Seyeong landed in master (3.5), but not yet in 3.4. We targeted the issue to 3.4 as well and need to confirm the fix. The intention is to backport the fix to all affected supported revisions.

[Mauricio Faria de Oliveira (mfo)]

Hello Arif, Seyeong, or anyone else affected,

Accepted maas 3.1.3 into ppa:maas/3.1-next and snap:3.1/edge
(code version 3.1.3-10930-g.2eb4e7525).

Please test this update and provide your feedback on this bug:

If it fixes the bug for you, please add a comment mentioning the version you tested and what testing has been performed, and change the tag from verification-needed-maas-3.1 to verification-done-maas-3.1.

If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-maas-3.1.

The update will be released after the bug(s) have been verified and quality assurance testing is successful.

Thank you in advance for helping!

[Mauricio Faria de Oliveira (mfo)]

Verification done with MAAS 3.1.3.

The DNS record for the MAAS server does (not) change
with MAAS 3.1.2 (3.1.3) when adding an IP address to
the network interface of the MAAS server.

On MAAS 3.1.2: DNS record changes from IP .2 (to .10)
On MAAS 3.1.3: DNS record continues at IP .2 (no .10)

MAAS 3.1.2:

 $ snap list maas
 Name Version Rev Tracking Publisher Notes
 maas 3.1.2-10926-g.5ad7060e5 30495 3.1/stable canonical✓ -

 $ maas admin version read | jq '.version, .subversion'
 "3.1.2"
 ""

 $ ip addr list dev nic-maas | grep -w inet
     inet 10.31.200.2/24 brd 10.31.200.255 scope global nic-maas

 $ grep $(hostname) /var/snap/maas/current/bind/zone.200.31.10.in-addr.arpa
 2 30 IN PTR maas-srv-31snap.maas.

 $ host maas-srv-31snap.maas 10.31.200.2 | grep 'has address'
 maas-srv-31snap.maas has address 10.31.200.2

 $ sudo ip addr add 10.31.200.10/24 dev nic-maas

 $ ip addr list dev nic-maas | grep -w inet
     inet 10.31.200.2/24 brd 10.31.200.255 scope global nic-maas
     inet 10.31.200.10/24 scope global secondary nic-maas

 (wait a bit)

 $ grep "Reloaded DNS configuration; ip .* connected to $(hostname)" /var/snap/maas/common/log/regiond.log
 2024-02-21 21:34:05 maasserver.region_controller: [info] Reloaded DNS configuration; ip 10.31.200.10 connected to maas-srv-31snap on nic-maas

 Note the change here:

 $ grep $(hostname) /var/snap/maas/current/bind/zone.200.31.10.in-addr.arpa
 10 30 IN PTR maas-srv-31snap.maas.
 2 30 IN PTR nic-maas.maas-srv-31snap.maas.

 $ host maas-srv-31snap.maas 10.31.200.2 | grep 'has address'
 maas-srv-31snap.maas has address 10.31.200.10

 $ sudo ip addr del 10.31.200.10/24 dev nic-maas

MAAS 3.1.3:

 $ sudo snap refresh --channel=3.1/edge maas

 $ ip addr list dev nic-maas | grep -w inet
     inet 10.31.200.2/24 brd 10.31.200.255 scope global nic-maas

 $ grep $(hostname) /var/snap/maas/current/bind/zone.200.31.10.in-addr.arpa
 2 30 IN PTR maas-srv-31snap.maas.

 $ host maas-srv-31snap.maas 10.31.200.2 | grep 'has address'
 maas-srv-31snap.maas has address 10.31.200.2

 $ sudo ip addr add 10.31.200.10/24 dev nic-maas

 $ ip addr list dev nic-maas | grep -w inet
     inet 10.31.200.2/24 brd 10.31.200.255 scope global nic-maas
     inet 10.31.200.10/24 scope global secondary nic-maas

 (wait a bit)

 $ grep "Reloaded DNS configuration; ip .* connected to $(hostname)" /var/snap/maas/common/log/regiond.log
 2024-02-21 21:34:05 maasserver.region_controller: [info] Reloaded DNS configuration; ip 10.31.200.10 connected to maas-srv-31snap on nic-maas
 2024-02-21 21:38:29 maasserver.region_controller: [info] Reloaded DNS configuration; ip 10.31.200.10 connected to maas-srv-31snap on nic-maas

 Note the change here, but this time the IP for first line (2) didn't change:

 $ grep $(hostname) /var/snap/maas/current/bind/zone.200.31.10.in-addr.arpa
 2 30 IN PTR maas-srv-31snap.maas.
 10 30 IN PTR nic-maas.maas-srv-31snap.maas.

 $ host maas-srv-31snap.maas 10.31.200.2 | grep 'has address'
 maas-srv-31snap.maas has address 10.31.200.2

[Mauricio Faria de Oliveira (mfo)]

MAAS 3.1.3 has been released:
- deb: ppa:maas/3.1 (1:3.1.3-10930-g.2eb4e7525-0ubuntu1~20.04.1)
- snap: 3.1/stable (3.1.3-10930-g.2eb4e7525)