reverse DNS not working for CIDR with non 0 last octet

[Marian Gasparovic]

Fix for LP:#1999368 solved wrong DNS zone but for some interfaces reverse DNS is not working

$ dig node6.silo1.lab0.solutionsqa @10.244.40.30

...
;; ANSWER SECTION:
node6.silo1.lab0.solutionsqa. 0 IN A 10.244.40.138
node6.silo1.lab0.solutionsqa. 0 IN A 10.244.40.139
node6.silo1.lab0.solutionsqa. 0 IN A 192.168.35.6
node6.silo1.lab0.solutionsqa. 0 IN A 10.244.40.127
node6.silo1.lab0.solutionsqa. 0 IN A 10.244.40.137
node6.silo1.lab0.solutionsqa. 0 IN A 10.244.40.105
node6.silo1.lab0.solutionsqa. 0 IN A 10.244.8.6
node6.silo1.lab0.solutionsqa. 0 IN A 192.168.33.34
node6.silo1.lab0.solutionsqa. 0 IN A 192.168.36.6

reverse DNS works properly only for 10.244.40.x and 10.244.8.x

$ dig -x 10.244.8.6 @10.244.40.30

; <<>> DiG 9.16.1-Ubuntu <<>> -x 10.244.8.6 @10.244.40.30
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65413
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 39d644935ec438f90100000063997646d93b4fc223566ebd (good)
;; QUESTION SECTION:
;6.8.244.10.in-addr.arpa. IN PTR

;; ANSWER SECTION:
6.8.244.10.in-addr.arpa. 0 IN PTR node6.silo1.lab0.solutionsqa.
6.8.244.10.in-addr.arpa. 0 IN PTR ens4.2678.node6.silo1.lab0.solutionsqa.

;; Query time: 3 msec
;; SERVER: 10.244.40.30#53(10.244.40.30)
;; WHEN: Wed Dec 14 07:07:50 UTC 2022
;; MSG SIZE rcvd: 174

but IPs from other spaces respond like this

$ dig -x 192.168.36.6 @10.244.40.30

; <<>> DiG 9.16.1-Ubuntu <<>> -x 192.168.36.6 @10.244.40.30
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18092
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 35612631b1234bb60100000063997663eb66f4e98e436e13 (good)
;; QUESTION SECTION:
;6.36.168.192.in-addr.arpa. IN PTR

;; ANSWER SECTION:
6.36.168.192.in-addr.arpa. 30 IN CNAME 6.0-26.36.168.192.in-addr.arpa.

;; AUTHORITY SECTION:
0-26.36.168.192.in-addr.arpa. 30 IN SOA 0-26.36.168.192.in-addr.arpa. nobody.example.com. 127 600 1800 604800 30

;; Query time: 0 msec
;; SERVER: 10.244.40.30#53(10.244.40.30)
;; WHEN: Wed Dec 14 07:08:19 UTC 2022
;; MSG SIZE rcvd: 180

Logs
https://oil-jenkins.canonical.com/artifacts/d92ad5f8-160c-435e-b5f6-f450f0c8218c/generated/generated/maas/logs-2022-12-14-09.56.42.tgz

[Marian Gasparovic (marosg)]

We are hitting this bug in rc2 still

[Marian Gasparovic (marosg)]

Logs from one of failed tests

https://oil-jenkins.canonical.com/artifacts/1f542f96-d35b-4311-816a-2d423520de73/generated/generated/maas/logs-2023-01-18-09.24.54.tgz

We have also passing tests in the same environment, I suspect it is a race condition

[Marian Gasparovic (marosg)]

Seeing this bug also in 3.3-rc3

[Marian Gasparovic (marosg)]

3.3-rc4 is hitting this bug as well

https://oil-jenkins.canonical.com/artifacts/fe610c57-32e0-49e2-8f06-a32f904ea608/generated/generated/maas/logs-2023-01-26-13.23.31.tgz

https://oil-jenkins.canonical.com/artifacts/78a72c3b-35f2-483f-ae43-c3d7a32b7bec/generated/generated/maas/logs-2023-01-26-14.29.06.tgz

[Marian Gasparovic (marosg)]

We did not hit it during 3.3.1 testing but we hit it again during maas_3.3.2-13177 testing

https://oil-jenkins.canonical.com/artifacts/09ce07e5-8ebe-4e96-9cd8-5cb31403d7b1/generated/generated/maas/logs-2023-04-07-15.47.30.tgz

https://oil-jenkins.canonical.com/artifacts/f090660e-9b6d-47a9-a749-27916a0267b1/generated/generated/maas/logs-2023-04-07-12.51.39.tgz

[Jerzy Husakowski (jhusakowski)]

Passing runs in silo1 have the following network definitions (forward and reverse DNS works):
- public-space 10.244.8.0/24
- oam-space 10.244.40.0/21
- internal-space 192.168.33.0/25
- ceph-replica-space 192.168.35.0/26
- ceph-access-space 192.168.36.0/26
Failing runs in silo 2 have the following network definitions:
- public-space 10.244.8.0/24 => forward and reverse DNS ok
- oam-space 10.246.40.0/21 => forward and reverse DNS ok
- internal-space 192.168.33.128/25 => rev dns fail
- ceph-replica-space 192.168.35.64/26 => rev dns fail
- ceph-access-space 192.168.36.64/26 => rev dns fail

Swapping the last three spaces between silo1 and silo2 leads to swapping test outcomes: the run in silo2 passes and the one in silo1 fails. It appears that MAAS - bind interaction in updating reverse zone configuration for network prefixes larger than /24 with non-zero last octet leads to incorrect reverse DNS configuration. Presence of these networks leads to reverse DNS failures in queries that correspond to such networks.

Note that multi-node (HA) deployment in silo2 seems to suffer from unexpected pacemaker/corosync behavior that restarts PostgreSQL process periodically. This appears to be a red herring that distracted us for a while, until the tests were switched to run with a single-node MAAS setup, without HA, with same results as above.

[Adam Collard (adam-collard)]

Latest fixes have landed for master, 3.4, and 3.3.

The issue was around MAAS incorrectly identifying the name of the zone (previously hardcoded 0, should have used the last octet of the CIDR)