DNS Forward failures

[Jeffrey Chang]

During testing MAAS 3.3.4, SQA bump into DNS forward failures in magpie layer.
https://solutions.qa.canonical.com/testruns/326dcd1e-9572-4253-887d-4881c38bfb19
We see this in 1 out of 5 runs.
MAAS log here, https://oil-jenkins.canonical.com/artifacts/326dcd1e-9572-4253-887d-4881c38bfb19/generated/generated/maas/logs-2023-06-20-19.14.47.tgz

App Version Status Scale Charm Channel Rev Exposed Message
magpie-ceph-access-space blocked 6 magpie latest/edge 13 no ports ok, bonds ok, icmp ok, local hostname ok (meowth), fwd dns failed: ['3'], local mtu ok, required: 9000
magpie-ceph-replica-space blocked 6 magpie latest/edge 13 no ports ok, bonds ok, icmp ok, local hostname ok (meowth), fwd dns failed: ['3'], local mtu ok, required: 9000
magpie-internal-space blocked 6 magpie latest/edge 13 no ports ok, bonds ok, icmp ok, local hostname ok (meowth), fwd dns failed: ['3'], local mtu ok, required: 1500
magpie-oam-space active 6 magpie latest/edge 13 no ports ok, bonds ok, icmp ok, local hostname ok (meowth), dns ok, local mtu ok, required: 1500
magpie-public-space active 6 magpie latest/edge 13 no ports ok, bonds ok, icmp ok, local hostname ok (meowth), dns ok, local mtu ok, required: 1500

In juju crashdump, https://oil-jenkins.canonical.com/artifacts/326dcd1e-9572-4253-887d-4881c38bfb19/generated/generated/magpie/juju-crashdump-magpie-2023-06-20-19.13.29.tar.gz
2023-06-20 19:05:16 DEBUG unit.magpie-public-space/0.juju-log server.go:316 magpie:3: DNS Reverse command: /usr/bin/dig -x 10.244.8.134 +short +tries=1 +time=3
2023-06-20 19:05:16 INFO unit.magpie-public-space/0.juju-log server.go:316 magpie:3: Reverse result for unit_id: 3, hostname: pangoro.silo2.lab0.solutionsqa.
eth1.2678.pangoro.silo2.lab0.solutionsqa., exitcode: 0
2023-06-20 19:05:16 INFO unit.magpie-public-space/0.juju-log server.go:316 magpie:3: Reverse OK for unit_id: 3
2023-06-20 19:05:16 INFO unit.magpie-public-space/0.juju-log server.go:316 magpie:3: Forward lookup for hostname: pangoro.silo2.lab0.solutionsqa.
eth1.2678.pangoro.silo2.lab0.solutionsqa., node: magpie-public-space/3, unit_id: 3
2023-06-20 19:05:16 DEBUG unit.magpie-public-space/0.juju-log server.go:316 magpie:3: DNS Forward command: /usr/bin/dig pangoro.silo2.lab0.solutionsqa. +short +tries=1 +time=3
2023-06-20 19:05:16 INFO unit.magpie-public-space/0.juju-log server.go:316 magpie:3: Forward result for unit_id: 3, ip: 10.244.8.134
10.246.64.161
10.246.64.210
192.168.33.165
192.168.36.71
192.168.35.71, exitcode: 0
2023-06-20 19:05:16 INFO unit.magpie-public-space/0.juju-log server.go:316 magpie:3: Forward OK for unit_id: 3
2023-06-20 19:05:16 ERROR unit.magpie-public-space/0.juju-log server.go:316 magpie:3: Original IP and Forward MATCH FAILED for unit_id: 3, Original: 10.244.8.134, Forward: Can not resolve hostname to IP '10.244.8.134\n10.246.64.161\n10.246.64.210\n192.168.33.165\n192.168.36.71\n192.168.35.71'

[Alberto Donato (ack)]

It looks like this could be a duplicate of LP:2012801

[Alberto Donato (ack)]

From what I see the error comes from the magpie charm because the check for forward resolution expects a single IP is returned in forward_dns() here: https://opendev.org/openstack/charm-magpie/src/branch/master/src/lib/charms/layer/magpie_tools.py

[Adam Collard (adam-collard)]

https://review.opendev.org/c/openstack/charm-magpie/+/888716

[Nobuto Murata (nobuto)]

In what circumstances MAAS returns multiple IPs to one DNS forward resolution? Why now do we have to update a logic in Magpie if it hasn't been changed for a while?

[Adam Collard (adam-collard)]

Magpie and MAAS both need to conform to DNS RFCs and norms, among which it is possible for there to be more than one IP for a given hostname lookup.

We are investigating why SolQA runs seem to only be hitting this 20% of the time, but in the meantime have landed fixes to Magpie for correctness.

[Alberto Donato (ack)]

Looking at the logs from one of the recently failed runs after the magpie charm fix, it seems sometimes the DNS is not responding:

unit-magpie-ceph-replica-space-0: 2023-07-26 08:28:48 DEBUG unit.magpie-ceph-replica-space/0.juju-log magpie:4: DNS Forward command: /usr/bin/dig 69.64-26.35.168.192.in-addr.arpa. +short +tries=1 +time=3
...
unit-magpie-ceph-replica-space-0: 2023-07-26 08:28:48 INFO unit.magpie-ceph-replica-space/0.juju-log magpie:4: Forward result for unit_id: 2, ip: No forward response, exitcode: 1

[Adam Collard (adam-collard)]

Extracted logs from magpie-ceph-access-space-0.log in the juju crashdump

https://pastebin.canonical.com/p/6q2hPxPfJv/

Note that magpie does a reverse lookup first - and gets a odd looking .in-addr.arpa

[Alberto Donato (ack)]

When prefix length for the subnet is less than a /24, MAAS generates glue records (see https://github.com/maas/maas/blob/master/src/maasserver/dns/zonegenerator.py#L482-L509), where it returns a PTR record pointing to another PTR for the corresponding /24.

Clients usually automatically resolve PTRs until they get a meaningful record (A, AAAA, CNAME).

The magpie charm should probably do the same, checking what kind of result is returned by dig, and resolving again if it's a PTR.

[Alberto Donato (ack)]

Actually, the reverse lookup returns a CNAME pointing to the subnet section, but that record has no address associated to it.

As an example:

$ dig -x 192.168.36.71 +nocomments

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> -x 192.168.36.71 +nocomments
;; global options: +cmd
;71.36.168.192.in-addr.arpa. IN PTR
71.36.168.192.in-addr.arpa. 17 IN CNAME 71.64-26.36.168.192.in-addr.arpa.
64-26.36.168.192.in-addr.arpa. 17 IN SOA silo2.lab0.solutionsqa. nobody.example.com. 180 600 1800 604800 30
64-26.36.168.192.in-addr.arpa. 30 IN SOA silo2.lab0.solutionsqa. nobody.example.com. 180 600 1800 604800 30
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Aug 01 07:02:31 UTC 2023
;; MSG SIZE rcvd: 190

$ dig 71.64-26.36.168.192.in-addr.arpa. +nocomments

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> 71.64-26.36.168.192.in-addr.arpa. +nocomments
;; global options: +cmd
;71.64-26.36.168.192.in-addr.arpa. IN A
64-26.36.168.192.in-addr.arpa. 30 IN SOA silo2.lab0.solutionsqa. nobody.example.com. 180 600 1800 604800 30
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Aug 01 07:02:49 UTC 2023
;; MSG SIZE rcvd: 137

[Christian Grabowski (cgrabowski)]

So this does seem to be a MAAS issue. When a forward answer is added to BIND from MAAS dynamically (i.e a host deploys) a reverse record is generated from the same update. Currently, MAAS will generate the reverse update using the forward update's zone, this makes BIND figure out which reverse zone it should be under. In cases where we don't have the directive to generate CNAMEs for glue records, this works. However, when we have the glue record directive for a given reverse zone, the reverse update is ignored, assuming it should use answers from the smaller subnet's reverse zone.

We will need to add logic to lookup the reverse zone rather than just generating the reverse update, and add it to the reverse update instead.