Multiple pools with PDNS4 backend

[Arnoud de Jonge]

I'm trying to add multiple pools with PDNS4 using tsig keys, using https://docs.openstack.org/designate/latest/admin/backends/pdns4.html. But all the zones I create in these pools got to ERROR.

My pools.yaml.

- name: netnod-cysogroup-fuga-ams2-europe
  description: NetNod Standard Europe A
  ns_records:
    - hostname: unicast-eu.dns.cyso.nl.
      priority: 1
    - hostname: anycast-eu.dns.cyso.eu.
      priority: 2
  nameservers:
    - host: 10.2.0.50
      port: 53
  targets:
    - type: pdns4
      description: PowerDNS4 10.2.0.50
      masters:
        - host: 10.2.0.20
          port: 5354
        - host: 10.2.0.21
          port: 5354
        - host: 10.2.0.22
          port: 5354
      options:
        host: 10.2.0.50
        port: 53
        api_endpoint: http://10.2.0.50:8081
        api_token: SECRET_TOKEN
        tsigkey_name: netnod-cysogroup-fuga-ams2-europe
- name: netnod-cysogroup-fuga-ams2-global
  description: NetNod Premium C
  ns_records:
    - hostname: unicast-eu.dns.cyso.nl.
      priority: 1
    - hostname: anycast-eu.dns.cyso.eu.
      priority: 2
    - hostname: anycast-global.dns.cyso.net.
      priority: 3
  nameservers:
    - host: 10.2.0.50
      port: 53
  targets:
    - type: pdns4
      description: PowerDNS4 10.2.0.50
      masters:
        - host: 10.2.0.20
          port: 5354
        - host: 10.2.0.21
          port: 5354
        - host: 10.2.0.22
          port: 5354
      options:
        host: 10.2.0.50
        port: 53
        api_endpoint: http://10.2.0.50:8081
        api_token: SECRET_TOKEN
        tsigkey_name: netnod-cysogroup-fuga-ams2-global

TSIG keys are added to the pools:

openstack tsigkey create --name netnod-cysogroup-fuga-ams2-europe --algorithm hmac-sha512 --secret $TSIG --scope POOL --resource-id 2d688d3a-4082-4042-b158-88bafc75298f
openstack tsigkey create --name netnod-cysogroup-fuga-ams2-global --algorithm hmac-sha512 --secret $TSIG --scope POOL --resource-id 2a338368-1820-4863-aba8-c584bcf373bd

pdnsutil list-tsig-keys
May 28 14:22:33 [bindbackend] Done parsing domains, 0 rejected, 0 new, 0 removed
netnod-cysogroup-fuga-ams2-europe. hmac-sha512. blablalbla1
netnod-cysogroup-fuga-ams2-global. hmac-sha512. blablalbla2

When I try to add a zone to a pool, it lands in the correct pool but goes to ERROR after a while. If I list the zone in PDNS it does not show the tsig keys as I would expect.

openstack zone create eu.nl. --email <email address hidden> --attribute pool_id:2d688d3a-4082-4042-b158-88bafc75298f
(venv) ubuntu@dev-provisioning:~/fuga-bootstrap$ openstack zone show eu.nl.
+----------------+----------------------------------------------+
| Field | Value |
+----------------+----------------------------------------------+
| action | CREATE |
| attributes | pool_id:2d688d3a-4082-4042-b158-88bafc75298f |
| | |
| created_at | 2021-05-28T13:41:56.000000 |
| description | None |
| email | <email address hidden> |
| id | 7afd78fc-4e64-4cba-bdcf-860c430ad647 |
| masters | |
| name | eu.nl. |
| pool_id | 2d688d3a-4082-4042-b158-88bafc75298f |
| project_id | 6b5c667caaaf4d81bd92b69f331a91a9 |
| serial | 1622209316 |
| status | ERROR |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | 2021-05-28T13:44:31.000000 |
| version | 2 |
+----------------+----------------------------------------------+

root@powerdns:~# pdnsutil show-zone eu.nl.
May 28 14:26:31 [bindbackend] Done parsing domains, 0 rejected, 0 new, 0 removed
This is a Slave zone
Masters: 10.2.0.20:5354 10.2.0.22:5354 10.2.0.21:5354
Last time we got update from master: Never
No SOA serial found in database
Zone is not actively secured
Metadata items:
 SOA-EDIT-API DEFAULT
No keys for zone 'eu.nl'.

The slave tsig key is not set when the zone is created in PowerDNS. These are the headers I captured with tcpdump:

{"account": "", "api_rectify": false, "dnssec": false, "id": "eu.nl.", "kind": "Slave", "last_check": 0, "masters": ["10.2.0.20:5354", "10.2.0.22:5354", "10.2.0.21:5354"], "name": "eu.nl.", "notified_serial": 0, "nsec3narrow": false, "nsec3param": "", "rrsets": [], "serial": 0, "soa_edit": "", "soa_edit_api": "DEFAULT", "url": "/api/v1/servers/localhost/zones/eu.nl."}

PDNS logs:
May 28 14:23:12 powerdns pdns_server[21389]: message repeated 6 times: [ No new unfresh slave domains, 0 queued for AXFR already, 0 in progress]
May 28 14:24:12 powerdns pdns_server[21389]: 1 slave domain needs checking, 0 queued for AXFR
May 28 14:24:12 powerdns pdns_server[21389]: While checking domain freshness: Query to '10.2.0.21' for SOA of 'eu.nl.' did not return a SOA
May 28 14:24:15 powerdns pdns_server[21389]: Received serial number updates for 0 zones, had 1 timeout
May 28 14:25:15 powerdns pdns_server[21389]: No new unfresh slave domains, 0 queued for AXFR already, 0 in progress

[Arnoud de Jonge (arnoud-dejonge-4)]

Hi there, it's me again.

Got as far as getting the TSIG key in PDNS.

Pool:

- name: fuga-global
  description: Fuga Global
  attributes: {}
  ns_records:
  - hostname: anycast-global.dns.fuga.cloud.
    priority: 3
  - hostname: anycast-eu.dns.fuga.cloud.
    priority: 2
  - hostname: unicast-eu.dns.fuga.cloud.
    priority: 1
  nameservers:
    - host: 10.1.0.10
      port: 53
  targets:
      masters:
        - host: 10.1.0.20
          port: 5354
        - host: 10.1.0.21
          port: 5354
        - host: 10.1.0.22
          port: 5354
      options:
        host: 10.1.0.10
        port: 53
        api_endpoint: http://10.1.0.10:8081
        api_token: zoekikop
        tsigkey_name: netnod-fuga-global.

Creating the domain:

openstack zone create --email <email address hidden> sparnebrau.nl. --attributes pool_id:247a3cd1-937a-4321-a5a7-f0ea2cd6d6ee

Zone is created with the correct TSIG.

root@pdns:~# pdnsutil show-zone sparnebrau.nl
May 18 14:30:05 [bindbackend] Done parsing domains, 0 rejected, 0 new, 0 removed
This is a Slave zone
Masters: 10.2.0.21:5354 10.2.0.22:5354 10.2.0.20:5354
Last time we got update from master: Never
No SOA serial found in database
Zone is not actively secured
Zone uses following TSIG key(s): netnod-fuga-global.
Metadata items:
 AXFR-MASTER-TSIG netnod-fuga-global.
 SOA-EDIT-API DEFAULT
No keys for zone 'sparnebrau.nl'.

So far so good. But the zone still goes to error after a short while.

2022-05-18 14:04:00.037 21 DEBUG designate.service [req-ea5cff22-4cf7-4de3-bb6e-66277a076c96 - - - - -] Handling UDP Request from: 10.2.0.11:16511 _dns_handle_udp /var/lib/kolla/venv/lib/python3.6/site-packages/designate/service.py:318
2022-05-18 14:04:00.049 21 ERROR designate.dnsutils [req-9eda91df-70c2-4761-b3ac-6e75524d4170 - - - - -] Unknown TSIG key from 10.2.0.11:16511: dns.message.UnknownTSIGKey: key 'netnod-fuga-global.' unknown

The key is defined in openstack as well as in PDNS and it is assigned to the correct pool. So why can't it find it?

[Arnoud de Jonge (arnoud-dejonge-4)]

In case this might be helpful to someone:

Found the cause. Our TSIG key name has a dot on the end. For some reason this dot is stripped from the name before a query in the database is done, so the key can not be found. I recreated the TSIG keys in openstack without the leading dot. This fixed it!

But a leading dot in a TSIG key is valid, and it should not have been stripped in the first place.