Designate DNS - designate.exceptions.ZoneMasterNotFound

[Hua Zhang]

Some exception messages are being displayed in /var/log/designate/designate-mdns.log

see https://paste.openstack.org/show/bxjCk91ypKhxZNsnd8kL/

I can easily reproduce this problem by creating a PRIMARY zone in a common Xena test env with the following commands.

#first create a common openstack test env with designate
DOMAIN_NAME=extdns.example.
openstack zone create --email extdns@example $DOMAIN_NAME

and I also tested ussuri and yoga, they both have the same problem.

[Michael Johnson (johnsom)]

Hi there,

I suspect this is a configuration issue in the pools.yaml being used.

Can you run the following command:

designate-manage pool generate_file --file /tmp/check-pool.yaml

Then view that file and look for the "masters" section under the "targets" section. Validate that there are "masters" defined for each pool in the file.

See this documentation page for more information:
https://docs.openstack.org/designate/latest/admin/pools.html#managing-pools

If that doesn't help resolve the issue, can you please pastebin the check-pool.yaml file you generated?

[Hua Zhang (zhhuabj)]

Hey @johnsom, thank you very much for your reply. Here is my pools.yaml, there have been "masters" defined for the default pool.

# cat /tmp/check-pool.yaml
- also_notifies: []
  attributes: {}
  description: Pool genergated by Juju
  id: 794ccc2c-d751-44fe-b57f-8894c9f5c842
  name: default
  nameservers:
  - host: 10.5.1.99
    port: 53
  ns_records:
  - hostname: ns1.extdns.example2.
    priority: 10
  targets:
  - masters:
    - host: 10.5.0.88
      port: 5354
    options:
      host: 10.5.1.99
      port: '53'
      rndc_host: 10.5.1.99
      rndc_key_file: /etc/designate/rndc.key
    type: bind9

and it seems that from time to time, the zonemasters listed in the MySQL database designate.zone_masters will disappear when the message is displayed.

1, Sometimes I saw a record was created in the table zone_masters

mysql> select * from zone_masters;
+----------------------------------+---------+---------------------+------------+-----------+------+----------------------------------+
| id | version | created_at | updated_at | host | port | zone_id |
+----------------------------------+---------+---------------------+------------+-----------+------+----------------------------------+
| fd17132d10a0417f9e69c6b9066ecda8 | 1 | 2022-11-07 11:30:17 | NULL | 10.5.0.88 | 5354 | 90c51bbbc03e4fefa87a8e53263497b9 |
+----------------------------------+---------+---------------------+------------+-----------+------+----------------------------------+
1 row in set (0.00 sec)

$ dig +noall +answer @10.5.0.88 -p 5354 extdns.example2. axfr
extdns.example2. 3600 IN SOA ns1.extdns.example2. extdns.example. 1667822134 3529 600 86400 3600
extdns.example2. 3600 IN NS ns1.extdns.example2.
ns1.extdns.example2. 3600 IN A 10.5.1.99
extdns.example2. 3600 IN SOA ns1.extdns.example2. extdns.example. 1667822134 3529 600 86400 3600

2, But sometimes I saw this record disappeared automatically,

mysql> select * from zone_masters;
Empty set (0.00 sec)

so then we saw the exception ZoneMasterNotFound

root@juju-781502-dns-1:/home/ubuntu# grep -r ZoneMasterNotFound /var/log/designate/designate-mdns.log |tail -n1
2022-11-09 12:34:26.687 154700 ERROR oslo_messaging.rpc.server designate.exceptions.ZoneMasterNotFound: Could not find ZoneMaster

[Hua Zhang (zhhuabj)]

I did some debug, zone_master record was deleted by delete_zone_master, here is its call stack

mysql> select * from zone_masters;
+----------------------------------+---------+---------------------+---------------------+------------+------+----------------------------------+
| id | version | created_at | updated_at | host | port | zone_id |
+----------------------------------+---------+---------------------+---------------------+------------+------+----------------------------------+
| 10a355b8d22d41ae8c35d52ce8db1ae5 | 214 | 2022-11-28 08:21:44 | 2022-11-28 08:22:20 | 10.5.2.242 | 5354 | 30ebe0fdd67647d885f93eb817328c6b |
+----------------------------------+---------+---------------------+---------------------+------------+------+----------------------------------+

# nc 127.0.0.1 4444
> /usr/lib/python3/dist-packages/designate/storage/impl_sqlalchemy/__init__.py(586)delete_zone_master()
-> zone_master = self._find_zone_masters(
(Pdb) where
  /usr/lib/python3/dist-packages/eventlet/greenpool.py(88)_spawn_n_impl()
-> func(*args, **kwargs)
  /usr/lib/python3/dist-packages/futurist/_green.py(71)__call__()
-> self.work.run()
  /usr/lib/python3/dist-packages/futurist/_utils.py(52)run()
-> result = self.fn(*self.args, **self.kwargs)
  /usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py(165)_process_incoming()
-> res = self.dispatcher.dispatch(message)
  /usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py(276)dispatch()
-> return self._do_dispatch(endpoint, method, ctxt, args)
  /usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py(196)_do_dispatch()
-> result = func(ctxt, **new_args)
  /usr/lib/python3/dist-packages/designate/rpc.py(238)exception_wrapper()
-> return f(self, *args, **kwargs)
  /usr/lib/python3/dist-packages/designate/central/service.py(151)notification_wrapper()
-> result = f(self, *args, **kwargs)
  /usr/lib/python3/dist-packages/designate/central/service.py(151)notification_wrapper()
-> result = f(self, *args, **kwargs)
  /usr/lib/python3/dist-packages/designate/storage/__init__.py(67)retry_wrapper()
-> **copy.deepcopy(kwargs))
  /usr/lib/python3/dist-packages/designate/storage/__init__.py(105)transaction_wrapper()
-> result = f(self, *args, **kwargs)
  /usr/lib/python3/dist-packages/designate/central/service.py(123)sync_wrapper()
-> return f(self, *args, **kwargs)
  /usr/lib/python3/dist-packages/designate/central/service.py(2380)update_status()
-> zone = self._update_zone_status(context, zone_id, status, serial)
  /usr/lib/python3/dist-packages/designate/central/service.py(2397)_update_zone_status()
-> self.storage.update_zone(context, zone)
  /usr/lib/python3/dist-packages/designate/storage/impl_sqlalchemy/__init__.py(375)update_zone()
-> self.delete_zone_master(context, attr.id)
> /usr/lib/python3/dist-packages/designate/storage/impl_sqlalchemy/__init__.py(586)delete_zone_master()
-> zone_master = self._find_zone_masters(

mysql> select * from zone_masters;
Empty set (0.00 sec)

it seems the return value of self.storage.get_zone(context, zone_id) didn't include zone_master so zone.masters is []

(Pdb) l
373 ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/designate/+/867073

[Erik Olof Gunnar Andersson (eandersson)]

Could you provide some more logs here? The code path that is failing isn't meant to work at all for a PRIMARY and only a SECONDARY domain. The same thing with zone_masters. Are you providing any additional parameters like --master when creating the domains?

Would it be possible to try to reproduce this on the Zed release of Designate?

[Hua Zhang (zhhuabj)]

Hi Erik, we have identified that the root cause is not from designate but from the charm itself. pls see lp bug 2003528 for more details, thanks for your review

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Zhang Hua <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/designate/+/867073
Reason: this is a charm issue, not designate issue