designate-manage pool update doesn't reflects targets master dns servers into zones.

[Zachary Zehring]

[Environment]

Ubuntu + Ussuri

[Description]

If running designate-manage pool update with new targets, those targets
gets properly updated in the pool target masters list, but those aren't
reflected into the zones that belongs to this pool, therefore, the masters
associated to that zones aren't updated causing failures as the expressed
in the Further Information section.

designate-manager pool update should offer an option to update the zones
associated to the pools with the new target masters and be able to apply
these changes into existing zones.

For the case of the bind9 backend the current workaround is to manually
run the rndc modzone command with the new masters, but that's not suitable
for large installations with multiple zones and pools.

[Further information]

We have a designate/designate-bind setup. We migrated designate units to different machines, replacing 3 designate units with 3 new units. However, this caused issues with existing zones, including creating new recordsets for these zones. The zone would result in having an ERROR status and a CREATE action.

Looking at the designate bind units, we see that designate is attempting to run:

'addzone $zone { type slave; masters {$new_designate_ips port 5354;}; file "slave.$zone.$hash"; };'

This addzone fails due to the zone already existing. However, we found that the zone configuration (using 'rndc showzone $zone' from designate-bind unit) still had the old designate ips for its masters. There are also logs in /var/log/syslog like the following:

May 20 06:27:10 juju-c27f05-15-lxd-1 named[72648]: transfer of '$zone' from $old_designate_ip#5354: failed to connect: host unreachable

We were able to resolve this issue by modifying the zone config on all designate-bind units:

juju run -a designate-bind -- rndc modzone $zone '{ type slave; file "slave.$zone.$hash"; masters { $new_designate_ip_1 port 5354; $new_designate_ip_2 port 5354; $new_designate_ip_3 port 5354; }; };'

After modifying the zone, the recordset creations completed and resolved almost immediately.

Would this be something the charm could do in an automated way when masters are removed/replaced, or is there a better way of fixing the zone configurations? For these designate migrations, we will have to enumerate over every zone to fix their configurations.

[Jorge Niedbalski (niedbalski)]

### Further observations ####

I am able to partially reproduce the problem.

Bundle used: http://paste.ubuntu.com/p/myxQJnJvyn/

$ openstack zone create --email <email address hidden> example.com.
$ rndc showzone example.com.

zone "example.com" { type slave; file "slave.example.com.f3e3fdaa-857e-4786-afef-2b4cb2d03357"; masters { 10.5.0.10 port 5354; 10.5.0.41 port 5354; 10.5.0.31 port 5354; }; };

$ juju remove-unit designate/0 designate/1 designate/2 --force
removing unit designate/0
removing unit designate/1
removing unit designate/2

root@juju-54f98f-1879798-4:/home/ubuntu# ack master /var/log/syslog
May 27 19:21:20 juju-54f98f-1879798-4 named[1505]: received control channel command 'addzone example.com { type slave; masters { 10.5.0.10 port 5354; 10.5.0.41 port 5354; 10.5.0.31 port 5354;}; file "slave.example.com.f3e3fdaa-857e-4786-afef-2b4cb2d03357"; };'
May 27 19:55:32 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: timeout retrying without EDNS master 10.5.0.10#5354 (source 0.0.0.0#0)
May 27 19:55:47 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: retry limit for master 10.5.0.10#5354 exceeded (source 0.0.0.0#0)
May 27 19:56:05 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: retry limit for master 10.5.0.41#5354 exceeded (source 0.0.0.0#0)
May 27 19:56:23 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: retry limit for master 10.5.0.31#5354 exceeded (source 0.0.0.0#0)

$ juju add-unit -n 3 designate

root@juju-54f98f-1879798-4:/home/ubuntu# ack addzone /var/log/syslog
May 27 19:21:20 juju-54f98f-1879798-4 named[1505]: received control channel command 'addzone example.com { type slave; masters { 10.5.0.10 port 5354; 10.5.0.41 port 5354; 10.5.0.31 port 5354;}; file "slave.example.com.f3e3fdaa-857e-4786-afef-2b4cb2d03357"; };'
May 27 19:21:20 juju-54f98f-1879798-4 named[1505]: added zone example.com in view _default via addzone

---

(Continues).

[Jorge Niedbalski (niedbalski)]

### Notes ####

* Proposed fix https://review.opendev.org/#/c/731603/
* Updated bug description

[Jorge Niedbalski (niedbalski)]

** Proposed change https://review.opendev.org/#/c/731603/ has been merged already.

[Jorge Niedbalski (niedbalski)]

Master/Train/Ussuri/Stein fixed upstream https://review.opendev.org/q/topic:%22bug%252F1879798%22+(status:open%20OR%20status:merged)

Needs backports for UCA

[Edward Hope-Morley (hopem)]

Not currently available in an upstream point release prior to Victoria:

$ git branch -r --contains b967e9f706373f1aad6db882c2295fbbe1fadfc9
  gerrit/stable/ussuri
$ git tag --contains b967e9f706373f1aad6db882c2295fbbe1fadfc9
$

[Jorge Niedbalski (niedbalski)]

I see the change in prior releases S/T/U

https://github.com/openstack/designate/commit/b967e9f706373f1aad6db882c2295fbbe1fadfc9
https://github.com/openstack/designate/commit/953492904772933f5f8e265d1ae6cc1e6385fcc6
https://github.com/openstack/designate/commit/0b5634643b4b69cd0a7d5499f258602604741d22

Can this be backported into the cloud-archive releases?

Thanks

[Edward Hope-Morley (hopem)]

@niedbalski to start the backport sru we will need an updated sru template in the description of this bug

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "designate-bionic-stein.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Dan Streetman (ddstreet)]

unsubscribing sts-sponsors, as this appears to be a cloudy bug that i assume the o7k team will sponsor, please feel free to re-subscribe us if you would like us to review/sponsor

[Nicolas Bock (nicolasbock)]

This patch is included upstream since Victoria.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Edward Hope-Morley (hopem)]

@nicolasbock this needs to be SRUd to Focal first before bionic-ussuri

[Edward Hope-Morley (hopem)]

There are a set of stable release updates pending which will include this point release - see bug 1941048

[Corey Bryant (corey.bryant)]

1:10.0.1-0ubuntu1 is currently in focal-proposed which is being tracked by the bug linked in comment #16 above.

[Corey Bryant (corey.bryant)]

Hello Zachary, or anyone else affected,

Accepted designate into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Zachary, or anyone else affected,

Accepted designate into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Nicolas Bock (nicolasbock)]

Verified Bionic/Stein:

Before:

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.2c99109d-e825-4ebd-9976-5844987bd376"; masters { 10.5.3.106 port 5354; 10.5.3.29 port 5354; 10.5.1.105 port 5354; }; };

juju add-unit designate # 10.5.0.89

zone is not updated on designate-bind unit to include the new zone master.

After:

juju remove-unit designate/0 # 10.5.3.106

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.2c99109d-e825-4ebd-9976-5844987bd376"; masters { 10.5.1.105 port 5354; 10.5.0.89 port 5354; 10.5.3.29 port 5354; }; };

[Nicolas Bock (nicolasbock)]

Verified Bionic/Train:

Before:

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.887b48bd-b5ec-4b07-9678-4fc4fe843686"; masters { 10.5.3.211 port 5354; 10.5.1.86 port 5354; 10.5.3.35 port 5354; }; };

juju add-unit designate # 10.5.2.227

zone is not updated on designate-bind unit to include the new zone master.

After:

juju remove-unit designate/0 # 10.5.3.35

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.887b48bd-b5ec-4b07-9678-4fc4fe843686"; masters { 10.5.2.227 port 5354; 10.5.1.86 port 5354; 10.5.3.211 port 5354; }; };

[Corey Bryant (corey.bryant)]

Fix released in stable point release

[Chris MacNaughton (chris.macnaughton)]

Released in stable point update

[Chris MacNaughton (chris.macnaughton)]

The verification of the Stable Release Update for designate has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Chris MacNaughton (chris.macnaughton)]

This bug was fixed in the package designate - 1:9.0.2-0ubuntu1~cloud2
---------------

 designate (1:9.0.2-0ubuntu1~cloud2) bionic-train; urgency=medium
 .
   [ Nicolas Bock ]
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters (LP: #1879798).
 .
 designate (1:9.0.2-0ubuntu1~cloud1) bionic-train; urgency=medium
 .
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters.

[Chris MacNaughton (chris.macnaughton)]

The verification of the Stable Release Update for designate has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Chris MacNaughton (chris.macnaughton)]

This bug was fixed in the package designate - 1:8.0.1-0ubuntu1~cloud3
---------------

 designate (1:8.0.1-0ubuntu1~cloud3) bionic-stein; urgency=medium
 .
   [ Nicolas Bock ]
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters (LP: #1879798).
 .
 designate (1:8.0.1-0ubuntu1~cloud2) bionic-stein; urgency=medium
 .
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters.

[Hua Zhang (zhhuabj)]

Hi team, has this bug really been resolved now? It seems that the change is already in focal and jammy, but I did two tests on focal and jammy, none of them were sucessful.

focal - https://paste.ubuntu.com/p/txgj9tGtB4/
jammy - https://paste.ubuntu.com/p/2MZwH6wG2s/

Is it because I need to run any more commands to trigger zone updates? Even so, shouldn't it be done together here?

[Hua Zhang (zhhuabj)]

designate/3 was created at 2023-07-13 09:33:51,

$ juju ssh designate/3 -- sudo grep -r 'designate\/3' /var/log/juju/ |grep started
/var/log/juju/unit-designate-3.log:2023-07-13 09:33:51 INFO juju.worker.uniter uniter.go:326 unit "designate/3" started

modzone was triggered at 2023-07-13 09:55:04.419

ubuntu@juju-5abc40-jammy-designate-1:~$ sudo grep -r 'modzone' /var/log/designate/designate-* |tail -n1
/var/log/designate/designate-worker.log:2023-07-13 09:55:04.419 53020 DEBUG oslo_concurrency.processutils [req-3a99b88c-7e95-4a44-9132-e1383cb7e375 - - - - -] CMD "sudo designate-rootwrap /etc/designate/rootwrap.conf rndc -s 10.5.2.44 -p 953 -k /etc/designate/rndc.key modzone extdns.example { type slave; masters { 10.5.3.135 port 5354; 10.5.3.71 port 5354;}; file "slave.extdns.example.aba56bfb-50c8-4e5c-8447-01666d2012a7"; };" returned: 0 in 6.484s execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:422

So it seems target.get("masters", []) in the code [1] doesn't have an updated master list.

[1] https://review.opendev.org/c/openstack/designate/+/731603/8/designate/manage/pool.py#53

[Hua Zhang (zhhuabj)]

After running 'sudo designate-manage pool update', masters is now correct. So this has already resolved the problem. Pls ignore the two noises I replied to above