designate-manage pool update doesn't reflects targets master dns servers into zones.

Bug #1879798 reported by Zachary Zehring
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Designate
Undecided
Jorge Niedbalski
Ubuntu Cloud Archive
Undecided
Unassigned
Stein
Undecided
Unassigned
Train
Undecided
Unassigned
Ussuri
Undecided
Unassigned
Victoria
Undecided
Unassigned
Wallaby
Undecided
Unassigned
Xena
Undecided
Unassigned
designate (Ubuntu)
Medium
Jorge Niedbalski
Focal
Medium
Unassigned

Bug Description

[Environment]

Ubuntu + Ussuri

[Description]

If running designate-manage pool update with new targets, those targets
gets properly updated in the pool target masters list, but those aren't
reflected into the zones that belongs to this pool, therefore, the masters
associated to that zones aren't updated causing failures as the expressed
in the Further Information section.

designate-manager pool update should offer an option to update the zones
associated to the pools with the new target masters and be able to apply
these changes into existing zones.

For the case of the bind9 backend the current workaround is to manually
run the rndc modzone command with the new masters, but that's not suitable
for large installations with multiple zones and pools.

[Further information]

We have a designate/designate-bind setup. We migrated designate units to different machines, replacing 3 designate units with 3 new units. However, this caused issues with existing zones, including creating new recordsets for these zones. The zone would result in having an ERROR status and a CREATE action.

Looking at the designate bind units, we see that designate is attempting to run:

'addzone $zone { type slave; masters {$new_designate_ips port 5354;}; file "slave.$zone.$hash"; };'

This addzone fails due to the zone already existing. However, we found that the zone configuration (using 'rndc showzone $zone' from designate-bind unit) still had the old designate ips for its masters. There are also logs in /var/log/syslog like the following:

May 20 06:27:10 juju-c27f05-15-lxd-1 named[72648]: transfer of '$zone' from $old_designate_ip#5354: failed to connect: host unreachable

We were able to resolve this issue by modifying the zone config on all designate-bind units:

juju run -a designate-bind -- rndc modzone $zone '{ type slave; file "slave.$zone.$hash"; masters { $new_designate_ip_1 port 5354; $new_designate_ip_2 port 5354; $new_designate_ip_3 port 5354; }; };'

After modifying the zone, the recordset creations completed and resolved almost immediately.

Would this be something the charm could do in an automated way when masters are removed/replaced, or is there a better way of fixing the zone configurations? For these designate migrations, we will have to enumerate over every zone to fix their configurations.

tags: added: scaleback
Changed in charm-designate-bind:
status: New → Confirmed
Changed in charm-designate:
status: New → Confirmed
Changed in charm-designate:
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in charm-designate-bind:
assignee: nobody → Jorge Niedbalski (niedbalski)
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

### Further observations ####

I am able to partially reproduce the problem.

Bundle used: http://paste.ubuntu.com/p/myxQJnJvyn/

$ openstack zone create --email <email address hidden> example.com.
$ rndc showzone example.com.

zone "example.com" { type slave; file "slave.example.com.f3e3fdaa-857e-4786-afef-2b4cb2d03357"; masters { 10.5.0.10 port 5354; 10.5.0.41 port 5354; 10.5.0.31 port 5354; }; };

$ juju remove-unit designate/0 designate/1 designate/2 --force
removing unit designate/0
removing unit designate/1
removing unit designate/2

root@juju-54f98f-1879798-4:/home/ubuntu# ack master /var/log/syslog
May 27 19:21:20 juju-54f98f-1879798-4 named[1505]: received control channel command 'addzone example.com { type slave; masters { 10.5.0.10 port 5354; 10.5.0.41 port 5354; 10.5.0.31 port 5354;}; file "slave.example.com.f3e3fdaa-857e-4786-afef-2b4cb2d03357"; };'
May 27 19:55:32 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: timeout retrying without EDNS master 10.5.0.10#5354 (source 0.0.0.0#0)
May 27 19:55:47 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: retry limit for master 10.5.0.10#5354 exceeded (source 0.0.0.0#0)
May 27 19:56:05 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: retry limit for master 10.5.0.41#5354 exceeded (source 0.0.0.0#0)
May 27 19:56:23 juju-54f98f-1879798-4 named[6653]: zone example.com/IN: refresh: retry limit for master 10.5.0.31#5354 exceeded (source 0.0.0.0#0)

$ juju add-unit -n 3 designate

root@juju-54f98f-1879798-4:/home/ubuntu# ack addzone /var/log/syslog
May 27 19:21:20 juju-54f98f-1879798-4 named[1505]: received control channel command 'addzone example.com { type slave; masters { 10.5.0.10 port 5354; 10.5.0.41 port 5354; 10.5.0.31 port 5354;}; file "slave.example.com.f3e3fdaa-857e-4786-afef-2b4cb2d03357"; };'
May 27 19:21:20 juju-54f98f-1879798-4 named[1505]: added zone example.com in view _default via addzone

---

(Continues).

Changed in charm-designate-bind:
status: Confirmed → Invalid
summary: - replacing designate units causes issues previously created zones
+ designate-manage pool update doesn't reflects targets master dns servers
+ into zones.
description: updated
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

### Notes ####

* Proposed fix https://review.opendev.org/#/c/731603/
* Updated bug description

Changed in designate (Ubuntu):
status: New → In Progress
assignee: nobody → Jorge Niedbalski (niedbalski)
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

** Proposed change https://review.opendev.org/#/c/731603/ has been merged already.

Changed in designate (Ubuntu):
status: In Progress → Fix Committed
Changed in charm-designate:
status: Confirmed → Invalid
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Master/Train/Ussuri/Stein fixed upstream https://review.opendev.org/q/topic:%22bug%252F1879798%22+(status:open%20OR%20status:merged)

Needs backports for UCA

Revision history for this message
Edward Hope-Morley (hopem) wrote :

Not currently available in an upstream point release prior to Victoria:

$ git branch -r --contains b967e9f706373f1aad6db882c2295fbbe1fadfc9
  gerrit/stable/ussuri
$ git tag --contains b967e9f706373f1aad6db882c2295fbbe1fadfc9
$

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :
Revision history for this message
Edward Hope-Morley (hopem) wrote :

@niedbalski to start the backport sru we will need an updated sru template in the description of this bug

Revision history for this message
Nicolas Bock (nicolasbock) wrote :
Revision history for this message
Nicolas Bock (nicolasbock) wrote :
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "designate-bionic-stein.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Mathew Hodson (mhodson)
Changed in designate (Ubuntu):
importance: Undecided → Medium
Changed in designate (Ubuntu Focal):
importance: Undecided → Medium
Mathew Hodson (mhodson)
affects: charm-designate → designate
Changed in designate:
status: Invalid → Fix Released
affects: charm-designate-bind → ubuntu-translations
no longer affects: ubuntu-translations
Revision history for this message
Dan Streetman (ddstreet) wrote :

unsubscribing sts-sponsors, as this appears to be a cloudy bug that i assume the o7k team will sponsor, please feel free to re-subscribe us if you would like us to review/sponsor

Revision history for this message
Nicolas Bock (nicolasbock) wrote :
Revision history for this message
Nicolas Bock (nicolasbock) wrote :

This patch is included upstream since Victoria.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in designate (Ubuntu Focal):
status: New → Confirmed
Revision history for this message
Edward Hope-Morley (hopem) wrote :

@nicolasbock this needs to be SRUd to Focal first before bionic-ussuri

Revision history for this message
Edward Hope-Morley (hopem) wrote :

There are a set of stable release updates pending which will include this point release - see bug 1941048

Revision history for this message
Corey Bryant (corey.bryant) wrote :

1:10.0.1-0ubuntu1 is currently in focal-proposed which is being tracked by the bug linked in comment #16 above.

Changed in designate (Ubuntu Focal):
status: Confirmed → Fix Committed
Changed in designate (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Corey Bryant (corey.bryant) wrote : Please test proposed package

Hello Zachary, or anyone else affected,

Accepted designate into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Zachary, or anyone else affected,

Accepted designate into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
Revision history for this message
Nicolas Bock (nicolasbock) wrote :

Verified Bionic/Stein:

Before:

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.2c99109d-e825-4ebd-9976-5844987bd376"; masters { 10.5.3.106 port 5354; 10.5.3.29 port 5354; 10.5.1.105 port 5354; }; };

juju add-unit designate # 10.5.0.89

zone is not updated on designate-bind unit to include the new zone master.

After:

juju remove-unit designate/0 # 10.5.3.106

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.2c99109d-e825-4ebd-9976-5844987bd376"; masters { 10.5.1.105 port 5354; 10.5.0.89 port 5354; 10.5.3.29 port 5354; }; };

tags: added: verification-stein-done verification-train-needed
removed: verification-stein-needed
Revision history for this message
Nicolas Bock (nicolasbock) wrote :

Verified Bionic/Train:

Before:

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.887b48bd-b5ec-4b07-9678-4fc4fe843686"; masters { 10.5.3.211 port 5354; 10.5.1.86 port 5354; 10.5.3.35 port 5354; }; };

juju add-unit designate # 10.5.2.227

zone is not updated on designate-bind unit to include the new zone master.

After:

juju remove-unit designate/0 # 10.5.3.35

$ juju ssh designate-bind/0 -- sudo rndc showzone example.com.
zone "example.com" { type slave; file "slave.example.com.887b48bd-b5ec-4b07-9678-4fc4fe843686"; masters { 10.5.2.227 port 5354; 10.5.1.86 port 5354; 10.5.3.211 port 5354; }; };

tags: added: verification-train-done
removed: verification-train-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Fix released in stable point release

Changed in designate (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

Released in stable point update

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote : Update Released

The verification of the Stable Release Update for designate has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

This bug was fixed in the package designate - 1:9.0.2-0ubuntu1~cloud2
---------------

 designate (1:9.0.2-0ubuntu1~cloud2) bionic-train; urgency=medium
 .
   [ Nicolas Bock ]
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters (LP: #1879798).
 .
 designate (1:9.0.2-0ubuntu1~cloud1) bionic-train; urgency=medium
 .
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

The verification of the Stable Release Update for designate has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

This bug was fixed in the package designate - 1:8.0.1-0ubuntu1~cloud3
---------------

 designate (1:8.0.1-0ubuntu1~cloud3) bionic-stein; urgency=medium
 .
   [ Nicolas Bock ]
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters (LP: #1879798).
 .
 designate (1:8.0.1-0ubuntu1~cloud2) bionic-stein; urgency=medium
 .
   * d/p/0001-Update-zones-masters-using-pool-target-masters.patch: Update zones
     masters using pool target masters.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers