Pool update fails when zones exist

Bug #1897936 reported by Mark Goddard
34
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Designate
Fix Released
Undecided
Mark Goddard

Bug Description

Seen on kolla-ansible deployed designate, current master (just prior to Victoria release).

# Steps to reproduce

* deploy designate
* update pools from pools.yaml
* create a zone
* update pools from same pools.yaml

# Expected result

Second pool update is a noop

# Actual result

Second pool update fails.

designate.exceptions.DuplicateZoneMaster: Duplicate ZoneMaster
2020-09-30 14:39:02.738 70 ERROR designate Traceback (most recent call last):
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/bin/designate-manage", line 10, in <module>
2020-09-30 14:39:02.738 70 ERROR designate sys.exit(main())
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/designate/cmd/manage.py", line 123, in main
2020-09-30 14:39:02.738 70 ERROR designate fn(*fn_args)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/designate/manage/pool.py", line 170, in update
2020-09-30 14:39:02.738 70 ERROR designate self._update_zones(pool)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/designate/manage/pool.py", line 69, in _update_zones
2020-09-30 14:39:02.738 70 ERROR designate zone)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/designate/central/rpcapi.py", line 166, in update_zone
2020-09-30 14:39:02.738 70 ERROR designate increment_serial=increment_serial)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 509, in call
2020-09-30 14:39:02.738 70 ERROR designate return self.prepare().call(ctxt, method, **kwargs)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-09-30 14:39:02.738 70 ERROR designate transport_options=self.transport_options)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
2020-09-30 14:39:02.738 70 ERROR designate transport_options=transport_options)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send
2020-09-30 14:39:02.738 70 ERROR designate transport_options=transport_options)
2020-09-30 14:39:02.738 70 ERROR designate File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send
2020-09-30 14:39:02.738 70 ERROR designate raise result
2020-09-30 14:39:02.738 70 ERROR designate designate.exceptions_Remote.DuplicateZoneMaster_Remote: Duplicate ZoneMaster

2020-09-30 14:39:02.738 70 ERROR designate oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '192.168.39.5-5354-a551e4f00bcf4ae786d0d380b84c89f1' for key 'unique_masters'")
2020-09-30 14:39:02.738 70 ERROR designate [SQL: INSERT INTO zone_masters (id, version, created_at, host, port, zone_id) VALUES (%(id)s, %(version)s, %(created_at)s, %(host)s, %(port)s, %(zone_id)s)]
2020-09-30 14:39:02.738 70 ERROR designate [parameters: {'id': '80675457ccd6447bb7568005535eff7f', 'version': 1, 'created_at': datetime.datetime(2020, 9, 30, 14, 39, 2, 714004), 'host': '192.168.39.5', 'port': 5354, 'zone_id': 'a551e4f00bcf4ae786d0d380b84c89f1'}]

Here is the pools.yaml I used:

- name: default-bind
  id: a95091de-f9d4-4d1b-8ee3-cdfe380b94ee
  description: Default BIND9 Pool
  attributes: {}
  ns_records:
    - hostname: sample.openstack.org.
      priority: 1
  nameservers:
    - host: 192.168.37.3
      port: 53
    - host: 192.168.37.4
      port: 53
    - host: 192.168.37.5
      port: 53
  targets:
    - type: bind9
      description: BIND9 Server 192.168.37.3
      masters:
        - host: 192.168.39.3
          port: 5354
        - host: 192.168.39.4
          port: 5354
        - host: 192.168.39.5
          port: 5354
      options:
        host: 192.168.37.3
        port: 53
        rndc_host: 192.168.37.3
        rndc_port: 953
        rndc_key_file: /etc/designate/rndc.key
    - type: bind9
      description: BIND9 Server 192.168.37.4
      masters:
        - host: 192.168.39.3
          port: 5354
        - host: 192.168.39.4
          port: 5354
        - host: 192.168.39.5
          port: 5354
      options:
        host: 192.168.37.4
        port: 53
        rndc_host: 192.168.37.4
        rndc_port: 953
        rndc_key_file: /etc/designate/rndc.key
    - type: bind9
      description: BIND9 Server 192.168.37.5
      masters:
        - host: 192.168.39.3
          port: 5354
        - host: 192.168.39.4
          port: 5354
        - host: 192.168.39.5
          port: 5354
      options:
        host: 192.168.37.5
        port: 53
        rndc_host: 192.168.37.5
        rndc_port: 953
        rndc_key_file: /etc/designate/rndc.key

Revision history for this message
Mark Goddard (mgoddard) wrote :

From RDO

designate==10.1.0.dev63
python-designateclient==4.1.0

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to designate (master)

Fix proposed to branch: master
Review: https://review.opendev.org/755349

Changed in designate:
assignee: nobody → Mark Goddard (mgoddard)
status: New → In Progress
Revision history for this message
Mark Goddard (mgoddard) wrote :

Seems to affect only multi-node environments, where the pool target masters are duplicated.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (master)

Reviewed: https://review.opendev.org/755349
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=4e86bc2a58f00371a2f22217cb8a79e5ac33677c
Submitter: Zuul
Branch: master

commit 4e86bc2a58f00371a2f22217cb8a79e5ac33677c
Author: Mark Goddard <email address hidden>
Date: Wed Sep 30 17:48:25 2020 +0100

    Fix pool update with duplicate masters

    If any zones exist that belong to a pool that has any duplicate masters
    among all targets, currently updating the pool will fail.

    This change fixes the issue by ensuring the pool target master list does
    not contain duplicates.

    Change-Id: I72bae79b4f8e0bab739875d6f783abdb4868950b
    Closes-Bug: #1897936

Changed in designate:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to designate (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/755400

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on designate (stable/victoria)

Change abandoned by Nicolas Bock (<email address hidden>) on branch: stable/victoria
Review: https://review.opendev.org/755400
Reason: I'll abandon this backport for now until we have resolved this discussion on master.

Revision history for this message
Andrew Bonney (andrewbonney) wrote :

Hi. Is there any update to this? We've just encountered it having upgraded designate to victoria.

Revision history for this message
Mark Goddard (mgoddard) wrote :

I've asked for the abandoned victoria backport to be restored: https://review.opendev.org/c/openstack/designate/+/755400

Revision history for this message
Ole Kleinschmidt (oklhost) wrote :

Same here while upgrading ussuri -> victoria.

Revision history for this message
Cagri Ersen (cagri-ersen) wrote :

Hi there,

Any progress for victoria on this?

Revision history for this message
Marc Schmitt (risson) wrote :

This bug has now also been introduced in Ussuri.
https://review.opendev.org/c/openstack/designate/+/743610

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/designate 12.0.0.0rc1

This issue was fixed in the openstack/designate 12.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/designate/+/755400
Committed: https://opendev.org/openstack/designate/commit/9e159d93d99bf1ed0c487bfe99450f59db509249
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 9e159d93d99bf1ed0c487bfe99450f59db509249
Author: Mark Goddard <email address hidden>
Date: Wed Sep 30 17:48:25 2020 +0100

    Fix pool update with duplicate masters

    If any zones exist that belong to a pool that has any duplicate masters
    among all targets, currently updating the pool will fail.

    This change fixes the issue by ensuring the pool target master list does
    not contain duplicates.

    Change-Id: I72bae79b4f8e0bab739875d6f783abdb4868950b
    Closes-Bug: #1897936
    (cherry picked from commit 4e86bc2a58f00371a2f22217cb8a79e5ac33677c)

tags: added: in-stable-victoria
Revision history for this message
Marcus Klein (marcus-klein) wrote :

Will the fix be backported to Ussuri? Current stable/ussuri does not allow any pool update currently with multiple masters.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to designate (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/designate/+/805143

Revision history for this message
Michal Arbet (michalarbet) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to designate (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/designate/+/815647

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/designate/+/805143
Committed: https://opendev.org/openstack/designate/commit/a9ab61bc1d98248af108d895ce2cb7696818dcda
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a9ab61bc1d98248af108d895ce2cb7696818dcda
Author: Mark Goddard <email address hidden>
Date: Wed Sep 30 17:48:25 2020 +0100

    Fix pool update with duplicate masters

    If any zones exist that belong to a pool that has any duplicate masters
    among all targets, currently updating the pool will fail.

    This change fixes the issue by ensuring the pool target master list does
    not contain duplicates.

    Change-Id: I72bae79b4f8e0bab739875d6f783abdb4868950b
    Closes-Bug: #1897936
    (cherry picked from commit 4e86bc2a58f00371a2f22217cb8a79e5ac33677c)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (stable/train)

Reviewed: https://review.opendev.org/c/openstack/designate/+/815647
Committed: https://opendev.org/openstack/designate/commit/58e441e4b06bb607ce24b893fb0f80ff21742446
Submitter: "Zuul (22348)"
Branch: stable/train

commit 58e441e4b06bb607ce24b893fb0f80ff21742446
Author: Mark Goddard <email address hidden>
Date: Wed Sep 30 17:48:25 2020 +0100

    Fix pool update with duplicate masters

    If any zones exist that belong to a pool that has any duplicate masters
    among all targets, currently updating the pool will fail.

    This change fixes the issue by ensuring the pool target master list does
    not contain duplicates.

    Change-Id: I72bae79b4f8e0bab739875d6f783abdb4868950b
    Closes-Bug: #1897936
    (cherry picked from commit 4e86bc2a58f00371a2f22217cb8a79e5ac33677c)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/designate 10.0.2

This issue was fixed in the openstack/designate 10.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/designate 11.0.1

This issue was fixed in the openstack/designate 11.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/designate train-eol

This issue was fixed in the openstack/designate train-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.