Failed to update zone

[David Ames]

Bionic Rocky mojo run zone is in error state:

$ openstack zone show 0fb6a359-7ca7-4f7c-bd3a-c9662f2b7f62
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| action | UPDATE |
| attributes | |
| created_at | 2018-11-06T18:00:26.000000 |
| description | None |
| email | <email address hidden> |
| id | 0fb6a359-7ca7-4f7c-bd3a-c9662f2b7f62 |
| masters | |
| name | mojo.serverstack. |
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | dfe11657ed08478ba631a7c1709b9626 |
| serial | 1541527495 |
| status | ERROR |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | 2018-11-06T18:16:09.000000 |
| version | 12 |
+----------------+--------------------------------------+

Designate worker log:

2018-11-06 18:20:36.686 11932 INFO designate.worker.tasks.zone [req-da12e7d1-0227-4945-9b3d-669890600b80 - - - - -] Could not UPDATE 150.5.10.in-addr.arpa. on enough targets. Updating status to ERROR
2018-11-06 18:20:38.512 11932 INFO designate.worker.tasks.zone [req-da12e7d1-0227-4945-9b3d-669890600b80 - - - - -] Failed to UPDATE zone mojo.serverstack. on target 05c6311d-0f21-4793-8635-84f7e58a0270 on attempt 5, Error: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.

[David Ames (thedac)]

2018-11-06 18:00:27.026 11932 INFO designate.worker.service [req-7f1e68dc-7b18-4b30-96e4-f0c26bb2f2a1 - - - - -] 3 targets setup
2018-11-06 18:00:27.057 11932 INFO designate.worker.tasks.zone [req-7f1e68dc-7b18-4b30-96e4-f0c26bb2f2a1 - - - - -] Attempting CREATE on zone mojo.serverstack.
2018-11-06 18:00:28.251 11932 INFO designate.worker.tasks.zone [req-703c8040-478f-485b-a34c-5e743275c1a5 - - - - -] Failed to CREATE zone mojo.serverstack. on target 05c6311d-0f21-4793-8635-84f7e58a0270 on attempt 1, Error: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.
2018-11-06 18:00:28.263 11932 INFO designate.worker.tasks.zone [req-703c8040-478f-485b-a34c-5e743275c1a5 - - - - -] Failed to CREATE zone mojo.serverstack. on target 9d4bfb47-4dbe-4b16-92c8-ee1e86917e36 on attempt 1, Error: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.
2018-11-06 18:00:28.272 11932 INFO designate.worker.tasks.zone [req-703c8040-478f-485b-a34c-5e743275c1a5 - - - - -] Failed to CREATE zone mojo.serverstack. on target a7be4b60-10c1-44ac-b327-04013838dc8e on attempt 1, Error: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.
2018-11-06 18:00:43.634 11932 INFO designate.backend.impl_bind9 [req-55111ff8-4ea4-4080-abc8-69974d2db500 - - - - -] RNDC call failure: Unexpected error while running command.
Command: sudo designate-rootwrap /etc/designate/rootwrap.conf rndc -s 10.5.0.12 -p 953 -k /etc/designate/rndc.key addzone mojo.serverstack { type slave; masters { 10.5.0.19 port 5354; 10.5.0.6 port 5354; 10.5.0.23 port 5354;}; file "slave.mojo.serverstack.0fb6a359-7ca7-4f7c-bd3a-c9662f2b7f62"; };
Exit code: 1
Stdout: ''
Stderr: "rndc: 'addzone' failed: already exists\n"
2018-11-06 18:00:43.639 11932 INFO designate.worker.tasks.zone [req-55111ff8-4ea4-4080-abc8-69974d2db500 - - - - -] Failed to CREATE zone mojo.serverstack. on target 9d4bfb47-4dbe-4b16-92c8-ee1e86917e36 on attempt 2, Error: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.
2018-11-06 18:00:43.685 11932 INFO designate.backend.impl_bind9 [req-55111ff8-4ea4-4080-abc8-69974d2db500 - - - - -] RNDC call failure: Unexpected error while running command.
Command: sudo designate-rootwrap /etc/designate/rootwrap.conf rndc -s 10.5.0.13 -p 953 -k /etc/designate/rndc.key addzone mojo.serverstack { type slave; masters { 10.5.0.6 port 5354; 10.5.0.19 port 5354; 10.5.0.23 port 5354;}; file "slave.mojo.serverstack.0fb6a359-7ca7-4f7c-bd3a-c9662f2b7f62"; };
Exit code: 1
Stdout: ''
Stderr: "rndc: 'addzone' failed: already exists\n"
2018-11-06 18:00:43.685 11932 INFO designate.worker.tasks.zone [req-55111ff8-4ea4-4080-abc8-69974d2db500 - - - - -] Failed to CREATE zone mojo.serverstack. on target 05c6311d-0f21-4793-8635-84f7e58a0270 on attempt 2, Error: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.
2018-11-06 18:00:43.757 11932 INFO designate.backend.impl_bind9 [req-55111ff8-4ea4-4080-abc8-69974d2db500 - - - - -] RNDC call failure: Unexpected error while running command.
Command: sudo designate-rootwrap /etc/designate/rootwrap.conf rndc -s 10.5.0.20 -p 953 -k /etc/designate/...

[David Ames (thedac)]

Target option port is None:

2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server [req-9582f628-7405-49fe-a24b-6f0422d2041f - - - - -] Exception during message handling: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/designate/rpc.py", line 167, in dispatch
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server return super(RPCDispatcher, self).dispatch(*args, **kwds)
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/designate/worker/service.py", line 141, in update_zone
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server self._do_zone_action(context, zone)
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/designate/worker/service.py", line 125, in _do_zone_action
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server return self.executor.run(task)
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/designate/worker/processing.py", line 77, in run
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server results = [r for r in self._executor.map(self.do, tasks)]
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/designate/worker/processing.py", line 77, in <listcomp>
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server results = [r for r in self._executor.map(self.do, tasks)]
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server yield fs.pop().result()
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server return self.__get_result()
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server raise s...

[David Ames (thedac)]

Print of self.target.to_dict()

{'pool_id': '794ccc2c-d751-44fe-b57f-8894c9f5c842', 'type': 'bind9', 'tsigkey_id': None, 'description': None, 'masters': [{'pool_target_id': '5d90ca98-26e9-4ae4-9769-9aa37741ba40', 'host': '10.5.0.23', 'port': 5354, 'id': '323bd470-7e12-4ad9-8594-7c436a4f7ad6', 'created_at': datetime.datetime(2018, 11, 6, 22, 38), 'updated_at': None, 'version': 1}, {'pool_target_id': '5d90ca98-26e9-4ae4-9769-9aa37741ba40', 'host': '10.5.0.17', 'port': 5354, 'id': 'd46a3775-ab96-4290-aaac-64fc79d4c850', 'created_at': datetime.datetime(2018, 11, 6, 22, 38), 'updated_at': None, 'version': 1}, {'pool_target_id': '5d90ca98-26e9-4ae4-9769-9aa37741ba40', 'host': '10.5.0.13', 'port': 5354, 'id': 'dfc96227-fb74-4135-9e3c-c3bc1749cf55', 'created_at': datetime.datetime(2018, 11, 6, 22, 38), 'updated_at': None, 'version': 1}], 'options': [{'pool_target_id': '5d90ca98-26e9-4ae4-9769-9aa37741ba40', 'key': 'rndc_host', 'value': '10.5.0.14', 'id': '4aac175b-10fa-44cc-aacf-0bcaf05f1d58', 'created_at': datetime.datetime(2018, 11, 6, 22, 38), 'updated_at': None, 'version': 1}, {'pool_target_id': '5d90ca98-26e9-4ae4-9769-9aa37741ba40', 'key': 'rndc_key_file', 'value': '/etc/designate/rndc.key', 'id': '980854bf-0e46-4a4f-a723-9cfdedefc1b5', 'created_at': datetime.datetime(2018, 11, 6, 22, 38), 'updated_at': None, 'version': 1}, {'pool_target_id': '5d90ca98-26e9-4ae4-9769-9aa37741ba40', 'key': 'host', 'value': '10.5.0.14', 'id': 'f2f84fd8-1da0-4f1a-98a6-0294a3090505', 'created_at': datetime.datetime(2018, 11, 6, 22, 38), 'updated_at': None, 'version': 1}], 'backend': <designate.backend.impl_bind9.Bind9Backend object at 0x7f89edabb240>, 'id': '5d90ca98-26e9-4ae4-9769-9aa37741ba40', 'created_at': datetime.datetime(2018, 11, 6, 22, 38), 'updated_at': None, 'version': 1}

Note there is no port in options. But it is elsewhere in the self.target object.

[David Ames (thedac)]

Hmm it is possible we need to put in port in our pools template:

    - type: bind9
      masters:

        - host: 10.5.0.23
          port: 5354

        - host: 10.5.0.17
          port: 5354

        - host: 10.5.0.13
          port: 5354

      options:
        host: 10.5.0.10
        rndc_host: 10.5.0.10
        rndc_key_file: /etc/designate/rndc.key

vs
https://docs.openstack.org/designate/rocky/admin/pools.html

  targets:
    - type: powerdns
      description: PowerDNS Database Cluster

      # List out the designate-mdns servers from which PowerDNS servers should
      # request zone transfers (AXFRs) from.
      masters:
        - host: 192.0.2.1
          port: 5354

      # PowerDNS Configuration options
      options:
        host: 192.0.2.2
        port: 53

[David Ames (thedac)]

Confirmed

2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/designate/worker/tasks/zone.py", line 126, in __call__
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server port = int(self.target.options.get('port'))
2018-11-06 22:27:28.496 23432 ERROR oslo_messaging.rpc.server TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

This is what is failing:
    def __call__(self):
        if not CONF['service:worker'].notify:
            # TODO(timsim): Remove this someday
            return True

        host = self.target.options.get('host')
        port = int(self.target.options.get('port'))

Simple reproducer and test is the current bionic-rocky designate charm amulet test.

Suggested fix:
Add the following in the pools.yaml template:

   - type: bind9
      masters:

        - host: 10.5.0.23
          port: 5354

        - host: 10.5.0.17
          port: 5354

        - host: 10.5.0.13
          port: 5354

      options:
        host: 10.5.0.14
        rndc_host: 10.5.0.14
        rndc_key_file: /etc/designate/rndc.key
+ port: 53

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/616052

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/616052
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=d850fa4ca1ddbb5d8c96e772b7c65d9a2bc7d2cd
Submitter: Zuul
Branch: master

commit d850fa4ca1ddbb5d8c96e772b7c65d9a2bc7d2cd
Author: David Ames <email address hidden>
Date: Tue Nov 6 18:48:20 2018 -0800

    Rocky requires the port option for targets

    Add the port option to the target in the pools.yaml template.

    Change-Id: I6ccaca587c487b42ba746d2b00975da1b1578761
    Closes-Bug: #1801985

[Drew Freiberger (afreiberger)]

I'm experiencing regression on 21.04 charm-designate with openstack-origin=cloud:bionic-stein (after upgrade from xenial-queens->bionic->bionic-rocky->bionic-stein) where the template/mitaka/pools.yaml is being used instead of the rocky pools.yaml.

[Drew Freiberger (afreiberger)]

My regression is linked to a different root issue. I've opened a new bug: https://bugs.launchpad.net/charm-designate/+bug/1928495