[any][17.08] notifications_designate.info queue has many unconsumed "dns.pool.update" messages due to ceilometer and designate notification_topics mismatch and missing events in event_definitions.yaml

[Dmitrii Shcherbakov]

ubuntu@maas:~/bundles⟫ juju config neutron-api openstack-origin
cloud:xenial-ocata
(openstack-client) ubuntu@maas:~/bundles⟫ juju config designate openstack-origin
cloud:xenial-ocata

# no consumers
ubuntu@juju-01e0c6-3-lxd-8:~$ sudo rabbitmqctl list_queues -p openstack name consumers messages | grep designate
notifications_designate.info 0 1164

openstack catalog list
http://paste.ubuntu.com/25317766/

# on designate/0
sudo cat /etc/designate/designate.conf
http://paste.ubuntu.com/25317485/

Designate is configured to use notifications_designate topic:

ubuntu@juju-01e0c6-3-lxd-2:~$ sudo grep notification /etc/designate/designate.conf
# Driver used for issuing notifications
notification_driver = messaging
notification_topics = notifications_designate
# List of notification handlers to enable, configuration of these needs to
enabled_notification_handlers = nova_fixed, neutron_floatingip

juju status
http://paste.ubuntu.com/25317520/
designate 4.0.0 active 3 designate jujucharms 9 ubuntu

# all other queues are empty
sudo rabbitmqctl list_queues -p openstack
http://paste.ubuntu.com/25317525/

Designate does need to process messages sent by neutron but this is a different case (messages are sent by designate itself, see the end of this bug description)

https://github.com/openstack/designate/blob/stable/ocata/designate/notification_handler/neutron.py#L45-L57

---

neutron-api has the right notification_topics set:

juju run --application neutron-api 'sudo grep topics /etc/neutron/neutron.conf'
- Stdout: |
    notification_topics = notifications,notifications_designate
  UnitId: neutron-api/0
- Stdout: |
    notification_topics = notifications,notifications_designate
  UnitId: neutron-api/1
- Stdout: |
    notification_topics = notifications,notifications_designate
  UnitId: neutron-api/2

notification_topics appears to be deprecated anyway:
https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/config-reference/source/samples/neutron/neutron.conf.sample#L1290
"# Deprecated group/name - [DEFAULT]/notification_topics"

I don't see any info messages posted in the designate driver code - only designate client usage here:
https://github.com/openstack/neutron/blob/master/neutron/services/externaldns/drivers/designate/driver.py#L64

E.g. https://github.com/openstack/python-designateclient/blob/stable/ocata/designateclient/v2/recordsets.py#L44-L66

related bug (although it talks about a case where there's no designate):
https://bugs.launchpad.net/charms/+source/neutron-api/+bug/1659943

---

Either way, there are no consumers for that queue.

ubuntu@juju-01e0c6-3-lxd-8:~$ sudo rabbitmqctl list_queues -p openstack name consumers messages | grep designate
notifications_designate.info 0 1164

A sample message obtained via rabbitmq management ui (they are all the same - I've checked):
http://paste.ubuntu.com/25317774/

"event_type": "dns.pool.update",

https://docs.openstack.org/designate/ocata/notifications.html#emitters
"Emitters
They are emitted by Central on the following events:
...
dns.pool.update
..."

https://github.com/openstack/designate/blob/stable/ocata/designate/central/service.py#L2213

I haven't found a consumer for those messages (or at least it is not neutron):

➜ python-designateclient git:(stable/ocata) grep -RiP pool
designateclient/functionaltests/v2/test_zone.py: self.assertEqual(self.zone.pool_id, zone.pool_id)
designateclient/v2/pools.py:class PoolController(V2Controller):
designateclient/v2/pools.py: url = '/pools'
designateclient/v2/pools.py: return self._get(url, response_key='pools')
designateclient/v2/client.py:from designateclient.v2.pools import PoolController
designateclient/v2/client.py: self.pools = PoolController(self)
doc/source/shell-v2-examples.rst: | pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
doc/source/shell-v2.rst: | pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |

➜ designate git:(stable/ocata) grep -RiP pool ; echo $?
1

[James Page (james-page)]

I see the same:

ubuntu@juju-2758f7-xenial-pike-23:~$ sudo rabbitmqctl list_queues -p openstack name consumers messages | grep designate
notifications_designate.info 0 962

[Dmitrii Shcherbakov (dmitriis)]

I think that ceilometer's event_definitions.yaml could be extended to consume those events.

https://github.com/openstack/ceilometer/blob/stable/ocata/etc/ceilometer/event_definitions.yaml#L405-L435

[Dmitrii Shcherbakov (dmitriis)]

So, to break the comms down, designate components do both sending and receiving via AMQP.

1. nova to designate

https://git.io/vFMbn
nova -> {compute.instance.create.end, compute.instance.delete.start} -> designate-sink -> nova-handler

cfg.CONF.register_opts([
    cfg.ListOpt('notification-topics', default=['notifications']),
    cfg.StrOpt('control-exchange', default='nova'),

2. neutron to designate

https://git.io/vFMb8
neutron -> {floatingip.update.end, floatingip.delete.start} -> designate-sink -> neutron-handler

cfg.CONF.register_opts([
    cfg.ListOpt('notification-topics', default=['notifications']),
    cfg.StrOpt('control-exchange', default='neutron'),

https://docs.openstack.org/designate/pike/contributor/architecture.html#designate-sink
designate-sink is an optional service which listens for event Notifications, such as compute.instance.create.end, handlers are available for Nova and Neutron. Notification events can then be used to trigger record creation & deletion.

3. central -> other consumers (e.g. ceilometer-agent-notification)

https://github.com/openstack/designate/blob/stable/ocata/designate/central/service.py#L163-L164
https://github.com/openstack/designate/blob/stable/ocata/designate/central/service.py#L2213

The designate charm hard-codes the notification_topics to "notifications_designate" (outgoing traffic)

https://git.io/vFMAP
notification_topics = notifications_designate

While per oslo.messaging the default is "notifications" which is overridden by that

https://github.com/openstack/oslo.messaging/blob/stable/ocata/oslo_messaging/notify/notifier.py#L49-L57

    cfg.ListOpt('topics',
                default=['notifications', ],
                deprecated_opts=[
                    cfg.DeprecatedOpt('topics',
                                      group='rpc_notifier2'),
                    cfg.DeprecatedOpt('notification_topics',
                                      group='DEFAULT')
                ],
                help='AMQP topic used for OpenStack notifications.'),
]

charm-ceilometer, OTOH, doesn't redefine anything like that, which is why not only dns.pool.update but other designate-related messages as well are not received by it - even those that are in event_definitions.yaml.

As I pasted above, there are also no event_definitions on the ceilometer side for dns.pool.* including dns.pool.update.

https://github.com/openstack/ceilometer/blob/stable/ocata/etc/ceilometer/event_definitions.yaml#L405-L435

The use-case may be event/audit logging but at least in this case we will solve the lack of other consumers.

AFAIKs, designate uses standard DNS protocol messages so any AMQP messages are rather for custom logic that could be implemented via ceilometer + aodh or pipelines in addition to event logging to panko https://docs.openstack.org/designate/pike/contributor/architecture.html#designate-minidns

For that reason I would add all possible emitted messages to event_definitions.yaml:

https://docs.openstack.org/designate/pike/admin/notifications.html#emitters

4. mdns -> DNS backend <- nothing to do here

[Dmitrii Shcherbakov (dmitriis)]

It's questionable which traits to add to event_definitions.yaml by default on the ceilometer side.

Each object has it's own set of traits (FIELDS) + inherited traits from base classes.

class Pool(base.DictObjectMixin, base.PersistentObjectMixin,
           base.DesignateObject):
    FIELDS = {

I would say that ceilometer needs to render event_definitions.yaml with a sane set of default event definitions but should allow one to supply a completely arbitrary event_definitions.yaml.

I could add PersistentObjectMixin and DesignateObject at least.

event_definitions.yaml is highly customizable and I do not think it is necessary to preseed it - it should just be taken from a user because modeling and parsing event and trait definitions in a charm is cumbersome. Another reason is that an operator may want to reduce unnecessary chatter in telemetry but given a charm manages the config it's not possible to just go and edit it.

https://docs.openstack.org/ceilometer/pike/contributor/events.html#definitions-file-format

Likewise, meters.yaml should be replaceable as well https://github.com/openstack/ceilometer/blob/stable/pike/ceilometer/data/meters.d/meters.yaml

This is necessary to avoid charm forks just to add another trait or two.

[James Page (james-page)]

@dmitriis

Re your comments in #4 - ceilometer should ship with sane defaults for events and meters - any work to improve coverage should be done as part of ceilometer, not as part of the charm IMHO.

[Dmitrii Shcherbakov (dmitriis)]

@james-page I do agree about upstream ceilometer defaults. I think the new object CRUD events were just not added there.

I think this goes both ways (upstream & charms) though to make things configurable in terms of traits (not just events processed).

An operator may want to limit event traits to FIELDS of certain mixins only or to parts of mixins. For that we may just allow supplying a yaml directly (if you need that granularity - feel free to write your own stuff in other words).

[Dmitrii Shcherbakov (dmitriis)]

the topic side of this:

https://review.openstack.org/#/q/status:open%20topic:designate-notifications

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521073
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=45792d3c865149f2c94cef42b19c1385d7f7b0ee
Submitter: Zuul
Branch: master

commit 45792d3c865149f2c94cef42b19c1385d7f7b0ee
Author: Dmitrii Shcherbakov <email address hidden>
Date: Fri Nov 17 15:31:01 2017 +0300

    drop notifications_designate

    A separate topic has been used for nova, neutron -> designate. Let's use
    a single one instead.

    Change-Id: I17d0036f6dcbfd0e87849091e453d2a8b2bc498f
    Closes-Bug: #1710831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521072
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=b89de21d47dc9114d20456452cd867041bcd83c5
Submitter: Zuul
Branch: master

commit b89de21d47dc9114d20456452cd867041bcd83c5
Author: Dmitrii Shcherbakov <email address hidden>
Date: Fri Nov 17 15:13:22 2017 +0300

    drop driver and topic settings in designate ctx

    A similar change will land in designate to drop this extra topic as
    well.

    Also, driver and topic overrides in a service-specific context are not a
    good idea.

    Change-Id: I804a34fb044090010ecfd2560594cc1f55e9bd21
    Closes-Bug: #1710831
    Closes-Bug: #1738100

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.11
Review: https://review.openstack.org/528304

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.11
Review: https://review.openstack.org/531334

[OpenStack Infra (hudson-openstack)]

Change abandoned by James Page (<email address hidden>) on branch: stable/17.11
Review: https://review.openstack.org/528304

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521075
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=1879b16ea543b076c9555fdc0258355d39af9ad0
Submitter: Zuul
Branch: master

commit 1879b16ea543b076c9555fdc0258355d39af9ad0
Author: Dmitrii Shcherbakov <email address hidden>
Date: Fri Nov 17 14:59:50 2017 +0300

    drop notifications_designate, add missing section

    A separate topic has been used for nova, neutron -> designate. Let's use
    a single one instead.

    Change-Id: I709a6aca1843918a67d43bc4808a0013168fbc55
    Closes-Bug: #1710831

[Dmitrii Shcherbakov (dmitriis)]

I think charm-based customization of event_definitions.yaml is needed after all although ceilometer needs to ship a more up-to-date list of events.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/531334
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=4adfc46f842e1aec20e739a7b6d94ab14e7f4cfe
Submitter: Zuul
Branch: stable/17.11

commit 4adfc46f842e1aec20e739a7b6d94ab14e7f4cfe
Author: Dmitrii Shcherbakov <email address hidden>
Date: Fri Nov 17 15:13:22 2017 +0300

    drop driver and topic settings in designate ctx

    A similar change will land in designate to drop this extra topic as
    well.

    Also, driver and topic overrides in a service-specific context are not a
    good idea.

    Closes-Bug: #1710831
    Closes-Bug: #1738100
    (cherry picked from commit b89de21d47dc9114d20456452cd867041bcd83c5)

    Change-Id: Ie2d50b1ccd31ac430ce09cb96aceace3a31b2b92

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.11
Review: https://review.openstack.org/534504

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.11
Review: https://review.openstack.org/534505

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.11
Review: https://review.openstack.org/534506

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/540833

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/540834

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/540837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/540833
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=269ba2bb1d48ea515b315a9a7226dc010784ccaa
Submitter: Zuul
Branch: master

commit 269ba2bb1d48ea515b315a9a7226dc010784ccaa
Author: Dmitrii Shcherbakov <email address hidden>
Date: Mon Feb 5 14:22:54 2018 +0300

    reintroduce designate_notifications topic

    Change I709a6aca1843918a67d43bc4808a0013168fbc55, commit hash
    1879b16ea543b076c9555fdc0258355d39af9ad0 introduced a
    backwards-incompatible change in the fact that designate stopped
    consuming messages using designate_notifications topic from nova and
    neutron.

    While designate.conf change in the mentioned commit is needed to make
    sure Designate emits its own notifications on the topic that ceilometer
    uses ("notifications"), for nova and neutron to designate notifications
    a separate topic should be used as before.

    A revert of the previous commit is not done as it addresses the previous
    issue but introduces a new one.

    Change-Id: I6b5fd1b5f9bae1cd4a1683fd476b456e81d6c79e
    Related-Bug: #1710831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/540834
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=6b9b02c11201d1afd960af0196eff56c6f81ffef
Submitter: Zuul
Branch: master

commit 6b9b02c11201d1afd960af0196eff56c6f81ffef
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Feb 8 17:06:50 2018 +0300

    emit notifications on notifications_designate

    Change I804a34fb044090010ecfd2560594cc1f55e9bd21, commit hash
    b89de21d47dc9114d20456452cd867041bcd83c5 dropped notifications_designate
    completely to solve a problem described in pad.lv/1738100, however, to
    make this change backwards compatible notifications_designate should be
    used in addition to "notifications" topic used by default.

    This way designate will continue to receive notifications on
    notifications_designate from nova, and ceilometer will get notifications
    using the "notifications" topic.

    Change-Id: I245f5b263994c204a5e521dad542ed83952f54b8
    Related-Bug: #1710831
    Related-Bug: #1738100

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/540837
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=ff2547c6d552e85890c86d0677126ca1c09a9a04
Submitter: Zuul
Branch: master

commit ff2547c6d552e85890c86d0677126ca1c09a9a04
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Feb 8 17:16:57 2018 +0300

    Revert "drop notifications_designate"

    This reverts commit 45792d3c865149f2c94cef42b19c1385d7f7b0ee.

    During the discussions in reviews relevant to the bugs below the
    decision was to use both notifications_designate and notifications
    topics.

    Related-Bug: #1710831
    Related-Bug: #1738100
    Change-Id: I496f54d9917fe8c8279ebb91cc2fed02b74b181c

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: stable/17.11
Review: https://review.openstack.org/534505

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: stable/17.11
Review: https://review.openstack.org/534504

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: stable/17.11
Review: https://review.openstack.org/534506
Reason: Old code.