designate-manage does not properly support DEFAULT/transport_url

[Alex Schultz]

The designate-manage command does not appear to support using the DEFAULT/transport_url. We've been working on adding support for DEFAULT/transport_url as that is the recommended way to configure messaging with oslo messaging.

See https://review.openstack.org/#/c/404970/

2016-12-20 19:39:25.762 28479 CRITICAL designate [designate-manage - - - - -] AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2016-12-20 19:39:25.762 28479 ERROR designate Traceback (most recent call last):
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/bin/designate-manage", line 10, in <module>
2016-12-20 19:39:25.762 28479 ERROR designate sys.exit(main())
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/designate/cmd/manage.py", line 132, in main
2016-12-20 19:39:25.762 28479 ERROR designate fn(*fn_args)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/designate/manage/pool.py", line 133, in update
2016-12-20 19:39:25.762 28479 ERROR designate self.context, {"name": xpool['name']})
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/designate/loggingutils.py", line 25, in wrapped
2016-12-20 19:39:25.762 28479 ERROR designate return func(*args, **kwargs)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/designate/central/rpcapi.py", line 342, in find_pool
2016-12-20 19:39:25.762 28479 ERROR designate return self.client.call(context, 'find_pool', criterion=criterion)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 465, in call
2016-12-20 19:39:25.762 28479 ERROR designate return self.prepare().call(ctxt, method, **kwargs)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-20 19:39:25.762 28479 ERROR designate retry=self.retry)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-20 19:39:25.762 28479 ERROR designate timeout=timeout, retry=retry)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 467, in send
2016-12-20 19:39:25.762 28479 ERROR designate retry=retry)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 411, in _send
2016-12-20 19:39:25.762 28479 ERROR designate msg.update({'_reply_q': self._get_reply_q()})
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 383, in _get_reply_q
2016-12-20 19:39:25.762 28479 ERROR designate conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 374, in _get_connection
2016-12-20 19:39:25.762 28479 ERROR designate purpose=purpose)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/common.py", line 402, in __init__
2016-12-20 19:39:25.762 28479 ERROR designate self.connection = connection_pool.create(purpose)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/pool.py", line 144, in create
2016-12-20 19:39:25.762 28479 ERROR designate return self.connection_cls(self.conf, self.url, purpose)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 610, in __init__
2016-12-20 19:39:25.762 28479 ERROR designate self.ensure_connection()
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 709, in ensure_connection
2016-12-20 19:39:25.762 28479 ERROR designate self.ensure(method=lambda: self.connection.connection)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 817, in ensure
2016-12-20 19:39:25.762 28479 ERROR designate ret, channel = autoretry_method()
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2016-12-20 19:39:25.762 28479 ERROR designate return fun(*args, **kwargs)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 507, in __call__
2016-12-20 19:39:25.762 28479 ERROR designate self.revive(create_channel())
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 242, in channel
2016-12-20 19:39:25.762 28479 ERROR designate chan = self.transport.create_channel(self.connection)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2016-12-20 19:39:25.762 28479 ERROR designate self._connection = self._establish_connection()
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2016-12-20 19:39:25.762 28479 ERROR designate conn = self.transport.establish_connection()
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-12-20 19:39:25.762 28479 ERROR designate conn = self.Connection(**opts)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 180, in __init__
2016-12-20 19:39:25.762 28479 ERROR designate (10, 30), # tune
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2016-12-20 19:39:25.762 28479 ERROR designate self.channel_id, allowed_methods, timeout)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 274, in _wait_method
2016-12-20 19:39:25.762 28479 ERROR designate wait()
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
2016-12-20 19:39:25.762 28479 ERROR designate return self.dispatch_method(method_sig, args, content)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2016-12-20 19:39:25.762 28479 ERROR designate return amqp_method(self, args)
2016-12-20 19:39:25.762 28479 ERROR designate File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 530, in _close
2016-12-20 19:39:25.762 28479 ERROR designate (class_id, method_id), ConnectionError)
2016-12-20 19:39:25.762 28479 ERROR designate AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2016-12-20 19:39:25.762 28479 ERROR designate

[Dr. Jens Harbott (j-harbott)]

It seems that this has been fixed in master, at least I cannot reproduce any more, while the issue still exists in stable/ocata. Looking through the commit logs, I found https://review.openstack.org/475273 as the possible "culprit".

Can anyone confirm this? Would it be possible to backport that patch to Ocata?

[Alex Schultz (alex-schultz)]

I have a long standing patch to test this out over in puppet but it wasn't passing even as of last week so i don't know if it actually works. https://review.openstack.org/#/c/404970/

[Eduardo Gonzalez (egonzalez90)]

Still seeing issues with designate in kolla if not using oslo_messaging_rabbit settings while doing pool update.

No logs even using --debug, just the command keeps forever.

$ designate_worker designate-manage --debug pool update --file /etc/designate/pools.yaml
Updating Pools Configuration
****************************

[Dr. Jens Harbott (j-harbott)]

Regarding the logs, look in /var/log/designate/designate-manage.log or similar, the designate developers decided that it was a good idea to confuse users by hiding most of their output there.

[Dr. Jens Harbott (j-harbott)]

Regarding the bug, I was indeed misled. Devstack doesn't remove the default guest user when setting up rabbitmq and designate-manage seems to happily and successfully use that as a fallback. I'll try to propose a patch to devstack that removes the guest user and will help discover such issues early.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/502441

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/502441
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=30e037063d714e2ce9792bb462d01ba8bf2a5c98
Submitter: Jenkins
Branch: master

commit 30e037063d714e2ce9792bb462d01ba8bf2a5c98
Author: Jens Harbott <email address hidden>
Date: Mon Sep 11 10:00:45 2017 +0000

    Delay rpc initialization until config is available

    When being run within __init__() the rpc.init() call sees an empty
    config, leading to errors when doing RPC call later. So we move it to a
    later stage where we have read the config and can setup RPC properly.

    Change-Id: I195b57bf9c001883764dc160ced40272c5eb4fd3
    Closes-Bug: 1651576

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/514620

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/514620
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=efaea88bf13889519b4e370f2b3082e30ca0e973
Submitter: Zuul
Branch: stable/pike

commit efaea88bf13889519b4e370f2b3082e30ca0e973
Author: Jens Harbott <email address hidden>
Date: Mon Sep 11 10:00:45 2017 +0000

    Delay rpc initialization until config is available

    When being run within __init__() the rpc.init() call sees an empty
    config, leading to errors when doing RPC call later. So we move it to a
    later stage where we have read the config and can setup RPC properly.

    Change-Id: I195b57bf9c001883764dc160ced40272c5eb4fd3
    Closes-Bug: 1651576
    (cherry picked from commit 30e037063d714e2ce9792bb462d01ba8bf2a5c98)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/designate 6.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/designate 5.0.1 release.