transport_url is being parsed different when forking with oslo.service

[Sam Yaple]

This is a strange one. I had setup my transport_url incorrectly which made me discover this. I am filing this bug because I believe there is one.

When using a correctly formatted transport_url that was missing data _and_ multiple workers for conductor it would incorrectly parse the transport_url.

Here is the incorrect transport_url:

rabbit://<user>:<pass>@server01,server02/openstack

The correct transport_url should be:

rabbit://<user>:<pass>@server01,<user>:<pass>@server02/openstack

According to the docs [1], when the username and password is not specified, the server should be omitted from the list of servers. So for the incorrect transport_url only server01 would be setup. Here is a list of scenarios and what the behavior I see is.

* with the incorrect transport_url and a single thread all services contact only server01
* with the incorrect transport_url and multiple threads all services contact only server01 _except_ conductor which will attempt to contact server02 with the default username and password
* with the correct transport_url all services contact server01 and server02 properly

To reproduce this, get two rabbitmq hosts and set the incorrect transport_url and at least 2 conductor workers. Look at the rabbitmq logs on server02 and youll see it is trying to login via the guest user. Set the number of workers to 1 and notice there is no bad logins in rabbitmq logs. Fix the transport_url and it will use all servers appropriately no matter the number of workers.

The following is a gut feeling. I believe the issue is around the point it gets forked with oslo.service. It may not properly be reparsing the transport_url. I couldn't prove that though.

Nova versions tested: 13.0.0 and 13.1.0
all libraries were within the upper-constraints for mitaka

[1] http://docs.openstack.org/developer/oslo.messaging/transport.html

[Sean Dague (sdague)]

this feels like it may very well be an issue in oslo.service itself, adding that

[ChangBo Guo(gcb) (glongwave)]

In http://docs.openstack.org/developer/oslo.messaging/transport.html

Username and password should be specified for each host, in case of lack of specification they will be omitted:

user:pass@host1:port1,host2:port2

[
  {"username": "user", "password": "pass", "host": "host1:port1"},
  {"host": "host2:port2"}
]

I doesn't mean the server is ommited , but username/password are ommited. so It works as design.

We can make it more clear about this, maybe a warning is logged in this case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/350438

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350438
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=34122ee3cb08b4de379b8095f4b18d64c38f62e2
Submitter: Jenkins
Branch: master

commit 34122ee3cb08b4de379b8095f4b18d64c38f62e2
Author: ChangBo Guo(gcb) <email address hidden>
Date: Wed Aug 3 16:08:57 2016 +0800

    Add warning when credential is not specified for each host

    User may not set username/password for each host in config
    option transport_url. All hosts should have username/password,
    or no host have username/password at the same time.
    This commit adds warning if user doesn't set properly.

    Closes-Bug: #1595965
    Change-Id: I4e204ce8365274566064168a2ba02c3278579717

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/oslo.messaging 5.9.0 release.

[Sean Dague (sdague)]

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'

[melanie witt (melwitt)]

It appears this has been fixed in oslo.messaging and was not related to nova. Please re-open this bug if that's not the case.