[repo] rabbitmq defaults to guest and doesnt work with multi-nova

[JJ Asghar]

I cloned down the testing stack and nova-schedular.log throws these errors:
dentials: Socket closed
2015-03-19 22:28:51.848 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:28:56.872 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:01.887 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:06.900 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:11.911 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:16.923 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:21.937 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:26.948 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:31.964 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:36.997 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:42.011 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:47.049 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:52.061 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:29:57.073 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:02.086 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:07.097 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:12.106 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:17.118 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:22.128 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:27.139 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:32.149 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:37.160 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:42.174 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:47.186 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:52.198 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:30:57.209 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:31:02.220 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
2015-03-19 22:31:07.236 572 ERROR oslo.messaging._drivers.impl_rabbit [req-b03c10ef-3a69-47b3-bbb2-3760f8d84983 ] AMQP server 10.0.1.60:5672 closed the connection. Check login credentials: Socket closed
(END)

If you look at the nova.conf

##### RABBITMQ #####
rabbit_userid=guest
rabbit_password=mypass
rabbit_virtual_host=/
rabbit_host=10.0.1.60
rabbit_port=5672
rabbit_use_ssl=false

This is not good for multi-nova rabbitmq connections.

[JJ Asghar (d-jj)]

https://github.com/stackforge/cookbook-openstack-ops-messaging/blob/master/recipes/rabbitmq-server.rb#L27

[JJ Asghar (d-jj)]

I think this is the most relevent portion why we need to override the mq:compute:attribute.

https://github.com/stackforge/cookbook-openstack-compute/blob/master/templates/default/nova.conf.erb#L42-L56

[Mark Vanderwiel (vanderwl)]

Humm, I'm not able to reproduce that, I get the rabbit_userid=admin in all my nova.conf's without the extra override. I'll run it again and check all the component users of rabbit and see what is in their config files.

The override is not a fix imo, it's just a workaround. It was designed to not have to do that, so we would like to keep digging further to understand why your env seems to be acting different.

Btw, I also checked our products env, and we do set the mq user in the env, but on any of the components, and it works there as well.

[JJ Asghar (d-jj)]

If you'd like i can screen share and show this to you. It's repo'd ever time.

I agree it's not a fix, but it keeps the testing-stack green which is more important than stopping everything just to find one problem.

[Mark Vanderwiel (vanderwl)]

I just verified that we don't need to even set the "rabbitmq":{ "default_user": "admin", "default_pass": "mypass" }, as the code is designed to basically ignore this. When left as defaults, guest/guest, the user guest gets deleted from the rabbit user list here: https://github.com/stackforge/cookbook-openstack-ops-messaging/blob/master/recipes/rabbitmq-server.rb#L64. And then the new mq user, "admin" is added here and used thru out the other cookbooks here: https://github.com/stackforge/cookbook-openstack-ops-messaging/blob/master/recipes/rabbitmq-server.rb#L71 and https://github.com/stackforge/cookbook-openstack-compute/blob/master/recipes/nova-common.rb#L85 which is initialized in Common here: https://github.com/stackforge/cookbook-openstack-common/blob/master/attributes/messaging.rb#L81.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167021

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/167021
Committed: https://git.openstack.org/cgit/stackforge/openstack-chef-repo/commit/?id=39414f452ad7820e8d2d955607670e1f0e93ed5c
Submitter: Jenkins
Branch: master

commit 39414f452ad7820e8d2d955607670e1f0e93ed5c
Author: JJ Asghar <email address hidden>
Date: Mon Mar 23 17:08:33 2015 -0500

    Fix for rabbit user issue.

    This creates the rabbit user to admin instead of guest. This is
    important for multi setups and also to confirm that we don't leverage
    the guest user for aio.

    Change-Id: I98d5c14badd8e1230a4f13d4d50a6c54556c19af
    Closes-Bug: 1434304

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-chef ocata-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-chef-repo ocata-eol release.