Nova connects to rabbitmq successfully but has invalid credentials

[Tyler Bishop]

From rabbitmq log:

=INFO REPORT==== 28-Aug-2015::10:54:20 ===
accepting AMQP connection <0.15664.0> (10.0.2.26:55772 -> 10.0.2.8:5672)

=INFO REPORT==== 28-Aug-2015::10:54:20 ===
Mirrored queue 'q-agent-notifier-security_group-update_fanout_c8d714e02b944c7f91dad2530a34ff01' in vhost '/': Adding mirror on node 'rabbit@os-controller-1003': <7448.19519.0>

=INFO REPORT==== 28-Aug-2015::10:54:20 ===
Mirrored queue 'q-agent-notifier-dvr-update_fanout_87fb0fc8e8224ffb88ea91ee20ad8e29' in vhost '/': Adding mirror on node 'rabbit@os-controller-1002': <7447.3416.0>

=INFO REPORT==== 28-Aug-2015::10:54:20 ===
Mirrored queue 'q-agent-notifier-dvr-update_fanout_87fb0fc8e8224ffb88ea91ee20ad8e29' in vhost '/': Adding mirror on node 'rabbit@os-controller-1003': <7448.19521.0>

=ERROR REPORT==== 28-Aug-2015::10:54:20 ===
closing AMQP connection <0.15305.0> (10.0.2.26:55758 -> 10.0.2.8:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'openstack' - invalid credentials",
                             'connection.start_ok'}}

=INFO REPORT==== 28-Aug-2015::10:54:21 ===
accepting AMQP connection <0.15747.0> (10.0.2.26:55773 -> 10.0.2.8:5672)

From Nova Log:

2015-08-28 10:54:19.524 14743 DEBUG oslo_concurrency.lockutils [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Lock "compute_resources" acquired by "_update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444
2015-08-28 10:54:19.827 14743 INFO nova.compute.resource_tracker [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Total usable vcpus: 48, total allocated vcpus: 1
2015-08-28 10:54:19.827 14743 INFO nova.compute.resource_tracker [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Final resource view: name=osc-1001.prd.cin1.corp.hosting.net phys_ram=257524MB used_ram=2560MB phys_disk=5GB used_disk=20GB total_vcpus=48 used_vcpus=1 pci_stats=<nova.pci.stats.PciDeviceStats object at 0x3f80250>
2015-08-28 10:54:19.886 14743 INFO nova.scheduler.client.report [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Compute_service record updated for ('osc-1001.prd.cin1.corp.hosting.net', 'osc-1001.prd.cin1.corp.hosting.net')
2015-08-28 10:54:19.886 14743 INFO nova.compute.resource_tracker [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Compute_service record updated for osc-1001.prd.cin1.corp.hosting.net:osc-1001.prd.cin1.corp.hosting.net
2015-08-28 10:54:19.887 14743 DEBUG oslo_concurrency.lockutils [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Lock "compute_resources" released by "_update_available_resource" :: held 0.363s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456
2015-08-28 10:54:19.922 14743 DEBUG nova.service [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Creating RPC server for service compute start /usr/lib/python2.7/site-packages/nova/service.py:188
2015-08-28 10:54:19.925 14743 INFO oslo_messaging._drivers.impl_rabbit [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Connecting to AMQP server on 10.0.2.8:5672
2015-08-28 10:54:19.943 14743 INFO oslo_messaging._drivers.impl_rabbit [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Connected to AMQP server on 10.0.2.8:5672
2015-08-28 10:54:19.969 14743 DEBUG nova.service [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] Join ServiceGroup membership for this service compute start /usr/lib/python2.7/site-packages/nova/service.py:206
2015-08-28 10:54:19.969 14743 DEBUG nova.servicegroup.drivers.db [req-91919deb-c6be-42a8-91f6-557f078f7d19 - - - - -] DB_Driver: join new ServiceGroup member osc-1001.prd.cin1.corp.hosting.net to the compute group, service = <nova.service.Service object at 0x364f9d0> join /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:59

Rabbit configuration in nova.conf
[oslo_messaging_rabbit]
rabbit_hosts=10.0.2.8:5672,10.0.2.7:5672,10.0.2.6:5672
rabbit_userid=openstack
rabbit_password=placeholderpassword

Functionality seems fine and nothing shows up in the nova log but rabbitmq references this hypervisor and with only the nova openstack service running i get the message.

[Sean Dague (sdague)]

This really isn't a nova bug, this section of the conf routes directly to oslo.messaging and nova never even knows it's a thing

[Julia Portnova (yportnova)]

I tried to reproduce this with later master.
Everything works as expected

http://paste.openstack.org/show/479110/