Nova ignores configs, uses wrong driver/protocol for AMQP from Havana -> Icehouse upgrade

[Will Foster]

This is related to an upgrade from Havana -> Icehouse.

It seems that nova-compute is passing the wrong protocol header to rabbitmq
which results in nova compute services never starting up, this is despite the fact that the nova.conf
driver for messaging is:

   rpc_backend=nova.openstack.common.rpc.impl_kombu

nova-compute logs the following and never starts up:

   2014-11-26 14:15:12.817 8585 ERROR oslo.messaging._drivers.impl_qpid
   [-] Unable to connect to AMQP server: client: 0-10, server: 9-1.
   Sleeping 5 seconds

This implies that the wrong driver is being used (hence the protocol
mismatch causing the connection termination?)

We see that the client (compute) wants to send protocol id major 1, protocol
id minor 1, version major 0, version minor 10, which rabbitmq doesn't
like (controller).

=ERROR REPORT==== 26-Nov-2014::01:32:33 ===
closing AMQP connection <0.2090.0> (192.168.0.7:50588 ->
192.168.0.222:5672)
{bad_version,{1,1,0,10}}

The configs are clean, and not telling it to use qpid so we believe
there may be some protocol field buried in the DB somewhere that it's
referencing still in place from Havana/QPID.

We believe that somewhere in the DB it's referencing some kind of protocol field
and it's being used regardless of what is in the config files or the driver it's using.

Of note, it also references the old controller IP address which is not
in the configs so we've brought up a virtual interface with that old IP
(192.168.0.222) to get past it for now.

Attached is a TCPDUMP of the activity.
We have been able to run sucessful db_sync for nova, neutron, keystone, glance, cinder
during the upgrade however this is blocking nova from starting.

[Will Foster (wfoster-b)]

Looks like this is solved, user/migration error :)
It looks like systemd spawns a new session for nova and if there's a nova.conf in /var/lib/nova it picks that one up first.

Removing this it will pick up the correct one due to:

https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L520