Comment 2 for bug 1589389

I checked the logs and see the following:

1) request to boot an instance indeed fails with MessagingTimeout in nova-api on node-5

2016-06-03T14:33:04.297454+00:00 err: 2016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions [req-3ff12a0b-d829-4217-adb3-c870e88a3b3
4 71d4f3e3083643ddb2b2eb1a7dcb74b4 8fa6f6fac6d149dfb84fc021c32619b1 - - -] Unexpected exception in API method#0122016-06-03 14:33:04.281 9426 E
RROR nova.api.openstack.extensions Traceback (most recent call last):#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions Fil
e "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped#0122016-06-03 14:33:04.281 9426 ERROR nova.api.open
stack.extensions return f(*args, **kwargs)#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/d
ist-packages/nova/api/validation/__init__.py", line 73, in wrapper#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions retu
rn func(*args, **kwargs)#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/
validation/__init__.py", line 73, in wrapper#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions return func(*args, **kwarg
s)#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py
", line 73, in wrapper#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions return func(*args, **kwargs)#0122016-06-03 14:33
:04.281 9426 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in
create#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions **create_kwargs)#0122016-06-03 14:33:04.281 9426 ERROR nova.api.
openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner#0122016-06-03 14:33:04.281 9426 ERROR nova.api
.openstack.extensions rv = f(*args, **kwargs)#0122016-06-03 14:33:04.281 9426 ERROR nova.api.openstack.extensions File "/usr/lib/python2.
7/dist-packages/nova/compute/api.p

Because it tries to do an RPC call to nova-network, which we do not actually deploy.

2) looks like nova.conf was updated, but nova-api was not restarted after that:

$ grep use_neutron ./node-5/etc/nova/nova.conf
use_neutron=True

2016-06-03T14:07:47.358934+00:00 debug: 2016-06-03 14:07:47.358 6789 DEBUG oslo_service.service [-] use_neutron = False log_opt_values /usr/lib/python2.7/dist-packages/oslo_config/cfg.py:2517

which makes nova-api think it should call nova-network, not neutron.

The workaround is to restart nova-api on all controller nodes. We'll need to check the Puppet manifests to make sure nova-api is properly restarted after all changes done to config files.