nova x509-create-cert times out

[Jamie Strandboge]

On a fresh 12.10 install and openstack installation following https://wiki.ubuntu.com/SecurityTeam/TestingOpenStack x509-create-cert fails.

$ nova x509-create-cert ~/openstack-certs/pk.pem ~/openstack-certs/cert.pem
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5ae0177f-ae7c-4230-9eac-2e1fc2c2ecd2)

rabbitmq is running.

/var/log/nova/nova-api.log tells me:
2012-07-30 14:03:04 INFO nova.api.openstack.wsgi [req-5ae0177f-ae7c-4230-9eac-2e1fc2c2ecd2 ad2dc0b86269455394cf3a1b630ac1b0 448c5952839d4b52aa87ff61c4c8950a] POST http://localhost:8774/v1.1/448c5952839d4b52aa87ff61c4c8950a/os-certificates
2012-07-30 14:03:04 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous call on cert ... from (pid=1629) multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:350
2012-07-30 14:03:04 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is 82a0e5656d6d49d4beca4aed11496d08 from (pid=1629) multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:353
2012-07-30 14:04:04 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 513, in ensure
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 590, in _consume
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 167, in drain_events
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 262, in drain_events
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 94, in drain_events
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 100, in wait_multi
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 159, in _wait_multiple
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 132, in read_timeout
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common raise m
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common timeout: timed out
2012-07-30 14:04:04 TRACE nova.openstack.common.rpc.common
2012-07-30 14:04:04 ERROR nova.api.openstack [req-5ae0177f-ae7c-4230-9eac-2e1fc2c2ecd2 ad2dc0b86269455394cf3a1b630ac1b0 448c5952839d4b52aa87ff61c4c8950a] Caught error: Timeout while waiting on RPC response.
2012-07-30 14:04:04 TRACE nova.api.openstack Traceback (most recent call last):
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 41, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack return req.get_response(self.application)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
2012-07-30 14:04:04 TRACE nova.api.openstack application, catch_exc_info=False)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
2012-07-30 14:04:04 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 185, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack return self.app(env, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack response = self.app(environ, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
2012-07-30 14:04:04 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 849, in __call__
2012-07-30 14:04:04 TRACE nova.api.openstack content_type, body, accept)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 897, in _process_stack
2012-07-30 14:04:04 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 972, in dispatch
2012-07-30 14:04:04 TRACE nova.api.openstack return method(req=request, **action_args)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/certificates.py", line 88, in create
2012-07-30 14:04:04 TRACE nova.api.openstack user_id=context.user_id, project_id=context.project_id)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/cert/rpcapi.py", line 58, in generate_x509_cert
2012-07-30 14:04:04 TRACE nova.api.openstack project_id=project_id))
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80, in call
2012-07-30 14:04:04 TRACE nova.api.openstack return rpc.call(context, self._get_topic(topic), msg, timeout)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 100, in call
2012-07-30 14:04:04 TRACE nova.api.openstack return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 712, in call
2012-07-30 14:04:04 TRACE nova.api.openstack rpc_amqp.get_connection_pool(conf, Connection))
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 367, in call
2012-07-30 14:04:04 TRACE nova.api.openstack rv = list(rv)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 328, in __iter__
2012-07-30 14:04:04 TRACE nova.api.openstack self.done()
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-07-30 14:04:04 TRACE nova.api.openstack self.gen.next()
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 325, in __iter__
2012-07-30 14:04:04 TRACE nova.api.openstack self._iterator.next()
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 595, in iterconsume
2012-07-30 14:04:04 TRACE nova.api.openstack yield self.ensure(_error_callback, _consume)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 526, in ensure
2012-07-30 14:04:04 TRACE nova.api.openstack error_callback(e)
2012-07-30 14:04:04 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 576, in _error_callback
2012-07-30 14:04:04 TRACE nova.api.openstack raise rpc_common.Timeout()
2012-07-30 14:04:04 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response.
2012-07-30 14:04:04 TRACE nova.api.openstack
2012-07-30 14:04:04 INFO nova.api.openstack [req-5ae0177f-ae7c-4230-9eac-2e1fc2c2ecd2 ad2dc0b86269455394cf3a1b630ac1b0 448c5952839d4b52aa87ff61c4c8950a] http://localhost:8774/v1.1/448c5952839d4b52aa87ff61c4c8950a/os-certificates returned with HTTP 500
2012-07-30 14:04:04 INFO nova.osapi_compute.wsgi.server [req-5ae0177f-ae7c-4230-9eac-2e1fc2c2ecd2 ad2dc0b86269455394cf3a1b630ac1b0 448c5952839d4b52aa87ff61c4c8950a] 127.0.0.1 - - [30/Jul/2012 14:04:04] "POST /v1.1/448c5952839d4b52aa87ff61c4c8950a/os-certificates HTTP/1.1" 500 335 60.119921

[Jamie Strandboge (jdstrand)]

My permissions seem to be set up correctly:
$ sudo rabbitmqctl list_permissions -p nova
Listing permissions in vhost "nova" ...
nova .* .* .*
...done.

[Jamie Strandboge (jdstrand)]

$ sudo rabbitmqctl list_queues -p nova
Listing queues ...
compute.openstack-quantal-amd64 0
scheduler 0
network 0
network_fanout_05f0d6549c2844e19f07c231f586da79 0
network.openstack-quantal-amd64 0
scheduler_fanout_6574c56f6f784928ba11af4a082c992a 0
compute 0
scheduler.openstack-quantal-amd64 0
compute_fanout_e524b27a69c343cb8ee864af410fc514 0
...done.

[Thierry Carrez (ttx)]

Is nova-cert installed and running ?

[Jamie Strandboge (jdstrand)]

No it was not. Somehow I had it installed in my 12.04 install even though it has no reverse dependencies in precise either. I've updated the instructions for our setup. Feel free to close this bug.