Nova showCellData request returns an "Unexpected API Error"

Bug #1611447 reported by Anna Babich
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Undecided
Unassigned

Bug Description

It has been reproduced against single-node devstack with cells enabled:
ENABLED_SERVICES=c-api,c-bak,c-sch,c-vol,cinder,dstat,g-api,g-reg,horizon,key,mysql,n-api,n-cell,n-cond,n-cpu,n-crt,n-net,n-obj,n-sch,rabbit,s-account,s-container,s-object,s-proxy,tempest

nova.conf:
[DEFAULT]
...
transport_url = rabbit://stackrabbit:secretrabbit@127.0.0.1:5672/
...
[cells]
name = region
cell_type = api
enable = True

nova-cells.conf:
[DEFAULT]
...
transport_url = rabbit://stackrabbit:secretrabbit@127.0.0.1:5672/
...
[cells]
name = child
cell_type = compute
enable = True

The http://developer.openstack.org/api-ref-compute-v2.1.html#listCells request has been successful and returned:

stack@ubuntu:~$ TOKEN=$(openstack token issue | awk '/ id /{print $4}')
stack@ubuntu:~$ NOVA_ENDPOINT=$(openstack endpoint show nova | awk '/ publicurl /{print $4}')
stack@ubuntu:~$ TENANT_ID=$(openstack project show admin | awk '/ id /{print $4}')
stack@ubuntu:~$ curl --include --request GET -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" $NOVA_ENDPOINT/$TENANT_ID/os-cells
HTTP/1.1 200 OK
Content-Length: 117
Content-Type: application/json
Openstack-Api-Version: compute 2.1
X-Openstack-Nova-Api-Version: 2.1
Vary: OpenStack-API-Version
Vary: X-OpenStack-Nova-API-Version
X-Compute-Request-Id: req-9d5ea58a-256f-42d3-bca6-c388e632ca02
Date: Tue, 09 Aug 2016 15:38:31 GMT

{"cells": [{"username": "stackrabbit", "rpc_host": "127.0.0.1", "type": "child", "name": "child", "rpc_port": 5672}]}

(It's unexpected, that parent cell is absent both here and in 'cells' table of Nova DB, but can suggest that single-node configuration may affect it)

After that I tried to send the http://developer.openstack.org/api-ref-compute-v2.1.html#showCellData request. The UUID of cell expected by this request isn't returned by previous one, and the 'uuid' field is absent in 'cells' Nova DB table:
mysql> select * from cells;
+---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
| created_at | updated_at | deleted_at | id | api_url | weight_offset | weight_scale | name | is_parent | deleted | transport_url |
+---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
| 2016-08-08 15:26:43 | NULL | NULL | 1 | NULL | 0 | 1 | child | 0 | 0 | rabbit://stackrabbit:secretrabbit@127.0.0.1:5672,stackrabbit:secretrabbit@127.0.0.1:5672/child_cell |
+---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

So, let's try to get cell's info by its name, and the result is:
stack@ubuntu:~/devstack$ curl --include --request GET -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" $NOVA_ENDPOINT/$TENANT_ID/os-cells/child
HTTP/1.1 500 Internal Server Error
Openstack-Api-Version: compute 2.1
X-Openstack-Nova-Api-Version: 2.1
Vary: OpenStack-API-Versio
Vary: X-OpenStack-Nova-API-Version
Content-Type: application/json; charset=UTF-8
Content-Length: 216
X-Compute-Request-Id: req-05f0f93a-c9ad-403b-ae3f-639b5f64d659
Date: Tue, 09 Aug 2016 15:40:32 GMT

{"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>", "code": 500}}

And there is n-api.log trace:
2016-08-09 11:39:32.215 24100 DEBUG nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Calling method '<bound method CellsController.show of <nova.api.openstack.compute.cells.CellsController object at 0x7f5ad1410f10>>' _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:636
2016-08-09 11:39:32.217 24100 DEBUG oslo_messaging._drivers.amqpdriver [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] CALL msg_id: b34cb4c869444efba1ba76844208cfba exchange 'nova' topic 'cells' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Unexpected exception in API method
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/common.py", line 530, in inner
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions return function(*args, **kwargs)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/compute/cells.py", line 164, in show
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions cell = self.cells_rpcapi.cell_get(context, id)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/cells/rpcapi.py", line 492, in cell_get
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions return cctxt.call(ctxt, 'cell_get', cell_name=cell_name)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions retry=self.retry)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 96, in _send
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions retry=retry)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID b34cb4c869444efba1ba76844208cfba
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions
2016-08-09 11:40:32.223 24100 INFO nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2016-08-09 11:40:32.223 24100 DEBUG nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1044
2016-08-09 11:40:32.226 24100 INFO nova.osapi_compute.wsgi.server [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] 127.0.0.1 "GET /v2.1/2bc2e7102ac94b71b260036c5ec13d10/os-cells/child HTTP/1.1" status: 500 len: 559 time: 60.0148242

Expected result:
Information about defined cell is returned without errors.

Tags: cells
Anna Babich (ababich)
tags: added: cells
Revision history for this message
Sean Dague (sdague) wrote :

What is in the n-cells log?

Changed in nova:
status: New → Incomplete
Revision history for this message
Anna Babich (ababich) wrote :
Revision history for this message
Anna Babich (ababich) wrote :
Revision history for this message
Anna Babich (ababich) wrote :

Found a trace in n-cell-region.log:
2016-08-09 11:39:32.230 24567 DEBUG oslo_messaging._drivers.amqpdriver [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] sending reply msg_id: b34cb4c869444efba1ba76844208cfba reply queue: reply_bf4f2fd40c694edc8aaa3a41b3e1f50c time elapsed: 0.0113621119963s _send_reply /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:73
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Can not send reply for message
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 146, in _process_incoming
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server message.reply(res)
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 98, in reply
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server self._send_reply(conn, reply, failure)
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 74, in _send_reply
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/common.py", line 285, in serialize_msg
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 184, in dumps
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server return json.dumps(obj, default=default, **kwargs)
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server sort_keys=sort_keys, **kw).encode(obj)
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server chunks = self.iterencode(o, _one_shot=True)
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server return _iterencode(o, 0)
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server ValueError: Circular reference detected
2016-08-09 11:39:32.231 24567 ERROR oslo_messaging.rpc.server

It looks like duplicate of https://bugs.launchpad.net/nova/+bug/1312002

Changed in nova:
status: Incomplete → Confirmed
Revision history for this message
sutefun (stefan-bujack) wrote :

Hy,

I also saw this bug.
Ubuntu 16.04
Openstack Mitaka
Package: nova-cells
Version: 2:13.1.0-0ubuntu1
Package: python
Version: 2.7.11-1

top level cell host with nova-api and nova-cells

nova-manage cell list
 Id Name Type Username Hostname Port VHost
--- ---------- ------ ---------- --------------- ----- ----------
  1 nova-cc01 child rabbit_cell1 IP 5672 /
--- ---------- ------ ---------- --------------- ----- ----------

child cell host with nova-cells nova-conductor nova-consoleauth nova-novncproxy nova- scheduler

nova-manage cell list
 Id Name Type Username Hostname Port VHost
--- ---------- ------ ---------- --------------- ----- ----------
  1 nova-tlc01 parent rabbit_api IP 5672 /
--- ---------- ------ ---------- --------------- ----- ----------

compute hosts with nova-compute

Revision history for this message
sutefun (stefan-bujack) wrote :
Revision history for this message
sutefun (stefan-bujack) wrote :
Revision history for this message
sutefun (stefan-bujack) wrote :
Revision history for this message
sutefun (stefan-bujack) wrote :
Revision history for this message
melanie witt (melwitt) wrote :

This bug hasn't been commented on in nearly 2 years and is about legacy cells v1, which has been deprecated and is slated for removal within the next two cycles.

Changed in nova:
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.