nova cell-show <cell name> causes ValueError: Circular reference detected

[Mh Raies]

I am using devstack development environment with n-cell enabled in localrc.
When I call API to list all cells a cell with name "child" is explored.
Again when I try to "cell-show" this cell "child" call is failed with an error response 500.
following is the operations followed -

 Setting in /etc/nova/nova.conf

        [cells]
        name = region
        cell_type = api
        enable = True

When API call to get cells -

REQUEST -
        curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token:<token>"

RESPONSE -
        HTTP/1.1 200 OK
        Content-Type: application/json
        Content-Length: 111
        X-Compute-Request-Id: req-e83c337a-a7d9-47e5-a472-72cd41126f21
        Date: Thu, 24 Apr 2014 04:12:34 GMT

        {"cells": [{"username": "guest", "rpc_host": "10.0.9.40", "type": "child", "name": "child", "rpc_port": 5672}]}

During particular cell show (using cell name as "child")-

REQUEST -
        curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells/child' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: <token>"

RESPONSE -
        HTTP/1.1 500 Internal Server Error
        Content-Length: 128
        Content-Type: application/json; charset=UTF-8
        X-Compute-Request-Id: req-ba2503bd-3948-4cce-8ee6-543aa597497b
        Date: Thu, 24 Apr 2014 04:10:33 GMT

        {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

Note: when using cell name as "api" or "region" 404 is returned.

nova-api logs during show call is traced as below -

    2014-04-24 11:15:20.219 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token from (pid=705) __call__ /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:603
    2014-04-24 11:15:20.220 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role from (pid=705) _remove_auth_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:662
    2014-04-24 11:15:20.232 DEBUG keystoneclient.middleware.auth_token [-] Storing token in cache from (pid=705) _cache_put /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:1121
    2014-04-24 11:15:20.233 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: 6cf0c59310fb4b189fc157b19d0e1026 with project_id : 2c3857a83f454b7cb073891ef47acd11 and roles: heat_stack_owner,admin from (pid=705) _build_user_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:910
    2014-04-24 11:15:20.238 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Matched GET /2c3857a83f454b7cb073891ef47acd11/os-cells/child from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
    2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Route path: '/{project_id}/os-cells/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
    2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>, 'project_id': u'2c3857a83f454b7cb073891ef47acd11', 'id': u'child'} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
    2014-04-24 11:15:20.240 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Calling method '<bound method Controller.show of <nova.api.openstack.compute.contrib.cells.Controller object at 0x3789c10>>' (Content-type='None', Accept='application/json') from (pid=705) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:945
    2014-04-24 11:15:20.253 ERROR nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Caught error: Circular reference detected
    Traceback (most recent call last):

      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
        incoming.message))

      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
        self._send_reply(conn, reply, failure, log_failure=log_failure)

      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
        conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))

      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
        _MESSAGE_KEY: jsonutils.dumps(raw_msg)}

      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
        return json.dumps(value, default=default, **kwargs)

      File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
        **kw).encode(obj)

      File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
        chunks = self.iterencode(o, _one_shot=True)

      File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
        return _iterencode(o, 0)

    ValueError: Circular reference detected

    2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return req.get_response(self.application)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack application, catch_exc_info=False)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    2014-04-24 11:15:20.253 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 616, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return self.app(env, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack response = self.app(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 917, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack content_type, body, accept)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 983, in _process_stack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1070, in dispatch
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return method(req=request, **action_args)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/common.py", line 602, in inner
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return function(*args, **kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/contrib/cells.py", line 264, in show
    2014-04-24 11:15:20.253 TRACE nova.api.openstack cell = self.cells_rpcapi.cell_get(context, id)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/opt/stack/nova/nova/cells/rpcapi.py", line 450, in cell_get
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return cctxt.call(ctxt, 'cell_get', cell_name=cell_name)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
    2014-04-24 11:15:20.253 TRACE nova.api.openstack wait_for_reply=True, timeout=timeout)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack timeout=timeout)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return self._send(target, ctxt, message, wait_for_reply, timeout)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack raise result
    2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
    2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    2014-04-24 11:15:20.253 TRACE nova.api.openstack incoming.message))
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
    2014-04-24 11:15:20.253 TRACE nova.api.openstack self._send_reply(conn, reply, failure, log_failure=log_failure)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
    2014-04-24 11:15:20.253 TRACE nova.api.openstack conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
    2014-04-24 11:15:20.253 TRACE nova.api.openstack _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return json.dumps(value, default=default, **kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
    2014-04-24 11:15:20.253 TRACE nova.api.openstack **kw).encode(obj)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    2014-04-24 11:15:20.253 TRACE nova.api.openstack chunks = self.iterencode(o, _one_shot=True)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    2014-04-24 11:15:20.253 TRACE nova.api.openstack return _iterencode(o, 0)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.259 INFO nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] http://10.0.9.40:8774/v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child returned with HTTP 500
    2014-04-24 11:15:20.259 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. from (pid=705) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1215
    2014-04-24 11:15:20.260 INFO nova.osapi_compute.wsgi.server [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] 10.0.9.40 "GET /v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child HTTP/1.1" status: 500 len: 335 time: 0.0411668

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/106991

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/106991
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Jay Pipes (jaypipes)]

This is back, but doesn't seem to be related strictly to cells any more:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVmFsdWVFcnJvcjogQ2lyY3VsYXIgcmVmZXJlbmNlIGRldGVjdGVkXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MTg3NTQzMTI2MjgsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

[Markus Zoeller (markus_z) (mzoeller)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY

[Anna Babich (ababich)]

The bug has been reproduced several times by different people: https://bugs.launchpad.net/nova/+bug/1611447

[Anna Babich (ababich)]

CONFIRMED FOR: MITAKA

[melanie witt (melwitt)]

This bug hasn't been commented on in nearly 2 years and is about legacy cells v1, which has been deprecated and is slated for removal within the next two cycles.