Marker could not be found?

[Sebastian Porombka]

I have a problem after grizzly->havana migration where i’m unable to rescue myself.
When I open the "Admin -> Resource-Usage View i get no results – only a red error box with the message "Error: Unable to retrieve tenant list.“.

Horizon log:
[Thu Oct 31 11:39:44 2013] [error] Creating a new keystoneclient connection to http://$controller:35357/v2.0.
[Thu Oct 31 11:39:44 2013] [error] REQ: curl -i -X GET http://$controller:35357/v2.0/tenants?marker=tenant_marker&limit=21 -H "User-Agent: python-keystoneclient" -H "Forwarded: for=131.234.5.178;by=python-keystoneclient" -H "X-Auth-Token: 82[…]f46"
[Thu Oct 31 11:39:44 2013] [error] REQ: curl -i -X GET http://$controller:35357/v2.0/tenants?marker=tenant_marker&limit=21 -H "User-Agent: python-keystoneclient" -H "Forwarded: for=131.234.5.178;by=python-keystoneclient" -H "X-Auth-Token: 82[…]46"
[Thu Oct 31 11:39:44 2013] [error] INFO:urllib3.connectionpool:Starting new HTTP connection (1): $controller
[Thu Oct 31 11:39:44 2013] [error] DEBUG:urllib3.connectionpool:"GET /v2.0/tenants?marker=tenant_marker&limit=21 HTTP/1.1" 400 88
[Thu Oct 31 11:39:44 2013] [error] RESP: [400] CaseInsensitiveDict({'date': 'Thu, 31 Oct 2013 11:39:47 GMT', 'vary': 'X-Auth-Token', 'content-length': '88', 'content-type': 'application/json'})
[Thu Oct 31 11:39:44 2013] [error] RESP BODY: {"error": {"message": "Marker could not be found", "code": 400, "title": "Bad Request"}}
[Thu Oct 31 11:39:44 2013] [error]
[Thu Oct 31 11:39:44 2013] [error] RESP: [400] CaseInsensitiveDict({'date': 'Thu, 31 Oct 2013 11:39:47 GMT', 'vary': 'X-Auth-Token', 'content-length': '88', 'content-type': 'application/json'})
[Thu Oct 31 11:39:44 2013] [error] RESP BODY: {"error": {"message": "Marker could not be found", "code": 400, "title": "Bad Request"}}
[Thu Oct 31 11:39:44 2013] [error]
[Thu Oct 31 11:39:44 2013] [error] Request returned failure status: 400
[Thu Oct 31 11:39:44 2013] [error] Request returned failure status: 400
[Thu Oct 31 11:39:44 2013] [error] Recoverable error: Marker could not be found (HTTP 400)

Keystone Log:
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Matched GET /tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Route path: '{path_info:.*}', defaults: {'controller': <keystone.contrib.ec2.routers.Ec2Extension object at 0x4156f10>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.352 17187 DEBUG routes.middleware [-] Match dict: {'controller': <keystone.contrib.ec2.routers.Ec2Extension object at 0x4156f10>, 'path_info': '/tenants'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Matched GET /tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Route path: '{path_info:.*}', defaults: {'controller': <keystone.contrib.s3.core.S3Extension object at 0x4156cd0>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.353 17187 DEBUG routes.middleware [-] Match dict: {'controller': <keystone.contrib.s3.core.S3Extension object at 0x4156cd0>, 'path_info': '/tenants'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.354 17187 DEBUG routes.middleware [-] Matched GET /tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.354 17187 DEBUG routes.middleware [-] Route path: '{path_info:.*}', defaults: {'controller': <keystone.contrib.admin_crud.core.CrudExtension object at 0x41517d0>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Match dict: {'controller': <keystone.contrib.admin_crud.core.CrudExtension object at 0x41517d0>, 'path_info': '/tenants'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Matched GET /tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.355 17187 DEBUG routes.middleware [-] Route path: '{path_info:.*}', defaults: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x4151e50>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.356 17187 DEBUG routes.middleware [-] Match dict: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x4151e50>, 'path_info': '/tenants'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.356 17187 DEBUG routes.middleware [-] Matched GET /tenants __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2013-10-31 12:39:47.357 17187 DEBUG routes.middleware [-] Route path: '/tenants', defaults: {'action': u'get_all_projects', 'controller': <keystone.identity.controllers.Tenant object at 0x4151ed0>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2013-10-31 12:39:47.357 17187 DEBUG routes.middleware [-] Match dict: {'action': u'get_all_projects', 'controller': <keystone.identity.controllers.Tenant object at 0x4151ed0>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2013-10-31 12:39:47.357 17187 DEBUG dogpile.core.dogpile [-] NeedRegenerationException _enter /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:94
2013-10-31 12:39:47.358 17187 DEBUG dogpile.core.dogpile [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:127
2013-10-31 12:39:47.358 17187 DEBUG dogpile.core.dogpile [-] value creation lock <dogpile.cache.region._LockWrapper object at 0x4fc77d0> acquired _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:131
2013-10-31 12:39:47.358 17187 DEBUG dogpile.core.dogpile [-] Calling creation function _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:148
2013-10-31 12:39:47.362 17187 DEBUG dogpile.core.dogpile [-] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2013-10-31 12:39:47.362 17187 DEBUG dogpile.core.dogpile [-] NeedRegenerationException _enter /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:94
2013-10-31 12:39:47.362 17187 DEBUG dogpile.core.dogpile [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:127
2013-10-31 12:39:47.362 17187 DEBUG dogpile.core.dogpile [-] value creation lock <dogpile.cache.region._LockWrapper object at 0x4fc7bd0> acquired _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:131
2013-10-31 12:39:47.363 17187 DEBUG dogpile.core.dogpile [-] Calling creation function _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:148
2013-10-31 12:39:47.365 17187 DEBUG dogpile.core.dogpile [-] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2013-10-31 12:39:47.366 17187 DEBUG dogpile.core.dogpile [-] NeedRegenerationException _enter /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:94
2013-10-31 12:39:47.366 17187 DEBUG dogpile.core.dogpile [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:127
2013-10-31 12:39:47.366 17187 DEBUG dogpile.core.dogpile [-] value creation lock <dogpile.cache.region._LockWrapper object at 0x4fc7bd0> acquired _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:131
2013-10-31 12:39:47.366 17187 DEBUG dogpile.core.dogpile [-] Calling creation function _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:148
2013-10-31 12:39:47.369 17187 DEBUG dogpile.core.dogpile [-] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2013-10-31 12:39:47.373 17187 WARNING keystone.common.wsgi [-] Marker could not be found
2013-10-31 12:39:47.373 17187 INFO access [-] 131.234.5.178 - - [31/Oct/2013:11:39:47 +0000] "GET http://$controller:35357/v2.0/tenants?marker=tenant_marker&limit=21 HTTP/1.0" 400 88

I have the following (ubuntu packaged) version of the possible involved components of openstack:

root@$controller:/# dpkg -l | grep keystone
ii keystone 1:2013.2-0ubuntu1~cloud0 OpenStack identity service - Daemons
ii python-keystone 1:2013.2-0ubuntu1~cloud0 OpenStack identity service - Python library
ii python-keystoneclient 1:0.3.2-0ubuntu1~cloud0 Client library for OpenStack Identity API

root@$dashboard:/# dpkg -l | grep openstack
ii openstack-dashboard 1:2013.2~rc3-0ubuntu1~cloud0 django web interface to Openstack
ii python-openstack-auth 1.1.1-0ubuntu1~cloud0 A django authentication backend for Openstack

What could get wrong?

[Dolph Mathews (dolph)]

Originally reported on the mailing list here-
http://lists.openstack.org/pipermail/openstack-dev/2013-October/018024.html

It looks like a variable substitution isn't occurring as expected ("tenant_marker" should probably be a value), but it's difficult to tell what is at fault here.

[David Lyle (david-lyle)]

This is a problem in the Horizon code. The string "tenant_marker" is being used rather than a uuid.

[Dolph Mathews (dolph)]

Thanks, David!

[David Lyle (david-lyle)]

Fixed by resolution of bug 1249279.

[Roman Sokolkov (rsokolkov)]

I've opened new bug on this issue.

https://bugs.launchpad.net/horizon/+bug/1260528