Cannot view object details with Ceph backend

[Rohit Karajgi]

Steps to reproduce:

With Ceph configured as the Object Storage backend,

Navigate to Object Store - > Containers.
Click on a Container in the list of containers.
For an Object in In the Objects panel, Click on More -> View Details.

Expected Result:
Details of object should be displayed in a pop-up modal.

Actual Result:
An Error pop-up appears with the text " Error: An error occurred. Please try again later."

I am able to retrieve object details using the swfitclient:

$ swift list
container1

$ swift stat container1 object1
       Account: v1
     Container: container1
        Object: functions
  Content Type: binary/octet-stream
Content Length: 59177
 Last Modified: Tue, 25 Mar 2014 09:46:45 GMT
          ETag: a837e4f4ac61417e6385a896cf4ba409
    Meta Mtime: 1394204154.954066
 Accept-Ranges: bytes
        Server: Apache/2.2.22 (Ubuntu)

[Rohit Karajgi (rohitk)]

Horizon throws the following error in the logs:

[Tue Mar 25 10:11:54 2014] [error] DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
[Tue Mar 25 10:11:54 2014] [error] WARNING:py.warnings:DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
[Tue Mar 25 10:11:54 2014] [error] Internal Server Error: /project/containers/container1/functions/object_detail
[Tue Mar 25 10:11:54 2014] [error] Traceback (most recent call last):
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 140, in get_response
[Tue Mar 25 10:11:54 2014] [error] response = response.render()
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/response.py", line 105, in render
[Tue Mar 25 10:11:54 2014] [error] self.content = self.rendered_content
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/response.py", line 82, in rendered_content
[Tue Mar 25 10:11:54 2014] [error] content = template.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
[Tue Mar 25 10:11:54 2014] [error] return self._render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
[Tue Mar 25 10:11:54 2014] [error] return self.nodelist.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 830, in render
[Tue Mar 25 10:11:54 2014] [error] bit = self.render_node(node, context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/debug.py", line 74, in render_node
[Tue Mar 25 10:11:54 2014] [error] return node.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/loader_tags.py", line 124, in render
[Tue Mar 25 10:11:54 2014] [error] return compiled_parent._render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
[Tue Mar 25 10:11:54 2014] [error] return self.nodelist.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 830, in render
[Tue Mar 25 10:11:54 2014] [error] bit = self.render_node(node, context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/debug.py", line 74, in render_node
[Tue Mar 25 10:11:54 2014] [error] return node.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/loader_tags.py", line 63, in render
[Tue Mar 25 10:11:54 2014] [error] result = block.nodelist.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 830, in render
[Tue Mar 25 10:11:54 2014] [error] bit = self.render_node(node, context)
[Tue Mar 25 10:11:54 2014] [error] File ...

[Rohit Karajgi (rohitk)]

The object detail headers returned by Rados gateway to swiftclient does not contain the 'x-timestamp' header that Horizon looks for.
'x-timestamp' is returned in the case of a Swift backend.

A closer look into the response headers:
{'content-length': '154', 'x-object-meta-mtime': '1394533200.312213', 'accept-ranges': 'bytes', 'server': 'Apache/2.2.22 (Ubuntu)', 'last-modified': 'Mon, 24 Mar 2014 12:03:59 GMT', 'etag': '7cb550ed2aa02c83247878e3df5fbcc5', 'date': 'Mon, 24 Mar 2014 14:01:08 GMT', 'content-type': 'binary/octet-stream'}

Ceph sets the last modified time in "x-object-meta-mtime", which, IMO, is the parameter that needs to be checked.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/82801

[Akihiro Motoki (amotoki)]

Where is the API defined? Is x-object-meta-mtime a defined attribute in Swift API?
I think this comes from the incompatibility between Ceph and Swift.

I don't object the change itself but as far as I understand we don't officially support Swift-compatible backednd.

[Luis Pabón (lpabon)]

@amotoki, I completely agree. I think this bug/feature should not be a launchpad.net(openstack) bug, but instead a Ceph bug/feature that needs to be added.

http://tracker.ceph.com/issues/8911

[Gary W. Smith (gary-w-smith)]

Marking invalid per above recommendation and the fact that this is being tracked as a bug in Ceph

[Jay Pipes (jaypipes)]

Why is this a bug in Ceph, guys? Where is X-Object-Meta-Mtime header a documented part of the Swift API?

Horizon should be able to deal with "missing" headers like this without any changes to the Ceph RadosGW project.

-jay

[David Lyle (david-lyle)]

We should not require an optional header. Changing status.

[Kevin Fox (kevpn)]

As a radosgw user, I'd agree that it should be made optional rather then required. Though I just tested it on our rdo juno/giant ceph, cloud and it seems to work without a patch. So maybe upgrading to a newer Ceph will fix the issue too?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthias Runge (<email address hidden>) on branch: master
Review: https://review.openstack.org/82801
Reason: no change here for the last 6 months. Feel free to restore, if you want to work on this patch.

[Vladislav Kuzmin (vkuzmin-u)]

I've checked it with upstream Horizon and Ceph and it works. I think we can close this bug since Ceph was patched http://tracker.ceph.com/issues/8911