Cannot view object details with Ceph backend

Bug #1297173 reported by Rohit Karajgi on 2014-03-25
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Medium
Rohit Karajgi

Bug Description

Steps to reproduce:

With Ceph configured as the Object Storage backend,

Navigate to Object Store - > Containers.
Click on a Container in the list of containers.
For an Object in In the Objects panel, Click on More -> View Details.

Expected Result:
Details of object should be displayed in a pop-up modal.

Actual Result:
An Error pop-up appears with the text " Error: An error occurred. Please try again later."

I am able to retrieve object details using the swfitclient:

$ swift list
container1

$ swift stat container1 object1
       Account: v1
     Container: container1
        Object: functions
  Content Type: binary/octet-stream
Content Length: 59177
 Last Modified: Tue, 25 Mar 2014 09:46:45 GMT
          ETag: a837e4f4ac61417e6385a896cf4ba409
    Meta Mtime: 1394204154.954066
 Accept-Ranges: bytes
        Server: Apache/2.2.22 (Ubuntu)

Rohit Karajgi (rohitk) on 2014-03-25
Changed in horizon:
assignee: nobody → Rohit Karajgi (rohitk)
Rohit Karajgi (rohitk) wrote :
Download full text (4.0 KiB)

Horizon throws the following error in the logs:

[Tue Mar 25 10:11:54 2014] [error] DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
[Tue Mar 25 10:11:54 2014] [error] WARNING:py.warnings:DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
[Tue Mar 25 10:11:54 2014] [error] Internal Server Error: /project/containers/container1/functions/object_detail
[Tue Mar 25 10:11:54 2014] [error] Traceback (most recent call last):
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 140, in get_response
[Tue Mar 25 10:11:54 2014] [error] response = response.render()
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/response.py", line 105, in render
[Tue Mar 25 10:11:54 2014] [error] self.content = self.rendered_content
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/response.py", line 82, in rendered_content
[Tue Mar 25 10:11:54 2014] [error] content = template.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
[Tue Mar 25 10:11:54 2014] [error] return self._render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
[Tue Mar 25 10:11:54 2014] [error] return self.nodelist.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 830, in render
[Tue Mar 25 10:11:54 2014] [error] bit = self.render_node(node, context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/debug.py", line 74, in render_node
[Tue Mar 25 10:11:54 2014] [error] return node.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/loader_tags.py", line 124, in render
[Tue Mar 25 10:11:54 2014] [error] return compiled_parent._render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
[Tue Mar 25 10:11:54 2014] [error] return self.nodelist.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 830, in render
[Tue Mar 25 10:11:54 2014] [error] bit = self.render_node(node, context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/debug.py", line 74, in render_node
[Tue Mar 25 10:11:54 2014] [error] return node.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/loader_tags.py", line 63, in render
[Tue Mar 25 10:11:54 2014] [error] result = block.nodelist.render(context)
[Tue Mar 25 10:11:54 2014] [error] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 830, in render
[Tue Mar 25 10:11:54 2014] [error] bit = self.render_node(node, context)
[Tue Mar 25 10:11:54 2014] [error] File ...

Read more...

Rohit Karajgi (rohitk) wrote :

The object detail headers returned by Rados gateway to swiftclient does not contain the 'x-timestamp' header that Horizon looks for.
'x-timestamp' is returned in the case of a Swift backend.

A closer look into the response headers:
{'content-length': '154', 'x-object-meta-mtime': '1394533200.312213', 'accept-ranges': 'bytes', 'server': 'Apache/2.2.22 (Ubuntu)', 'last-modified': 'Mon, 24 Mar 2014 12:03:59 GMT', 'etag': '7cb550ed2aa02c83247878e3df5fbcc5', 'date': 'Mon, 24 Mar 2014 14:01:08 GMT', 'content-type': 'binary/octet-stream'}

Ceph sets the last modified time in "x-object-meta-mtime", which, IMO, is the parameter that needs to be checked.

Fix proposed to branch: master
Review: https://review.openstack.org/82801

Changed in horizon:
status: New → In Progress
Rohit Karajgi (rohitk) on 2014-03-25
tags: added: havana-backport-potential
Akihiro Motoki (amotoki) wrote :

Where is the API defined? Is x-object-meta-mtime a defined attribute in Swift API?
I think this comes from the incompatibility between Ceph and Swift.

I don't object the change itself but as far as I understand we don't officially support Swift-compatible backednd.

Luis Pabón (lpabon) wrote :

@amotoki, I completely agree. I think this bug/feature should not be a launchpad.net(openstack) bug, but instead a Ceph bug/feature that needs to be added.

http://tracker.ceph.com/issues/8911

Gary W. Smith (gary-w-smith) wrote :

Marking invalid per above recommendation and the fact that this is being tracked as a bug in Ceph

Changed in horizon:
status: In Progress → Invalid
Jay Pipes (jaypipes) wrote :

Why is this a bug in Ceph, guys? Where is X-Object-Meta-Mtime header a documented part of the Swift API?

Horizon should be able to deal with "missing" headers like this without any changes to the Ceph RadosGW project.

-jay

David Lyle (david-lyle) wrote :

We should not require an optional header. Changing status.

Changed in horizon:
status: Invalid → Confirmed
importance: Undecided → Medium
Kevin Fox (kevpn) wrote :

As a radosgw user, I'd agree that it should be made optional rather then required. Though I just tested it on our rdo juno/giant ceph, cloud and it seems to work without a patch. So maybe upgrading to a newer Ceph will fix the issue too?

Change abandoned by Matthias Runge (<email address hidden>) on branch: master
Review: https://review.openstack.org/82801
Reason: no change here for the last 6 months. Feel free to restore, if you want to work on this patch.

Vladislav Kuzmin (vkuzmin-u) wrote :

I've checked it with upstream Horizon and Ceph and it works. I think we can close this bug since Ceph was patched http://tracker.ceph.com/issues/8911

Changed in horizon:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers