Glance fail to upload images to radosgw with Ceph Nautilus

Bug #1824533 reported by Alfredo Moralejo on 2019-04-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
Status tracked in Stein
Queens
Undecided
Unassigned
Rocky
Undecided
Unassigned
Stein
Undecided
Unassigned

Bug Description

Configuring Ceph RadosGW Nautilus as swift backend for glance, it fails to upload images with following error:

# openstack image create --file hosts test-nuevo
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.

Logs in glance api:

2019-04-12 13:15:25.477 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] REQ: curl -i http://[::1]:8080/swift
/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b -X PUT -H "Content-Length: 0" -H "ETag: d41d8cd98f00b204e9800998ecf8427e" -H "X-Object-Manifest: glance/1aea2157-a2be-4337-be03-3faaa458018b-" -H "X-Auth-Token: gA
AAAABcsI9qJZ_N..."
2019-04-12 13:15:25.477 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP STATUS: 422 Unprocessable Entit
y
2019-04-12 13:15:25.478 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP HEADERS: {u'Content-Length': u'
19', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Thu, 01 Jan 1970 00:00:00 GMT', u'etag': u'b0e641c998cc3eae6fa2f8726d98cddd', u'X-Trans-Id': u'tx000000000000000000009-005cb08f6c-b491-default', u'Date': u'Fr
i, 12 Apr 2019 13:15:25 GMT', u'Content-Type': u'text/plain; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000009-005cb08f6c-b491-default'}
2019-04-12 13:15:25.478 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP BODY: UnprocessableEntity
2019-04-12 13:15:25.478 789960 ERROR glance_store._drivers.swift.store [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Failed to add
 object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.: ClientException: Object PUT failed: http://::1:8080/
swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity
2019-04-12 13:15:25.481 789960 ERROR glance.api.v2.image_data [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Failed to upload image
 data due to internal error: BackendException: Failed to add object to Swift.
2019-04-12 13:15:25.509 789960 ERROR glance.common.wsgi [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Caught error: Failed to add
object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.: BackendException: Failed to add object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.

Reviewed: https://review.openstack.org/651337
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=ed356fc6b46e133899e1bf1c54df239536ea78b9
Submitter: Zuul
Branch: master

commit ed356fc6b46e133899e1bf1c54df239536ea78b9
Author: Alfredo Moralejo <email address hidden>
Date: Tue Apr 9 19:27:29 2019 +0200

    Do not include ETag when puting manifest in chunked uploads

    While testing glance with Ceph Rados Gateway using latest Ceph release
    (Nautilus), i've found that glance fails to upload the manifest using
    dynamic large objects mode because of the value used in ETag request.

    This issue has been reported to Ceph as it seems related to some recent
    change in radosgw code [1].

    However, checking at the upload workflow used by glance and comparing
    to Swift documentation [2], I wonder if adding the etag is actually
    providing any value. In the Swift the ETag header is used to validate
    integrity when uploading chunks, not the manifest while glance is doing
    exactly the oposite, not sending the etag in the chunks (I guess to
    avoid checksuming big images, which makes sense to me) and sending it
    when puting the manifest.

    This patch is removing the etag header when sending the PUT request for
    the manifest in chunked uploads.

    [1] https://tracker.ceph.com/issues/39160
    [2] https://docs.openstack.org/swift/latest/api/large_objects.html#dynamic-large-objects

    Closes-bug: #1824533
    Change-Id: I0b563dfcdc30026669fb089c82db8c3df7edc808

Changed in glance-store:
status: New → Fix Released
Donny Davis (donny-g) wrote :

I can confirm the proposed patch works with Stein/Nautilus RGW in my deployment

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.