Glance fail to upload images to radosgw with Ceph Nautilus

[Alfredo Moralejo]

Configuring Ceph RadosGW Nautilus as swift backend for glance, it fails to upload images with following error:

# openstack image create --file hosts test-nuevo
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.

Logs in glance api:

2019-04-12 13:15:25.477 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] REQ: curl -i http://[::1]:8080/swift
/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b -X PUT -H "Content-Length: 0" -H "ETag: d41d8cd98f00b204e9800998ecf8427e" -H "X-Object-Manifest: glance/1aea2157-a2be-4337-be03-3faaa458018b-" -H "X-Auth-Token: gA
AAAABcsI9qJZ_N..."
2019-04-12 13:15:25.477 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP STATUS: 422 Unprocessable Entit
y
2019-04-12 13:15:25.478 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP HEADERS: {u'Content-Length': u'
19', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Thu, 01 Jan 1970 00:00:00 GMT', u'etag': u'b0e641c998cc3eae6fa2f8726d98cddd', u'X-Trans-Id': u'tx000000000000000000009-005cb08f6c-b491-default', u'Date': u'Fr
i, 12 Apr 2019 13:15:25 GMT', u'Content-Type': u'text/plain; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000009-005cb08f6c-b491-default'}
2019-04-12 13:15:25.478 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP BODY: UnprocessableEntity
2019-04-12 13:15:25.478 789960 ERROR glance_store._drivers.swift.store [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Failed to add
 object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.: ClientException: Object PUT failed: http://::1:8080/
swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity
2019-04-12 13:15:25.481 789960 ERROR glance.api.v2.image_data [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Failed to upload image
 data due to internal error: BackendException: Failed to add object to Swift.
2019-04-12 13:15:25.509 789960 ERROR glance.common.wsgi [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Caught error: Failed to add
object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.: BackendException: Failed to add object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/651337
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=ed356fc6b46e133899e1bf1c54df239536ea78b9
Submitter: Zuul
Branch: master

commit ed356fc6b46e133899e1bf1c54df239536ea78b9
Author: Alfredo Moralejo <email address hidden>
Date: Tue Apr 9 19:27:29 2019 +0200

    Do not include ETag when puting manifest in chunked uploads

    While testing glance with Ceph Rados Gateway using latest Ceph release
    (Nautilus), i've found that glance fails to upload the manifest using
    dynamic large objects mode because of the value used in ETag request.

    This issue has been reported to Ceph as it seems related to some recent
    change in radosgw code [1].

    However, checking at the upload workflow used by glance and comparing
    to Swift documentation [2], I wonder if adding the etag is actually
    providing any value. In the Swift the ETag header is used to validate
    integrity when uploading chunks, not the manifest while glance is doing
    exactly the oposite, not sending the etag in the chunks (I guess to
    avoid checksuming big images, which makes sense to me) and sending it
    when puting the manifest.

    This patch is removing the etag header when sending the PUT request for
    the manifest in chunked uploads.

    [1] https://tracker.ceph.com/issues/39160
    [2] https://docs.openstack.org/swift/latest/api/large_objects.html#dynamic-large-objects

    Closes-bug: #1824533
    Change-Id: I0b563dfcdc30026669fb089c82db8c3df7edc808

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/653486

[Donny Davis (donny-g)]

I can confirm the proposed patch works with Stein/Nautilus RGW in my deployment

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653486
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=053614be148a5b085b186c82ba630bf6ce2aa067
Submitter: Zuul
Branch: stable/stein

commit 053614be148a5b085b186c82ba630bf6ce2aa067
Author: Alfredo Moralejo <email address hidden>
Date: Tue Apr 9 19:27:29 2019 +0200

    Do not include ETag when puting manifest in chunked uploads

    While testing glance with Ceph Rados Gateway using latest Ceph release
    (Nautilus), i've found that glance fails to upload the manifest using
    dynamic large objects mode because of the value used in ETag request.

    This issue has been reported to Ceph as it seems related to some recent
    change in radosgw code [1].

    However, checking at the upload workflow used by glance and comparing
    to Swift documentation [2], I wonder if adding the etag is actually
    providing any value. In the Swift the ETag header is used to validate
    integrity when uploading chunks, not the manifest while glance is doing
    exactly the oposite, not sending the etag in the chunks (I guess to
    avoid checksuming big images, which makes sense to me) and sending it
    when puting the manifest.

    This patch is removing the etag header when sending the PUT request for
    the manifest in chunked uploads.

    [1] https://tracker.ceph.com/issues/39160
    [2] https://docs.openstack.org/swift/latest/api/large_objects.html#dynamic-large-objects

    Closes-bug: #1824533
    Change-Id: I0b563dfcdc30026669fb089c82db8c3df7edc808
    (cherry picked from commit ed356fc6b46e133899e1bf1c54df239536ea78b9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 0.29.0 release.