Unprocessable Entity error for large images on Ceph Swift store

[Mike Dorman]

There is an implementation difference between Ceph Swift and OS Swift in how the ETag/checksum of a dynamic large object (DLO) manifest object is verified.

OS Swift verifies it just like any other object, md5’ing the content of the object:
https://github.com/openstack/swift/blob/master/swift/obj/server.py#L439-L459

Ceph Swift actually does the full DLO checksum across all the component objects:
https://github.com/ceph/ceph/blob/master/src/rgw/rgw_op.cc#L1765-L1781

The Glance Swift store driver assumes the OS Swift behavior, and sends an ETag of md5("") with the PUT request for the manifest object. Technically, this is correct, since that object itself is a zero-byte object:
https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L552

However, when using a Ceph Swift store, this results in a 422 Unprocessable Entity response from Swift, because the provided ETag doesn't match the expected ETag for the DLO.

It would seem to make sense to just not send any ETag with the manifest object PUT request. It is not required by the API, and only marginally improves the validation of the object.

[Stuart McLaren (stuart-mclaren)]

Hi Mike.

I see that omitting the etag on upload would make uploads work against Ceph's Swift.

However my preference would be to fix the Swift compatibility in Ceph -- that would work for all applications, not just Glance.
It would also ensure that gate tests etc which run against OpenStack Swift would be guaranteed to work with Ceph Swift as the backend. (Otherwise future bugs breaking compatability could be introduced.)

[Mike Dorman (mdorman-m)]

I see your point. However, one could argue that the OpenStack Swift implementation is the one that should be fixed. Technically, according to the Swift docs, Ceph Swift is calculating the ETag correctly.

Fundamentally it comes down to the question: should manifest objects treated as a regular, zero-length, object; or treated specially as a DLO object?

It seems to me that since the specific behavior in this situation isn't spelled out in the API spec, then Glance can't really make any assumptions about what the correct behavior should be. By removing the ETag from this request, as well as the associated tests, it would eliminate exposing this implementation difference, which doesn't have any bearing on API compatibility.

I have also opened Ceph bug http://tracker.ceph.com/issues/9973 . I would be happy t open an OS Swift bug as well, to get this on their radar, if people think that would be a productive thing to do.

[Mike Dorman (mdorman-m)]

This has been fixed in upstream Ceph: http://tracker.ceph.com/issues/9973

[Josh Durgin (jdurgin)]

Fixed in ceph, no changes needed to glance_store.