Can't delete image from glance on env with ceph\radosgw

[Yury Tregubov]

On MOS 10.0 Several tests are failed with following traceback:

Traceback (most recent call last):

  File "tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc

    return func(*args, **kwargs)

  File "tempest/lib/services/image/v1/images_client.py", line 98, in delete_image

    resp, body = self.delete(url)

  File "tempest/lib/common/rest_client.py", line 307, in delete

    return self.request('DELETE', url, extra_headers, headers, body)

  File "tempest/lib/common/rest_client.py", line 664, in request

    self._error_checker(resp, resp_body)

  File "tempest/lib/common/rest_client.py", line 766, in _error_checker

    raise exceptions.BadRequest(resp_body, resp=resp)

tempest.lib.exceptions.BadRequest: Bad request

Details: 400 Bad Request

Error decoding your request. Either the URL or the request body contained characters that could not be decoded by Glance

The last reproduction was seen on mos10.0 iso #1360.

Tempest logs are attached.

Diagnostic snapshot is: https://drive.google.com/a/mirantis.com/file/d/0B1Crk-sAvGanRzRGeEp3VExLZXM/view?usp=sharing

[Oleksiy Butenko (obutenko)]

I tried delete image from glance via cli and recived the same error.
In glance-api.log found:
"curl -i http://10.109.1.9:8080/swift/v1/glance?format=json&prefix=f1c8ec73-20bb-40a3-bc39-aa543c254c42- "
This error reproduced only on env with radosgw, and as a now, url for delete object
should contain chunk.
I try upgrade python-swiftclient to latest version 3.3.0 (now we have installed 3.1.0) bug it didn't help.
After downgrade to python-swiftclient==3.0.0 url for delete have format:
http://10.109.1.9:8080/swift/v1/glance/f1c8ec73-20bb-40a3-bc39-aa543c254c42-00001
and the image has been removed successfully.
In global requirements for newton:
https://github.com/openstack/requirements/blob/stable/newton/global-requirements.txt
python-swiftclient>=2.2.0 #
So, I think we can downgrade version

[Oleksiy Butenko (obutenko)]

I think we can downgrade version as a workaround only

[Radoslaw Zarzynski (rzarzynski)]

It looks that newer versions of the swiftclient aren't able
to live to with gzipped chunked transfer encoding that Apache
applies while exposing the RadosGW. [1]

swiftclient throws UnicodeDecodeError exception when it comes
to interpret the magic number (0x1f 0x8b) present in the gzip
header [2].

Disabling mod_gzip (SetEnv no-gzip 1) in RadosGW's VirtualHosts
on all controllers fixes the issue.

Anyway, Ceph RadosGW definitely isn't a culprit here.

[1] https://gist.github.com/rzarzynski/56e5fae3b00b284d7ff256a7584dc329#file-rgw-over-apache-gzipped-response-txt
[2] https://gist.github.com/rzarzynski/eb102395214e5a1bd36865f124d2a5a3

[Vitalii Gridnev (vgridnev)]

Fix proposed: https://review.fuel-infra.org/#/c/31589/

[Vitalii Gridnev (vgridnev)]

Issue was introduced by the following change: https://review.openstack.org/#/c/184956/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/31589
Submitter: Pkgs Jenkins <email address hidden>
Branch: 10.0/newton

Commit: 7992ddac75f807fd51a01dabc3a371d1c825262a
Author: Vitaly Gridnev <email address hidden>
Date: Tue Mar 7 09:34:25 2017

Fix logging of the gzipped body

Seems like issue was introduced by the following change
https://review.openstack.org/#/c/184956/

Change-Id: Ib0b8d90a73e21ed78019186d6cb2b55de03a8190
Closes-bug: 1665351

[Oleksiy Butenko (obutenko)]

still reproduced on ci

[Oleksiy Butenko (obutenko)]

We don't have stable iso with fix

[Yury Tregubov (ytregubov)]

Verified on mos 10.0 build #1566