Glance failed to upload image to swift storage

[Roman Podoliaka]

When glance configured with swift backend, and swift API provides via RadosGW is unable to upload image.

Upstream bug: https://bugs.launchpad.net/glance/+bug/1518431

Command:
glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64.img --visibility public --progress
Logs:
http://paste.openstack.org/show/479621/

[Serge Kovaleff (serge-kovaleff)]

This bug prevents RadosGW from working. Which in turn prevents deployment Ironic on top of Ceph.

In other words without this MOS with Ironic vill stuck to SWIFT.

[Mike Fedosin (mfedosin)]

Swift driver is designed to work with Swift backend. If you use it with some thirdparty solutions, we can't guarantee its operability.

RadosGW claimed to be 100% compatible with Swift API, but here we see inconsistencies. Even if we fix it somehow in Glance the problem will exist and may appear in other projects.

So I think it's better to address this bug to mos-ceph team.

[Serge Kovaleff (serge-kovaleff)]

 Dear Ceph team, is there any comments or updates on this ticket?

[Radoslaw Zarzynski (rzarzynski)]

It seems that the observed behavior is a result of sending PUT requests
without specifying Content-Length HTTP header. RFC 7230 states that
the header should be applied [1]:

    "3.3.2. Content-Length
    (...)
    A user agent SHOULD send a Content-Length in a request message when
    no Transfer-Encoding is sent and the request method defines a meaning
    for an enclosed payload body. For example, a Content-Length header
    field is normally sent in a POST request even when the value is 0
    (indicating an empty payload body)."

Most likely the improper requests have not been even delivered to radosgw
instance, but rejected by intermediary Apache server that is used as a FastCGI
front-end to radosgw. We already saw similar behavior in #1505273.

You may enforce curl to generate Content-Length for a request with empty
body through appending "-d ''" or "-H 'Content-Length: 0" to the argument list.

[1] https://tools.ietf.org/html/rfc7230#page-30
[2] https://bugs.launchpad.net/mos/+bug/1505273

[Radoslaw Zarzynski (rzarzynski)]

Reassigning the bug to MOS Glance Team because sending
properly-formed HTTP requests is client's responsibility.

For details please refer to my previous comment [1].

[1] https://bugs.launchpad.net/mos/+bug/1521623/comments/4

[Kairat Kushaev (kkushaev)]

@Radoslaw,
AFAIK, Glance do not generate Content-Length header when transfer-encoding = chunked because we don't know content length here until final chunk is generated (so we in compliance with ietf spec here because transfer-encoding is specified).

It seems that this is WSGI framework problem because some WSGI servers do not support chunked requests.
See Keystone bug for example: https://bugs.launchpad.net/keystone/+bug/1346211.

AFAIK that latest version of mod_wsgi supports chunked requests as eventlet does (that's why swift works correctly with Glance).
I see that we cannot generate requests with Content-Length here, because the length is unknown.
I think that WSGI server for RadosGW need to understand chunked-requests instead.
After reviewing 1505273 it seems that the problem is the same: WSGI server doesn't support chunked requests.

Radoslaw, is it possible to deploy RadosGW im MOS under eventlet or other WSGI framework that treats chunked requests properly?
I am not an expert here so waiting for your answer.

[Bogdan Dobrelya (bogdando)]

Fuel w/a https://review.openstack.org/#/c/258521

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/258521
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e8567ee17b6f013d8d0f1460415523af084adcc7
Submitter: Jenkins
Branch: master

commit e8567ee17b6f013d8d0f1460415523af084adcc7
Author: Andrey Shestakov <email address hidden>
Date: Wed Dec 16 16:05:18 2015 +0200

    Change apache2 CGI module for RadosGW

    In apache-2.4.7 fastcgi module is not support chunked transfer.
    It breaks uploads of objects to RadosGW.
    Fix available since apache-2.4.10 (vivid) and upgrade apache in MOS is not
    possible, so this workaround proposed.
    RadosGW runs with native CGI server on 127.0.0.1:9000 and apache uses
    mod_proxy_fcgi to proxy user requests to rados.

    Closes-bug: #1521623
    Closes-bug: #1505273

    Change-Id: Ia3959c4009c6c673074a9b7536b1cc87834c301d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/261353

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/261353
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b092234605defaf6e41b15dc949472bd4b2d04ee
Submitter: Jenkins
Branch: stable/8.0

commit b092234605defaf6e41b15dc949472bd4b2d04ee
Author: Andrey Shestakov <email address hidden>
Date: Wed Dec 16 16:05:18 2015 +0200

    Change apache2 CGI module for RadosGW

    In apache-2.4.7 fastcgi module is not support chunked transfer.
    It breaks uploads of objects to RadosGW.
    Fix available since apache-2.4.10 (vivid) and upgrade apache in MOS is not
    possible, so this workaround proposed.
    RadosGW runs with native CGI server on 127.0.0.1:9000 and apache uses
    mod_proxy_fcgi to proxy user requests to rados.

    Closes-bug: #1521623
    Closes-bug: #1505273

    Change-Id: Ia3959c4009c6c673074a9b7536b1cc87834c301d
    (cherry picked from commit e8567ee17b6f013d8d0f1460415523af084adcc7)

[Roman Podoliaka (rpodolyaka)]

Only important for Ironic, which we don't have in 7.0

[Pavlo Shchelokovskyy (pshchelo)]

various reports in Internet (e.g. [1]) and our colleagues hint that some fast_cgi versions (including some Apache ones) have a bug where created semaphores are not cleaned up, leading to a situation when node might be completely locked up. The bug most probably will not manifest immediately, only after a certain amount of using the running service.

We should double-check that our version is not affected with e.g. observing that output of

ipcs -s

is not constantly growing over time under constant load.

Should I file a separate bug just in case?

[1] https://forums.cpanel.net/threads/no-space-left-on-device-apache-error-but-there-was.96345/

[Alexander Zatserklyany (zatserklyany)]

Fix released.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "507"
---------------------
root@node-9:~# glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64-disk1.img --visibility public --progress
...
see http://paste.openstack.org/show/485717/
...
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 77987bde53c54dc436e299af51d7b03f |
| container_format | bare |
| created_at | 2016-02-02T11:26:41Z |
| direct_url | rbd://f1f4fb66-b546-43fe-a91d- |
| | 6091f91d15e0/images/1dab5ce0-33e8-40e7-9b05-b0fdf7fbc9a9/snap |
| disk_format | raw |
| id | 1dab5ce0-33e8-40e7-9b05-b0fdf7fbc9a9 |
| min_disk | 0 |
| min_ram | 0 |
| name | trusty_ext4 |
| owner | a894eccfb1564f49ac939f8b1d013967 |
| protected | False |
| size | 258474496 |
| status | active |
| tags | [] |
| updated_at | 2016-02-02T11:27:16Z |
| virtual_size | None |
| visibility | public |
+------------------+----------------------------------------------------------------------------------+
root@node-9:~# glance image-list
+--------------------------------------+-------------+
| ID | Name |
+--------------------------------------+-------------+
| a471b5ce-3ce9-4813-a901-2e12607d74a3 | TestVM |
| 1dab5ce0-33e8-40e7-9b05-b0fdf7fbc9a9 | trusty_ext4 |
+--------------------------------------+-------------+
root@node-9:~# glance image-download 1dab5ce0-33e8-40e7-9b05-b0fdf7fbc9a9 ...