swift backend: openstack image-download / glance image-save fails

[Thomas Goirand]

Hi,

I'm the maintainer of OpenStack in Debian. Using a deployment with Swift as backend, with glance-api.conf configured this way:

[swift]
swift_store_cacert=/etc/ssl/certs/oci-pki-oci-ca-chain.pem
swift_store_create_container_on_put=True
swift_store_endpoint_type=internalURL
swift_store_config_file=/etc/glance/glance-swift.conf
default_swift_reference=ref1

# cat /etc/glance/glance-swift.conf

[ref1]
user = services:glance
key = PASSWORD
auth_version = 3
auth_address = https://<API_ADDRESS>:443/identity/v3
user_domain_id=default
project_domain_id=default

I could upload an image to Glance. It's really there in Swift, I checked for that fact. Though saving the image fails: when I do "openstack image save", my swift-proxy recieves a HTTP/1.1 499 (ie: Client Closed Request), then glance-api returns a 502 (bad gatway). Unfortunately, using uwsgi and https for Glance wasn't very much verbose, so I downgraded Glance-api to use eventlet without ssl, and then I could see in the glance-api.log:

2020-12-14 10:43:47.367 16080 DEBUG swiftclient [req-f1a898a5-e202-45b7-80e1-7bb68c3b3f52 dcc01371101246afacc8403030921f53 d71a5d98aef04386b57736a4ea4f3644 - default default] RESP STATUS: 200 OK http_log /usr/lib/python3/dist-packages/swiftclient/client.py:188
2020-12-14 10:43:47.367 16080 DEBUG swiftclient [req-f1a898a5-e202-45b7-80e1-7bb68c3b3f52 dcc01371101246afacc8403030921f53 d71a5d98aef04386b57736a4ea4f3644 - default default] RESP HEADERS: {'Content-Type': 'application/octet-stream', 'Etag': 'aef23ab9c77b8caa2e6042fa30aadd95', 'Last-Modified': 'Mon, 14 Dec 2020 10:18:04 GMT', 'X-Timestamp': '1607941083.89546', 'Accept-Ranges': 'bytes', 'X-Trans-Id': 'txdabda903f73c47d1a266e-005fd741e1', 'X-Openstack-Request-Id': 'txdabda903f73c47d1a266e-005fd741e1', 'Connection': 'close', 'Strict-Transport-Security': 'max-age=63072000'} http_log /usr/lib/python3/dist-packages/swiftclient/client.py:189
2020-12-14 10:43:47.368 16080 WARNING glance.location [req-f1a898a5-e202-45b7-80e1-7bb68c3b3f52 dcc01371101246afacc8403030921f53 d71a5d98aef04386b57736a4ea4f3644 - default default] Get image 8d2ca7c8-de71-41c1-a6bc-73dd0dd37646 data failed: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.
2020-12-14 10:43:47.368 16080 ERROR glance.location [req-f1a898a5-e202-45b7-80e1-7bb68c3b3f52 dcc01371101246afacc8403030921f53 d71a5d98aef04386b57736a4ea4f3644 - default default] Glance tried all active locations to get data for image 8d2ca7c8-de71-41c1-a6bc-73dd0dd37646 but all have failed.

Then later on, I get some:

  File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
    return resp(environ, start_response)
TypeError: 'ImageProxy' object is not callable

but that's a consequence of Glance-api not being able to properly download the image from Swift (so I didn't past all the stack-dump above).

My setup is with the packages from Debian (which I maintain), running Victoria over Buster. If you want to try, with Buster you can do:

apt-get install extrepo
extrepo enable openstack_victoria
apt-get update
apt-get install glance-api...

If you are brave enough, you can also try directly in Debian Unstable (that's the same packages which I upload there, and maintain as backports for Debian Stable).

Cheers,

Thomas Goirand

[Abhishek Kekane (abhishek-kekane)]

Setup environment using devstack + swift, not able to reproduce this issue with both openstackclient as well as python-glanceclient.

Need to replicate exact environment for testing.

[Jim Scadden (jimscadden)]

I've hit this same issue. If it helps, the following warning log entry:

WARNING glance.location [...] Get image xxx data failed: int() argument must be a string, a bytes-like object or a number, not 'NoneType'.

This appears to originate from the following line in _RetryBody.__init__ from /usr/lib/python3/dist-packages/swiftclient/client.py:

self.expected_length = int(self.resp.getheader('Content-Length'))

Looking at the previous log entry, which helpfully dumps some response headers, there isn't any mention of Content-Length.

I'm also seeing the following error on the swift proxy log:

uwsgi_response_write_body_do() TIMEOUT !!!
2021-01-05 18:33:14.670 15274 CRITICAL castellan [-] Unhandled error: OSError: write error

 I'm guessing that it's prematurely terminating the download as the expected length is nil?

[Jim Scadden (jimscadden)]

I've found that comment out the following line in /etc/swift/swift-proxy-server-uwsgi.ini and restarting swift-proxy enabled me to download images again:

rem-header = Content-Length

[Thomas Goirand (thomas-goirand)]

Hi Jim,

Thanks a lot for your finding. I've removed this from the default swift-proxy uwsgi config in the Debian package. I'll trust what you say, and vote for closing this bug then.

Cheers,

Thomas Goirand

[Jim Scadden (jimscadden)]

I've upgraded to the Debian package and can confirm the issue is gone. I'm also happy for this bug to be closed.

[Erno Kuvaja (jokke)]

Thanks Thomas and Jim,

Closed the bug as invalid.