Glance image upload stuck in saving with Swift backend

Bug #1825834 reported by Andrii Yaurov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Incomplete
Medium
Andrii Yaurov

Bug Description

When creating a glance image with --copy-from option, if the network connection between Glance's backend (Swift) drops, the image gets stuck in 'saving' state despite Swift registering a connection timeout.

The bug is observed in MOS 9.x

Steps to reproduce:

root@cic-1:~# glance --os-image-api-version 1 image-create --name testTrusty --disk-format qcow2 --container-format bare --copy-from http://10.80.245.8:8000/sparse-file.img --is-public True --progress
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2019-04-11T17:36:44.000000 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 97b91f48-b553-47da-9e46-dda4a0ea5e27 |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | testTrusty |
| owner | 7b0003268773455db9c64af90fff54c1 |
| protected | False |
| size | 10737418240 |
| status | queued |
| updated_at | 2019-04-11T17:36:44.000000 |
| virtual_size | None |
+------------------+--------------------------------------+
root@cic-1:~#
root@cic-1:~#
root@cic-1:~# glance image-list
+--------------------------------------+------------------+
| ID | Name |
+--------------------------------------+------------------+
| 957fdad7-70d8-4039-bd1d-1dc6485b4a3d | |
| eb37059e-7c56-4c4c-ad52-68d4ec043140 | emarklj_Ubuntu-1 |
| 091d0a0e-c756-49b4-875e-e655a638b9f9 | emhnprp_test |
| 32c02d01-cbf7-49ef-ab44-789bb84bb2e2 | node-image |
| f9c3eead-5baa-4557-a4fa-7ae721bc709b | test123 |
| 97b91f48-b553-47da-9e46-dda4a0ea5e27 | testTrusty |
| a47abfc8-f97f-4211-a720-1c3f70d3163b | TestVM |
+--------------------------------------+------------------+
root@cic-1:~# glance image-show 97b91f48-b553-47da-9e46-dda4a0ea5e27
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2019-04-11T17:36:44Z |
| disk_format | qcow2 |
| id | 97b91f48-b553-47da-9e46-dda4a0ea5e27 |
| min_disk | 0 |
| min_ram | 0 |
| name | testTrusty |
| owner | 7b0003268773455db9c64af90fff54c1 |
| protected | False |
| size | 10737418240 |
| status | saving |
| tags | [] |
| updated_at | 2019-04-11T17:36:44Z |
| virtual_size | None |
| visibility | public |
+------------------+--------------------------------------+

While creating image, remove the connection between cic and image source.

You will see similar log from swift-proxy-server:
ERROR Client read timeout (60s) (txn: tx986fb15e8a5d46f0954b6-005caf7b42) (client_ip: 192.168.13.2)

However, image will still remain in the 'saving' state.

Changed in mos:
assignee: nobody → Oleksiy Molchanov (omolchanov)
Changed in mos:
status: New → In Progress
importance: Undecided → Medium
milestone: none → 9.x-updates
Changed in mos:
milestone: 9.x-updates → 9.2-mu-12
Changed in mos:
milestone: 9.2-mu-12 → 9.2-mu-13
Revision history for this message
chandramouli Chekuri (cchekuri) wrote :

Http is deprecated. And when the http connection is interrupted, swift-proxy-server writes the below error –

swift-proxy-server[26387]: ERROR Client read timeout (60s) (txn: tx97dabc1e443545b4a9cc8-005ca734f3) (client_ip:

And also the swift-object-server writes the http code 408.

swift-object-server[18707]: 10.2.32.1 - - [05/Apr/2019:10:59:59 +0000] "PUT /1/681/AUTH_ae3cc7c210a340f9bb28565ca6c5672b/glance/6bb5e09f-1606-4399-91f0-8ce47e6dfb78-00008" 408 - "PUT http://129.6.32.30:8080/v1/AUTH_ae3cc7c210a340f9bb28565ca6c5672b/glance/6bb5e09f-1606-4399-91f0-8ce47e6dfb78-00008" "tx97dabc1e443545b4a9cc8-005ca734f3" "proxy-server 26387" 60.0068 "-" 18707 0

But in our case we have observed that the swift proxy server reports no error back. They expect that swift-proxy should report http exception as per below code.

Could you please clarify on expected behavior from swift-proxy in such case. Should it report back to glance/swift client/or any other

Revision history for this message
Roman Lubianyi (rlubianyi) wrote :
Download full text (5.2 KiB)

I have checked behavior when the http connection is interrupted in the next environment:
MOS 9.2
glance API v1
packages:
swift - 2.7.1-4~u14.04+mos2
python-swiftclient - 1:3.0.0-2~u14.04+mos1
glance-api - 2:12.0.0-5~u14.04+mos26
python-glance-store - 0.13.1-1~u14.04+mos9
python-glanceclient - 1:2.0.1-3~u14.04+mos2

In my environment when the upload of chunk failed with 408 code[1], glance_store swift driver received ClientException: put_object[2]. After the exception, glance performs cleaning of chunks form the swift storage[3].

[1]
<46>Jun 25 09:01:54 node-6 swift-object-server: 10.109.2.4 - - [25/Jun/2019:09:01:54 +0000] "PUT /1/618/AUTH_16e3902a3c674e37908c18f1dd1cf156/glance/c224d9f7-54a3-4a8b-9210-470cbc8ffb8d-00025" 408 - "PUT http://10.109.1.3:8080/v1/AUTH_16e3902a3c674e37908c18f1dd1cf156/glance/c224d9f7-54a3-4a8b-9210-470cbc8ffb8d-00025" "txff387b2c731d45d2839f0-005d11e2af" "proxy-server 20292" 82.9192 "-" 25816 0

[2]
<147>Jun 25 09:03:48 node-6 glance-api: 2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store [-] Error during chunked upload to backend, deleting stale chunks
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store Traceback (most recent call last):
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 597, in add
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store content_length=content_length)
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1709, in put_object
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store response_dict=response_dict)
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1601, in _retry
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store reset_func(func, *args, **kwargs)
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1689, in _default_reset
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store % (container, obj))
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store ClientException: put_object('glance', 'c224d9f7-54a3-4a8b-9210-470cbc8ffb8d-00025', ...) failure and no ability to reset contents for reupload.
2019-06-25 09:03:48.185 8717 ERROR glance_store._drivers.swift.store
<151>Jun 25 09:03:48 node-6 glance-api: 2019-06-25 09:03:48.202 8717 DEBUG glance_store._drivers.swift.store [-] Deleting chunk c224d9f7-54a3-4a8b-9210-470cbc8ffb8d-00001 _delete_stale_chunks /usr/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py:515

[3]
<151>Jun 25 09:03:48 node-6 glance-api: 2019-06-25 09:03:48.611 8717 DEBUG swiftclient [-] REQ: curl -i http://10.109.1.3:8080/v1/AUTH_16e3902a3c674e37908c18f1dd1cf156/glance/c224d9f7-54a3-4a8b-9210-470cbc8ffb8d-00001 DELETE -H "X-Auth-Token: gAAAAABdEd3Gm5uQ..." http_log /usr/lib/python2.7/dist-packages/swiftclient/client.py:164
<151>Jun 25 09:03:48 node-6 glance-api: 2019...

Read more...

Revision history for this message
Roman Lubianyi (rlubianyi) wrote :

The glance[gl1.log]

Revision history for this message
Roman Lubianyi (rlubianyi) wrote :

The swift[sw1.log]

Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

The issue did not reproduce. Moving to incomplete.

Changed in mos:
status: In Progress → Incomplete
assignee: Oleksiy Molchanov (omolchanov) → Denis Meltsaykin (dmeltsaykin)
assignee: Denis Meltsaykin (dmeltsaykin) → nobody
assignee: nobody → Andrii Yaurov (ayaurov)
Changed in mos:
milestone: 9.2-mu-13 → 9.x-updates
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.