Glance image upload stuck in saving with Swift backend
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Incomplete
|
Medium
|
Andrii Yaurov |
Bug Description
When creating a glance image with --copy-from option, if the network connection between Glance's backend (Swift) drops, the image gets stuck in 'saving' state despite Swift registering a connection timeout.
The bug is observed in MOS 9.x
Steps to reproduce:
root@cic-1:~# glance --os-image-
+------
| Property | Value |
+------
| checksum | None |
| container_format | bare |
| created_at | 2019-04-
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 97b91f48-
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | testTrusty |
| owner | 7b0003268773455
| protected | False |
| size | 10737418240 |
| status | queued |
| updated_at | 2019-04-
| virtual_size | None |
+------
root@cic-1:~#
root@cic-1:~#
root@cic-1:~# glance image-list
+------
| ID | Name |
+------
| 957fdad7-
| eb37059e-
| 091d0a0e-
| 32c02d01-
| f9c3eead-
| 97b91f48-
| a47abfc8-
+------
root@cic-1:~# glance image-show 97b91f48-
+------
| Property | Value |
+------
| checksum | None |
| container_format | bare |
| created_at | 2019-04-
| disk_format | qcow2 |
| id | 97b91f48-
| min_disk | 0 |
| min_ram | 0 |
| name | testTrusty |
| owner | 7b0003268773455
| protected | False |
| size | 10737418240 |
| status | saving |
| tags | [] |
| updated_at | 2019-04-
| virtual_size | None |
| visibility | public |
+------
While creating image, remove the connection between cic and image source.
You will see similar log from swift-proxy-server:
ERROR Client read timeout (60s) (txn: tx986fb15e8a5d4
However, image will still remain in the 'saving' state.
Changed in mos: | |
assignee: | nobody → Oleksiy Molchanov (omolchanov) |
Changed in mos: | |
status: | New → In Progress |
importance: | Undecided → Medium |
milestone: | none → 9.x-updates |
Changed in mos: | |
milestone: | 9.x-updates → 9.2-mu-12 |
Changed in mos: | |
milestone: | 9.2-mu-12 → 9.2-mu-13 |
Changed in mos: | |
milestone: | 9.2-mu-13 → 9.x-updates |
Http is deprecated. And when the http connection is interrupted, swift-proxy-server writes the below error –
swift-proxy- server[ 26387]: ERROR Client read timeout (60s) (txn: tx97dabc1e44354 5b4a9cc8- 005ca734f3) (client_ip:
And also the swift-object-server writes the http code 408.
swift-object- server[ 18707]: 10.2.32.1 - - [05/Apr/ 2019:10: 59:59 +0000] "PUT /1/681/ AUTH_ae3cc7c210 a340f9bb28565ca 6c5672b/ glance/ 6bb5e09f- 1606-4399- 91f0-8ce47e6dfb 78-00008" 408 - "PUT http:// 129.6.32. 30:8080/ v1/AUTH_ ae3cc7c210a340f 9bb28565ca6c567 2b/glance/ 6bb5e09f- 1606-4399- 91f0-8ce47e6dfb 78-00008" "tx97dabc1e4435 45b4a9cc8- 005ca734f3" "proxy-server 26387" 60.0068 "-" 18707 0
But in our case we have observed that the swift proxy server reports no error back. They expect that swift-proxy should report http exception as per below code.
Could you please clarify on expected behavior from swift-proxy in such case. Should it report back to glance/swift client/or any other