ceph backend reporting meaningless error when no space left

Bug #1808456 reported by Liang Fang on 2018-12-14
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Undecided
Unassigned
glance_store
Low
Unassigned

Bug Description

When uploading image, but there's no space left in ceph(rbd) backend, client(such as glanceclient) will receive a meaningless error:
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

steps to reproduce:
-------------------------------------------------
1.Prepare ceph backend for glance, make the free space small enough, e.g. 10MB.
To be simple, you also can modify ceph's code(function resize), to let it raise errno.ENOSPC. I did this way.
2.uploading image: glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c

logs
-------------------------------------------------
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
> /usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=============================>] 100%
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T02:08:36Z |
| disk_format | raw |
| id | 8c2e48f0-aafc-4744-95b6-fe0b6fbfe975 |
| min_disk | 0 |
| min_ram | 0 |
| name | img2-ceph |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | 3242a198f7044fcd9b756866ec296391 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2018-12-14T02:08:36Z |
| virtual_size | Not available |
| visibility | public |
+------------------+--------------------------------------+
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

expected
-------------------------------------------------
The correct message should be something related "Storage Full", rbd.py should raise glance_store.StorageFull, and this exception will be caught by notifier.py.

Some code snippet in notifier.py:
        except glance_store.StorageFull as e:
            msg = (_("Image storage media is full: %s") %
                   encodeutils.exception_to_unicode(e))
            _send_notification(notify_error, 'image.upload', msg)
            raise webob.exc.HTTPRequestEntityTooLarge(explanation=msg)

After doing this, the expected behavior will be:
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
> /usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=============================>] 100%
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T01:41:36Z |
| disk_format | raw |
| id | 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 |
| min_disk | 0 |
| min_ram | 0 |
| name | img2-ceph |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | 3242a198f7044fcd9b756866ec296391 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2018-12-14T01:41:36Z |
| virtual_size | Not available |
| visibility | public |
+------------------+--------------------------------------+
413 Request Entity Too Large: Image storage media is full: Failed to store image 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 insufficient space available (HTTP 413)

Changed in glance:
status: New → Invalid
Changed in glance-store:
importance: Undecided → Low
Liang Fang (liangfang) on 2019-02-20
Changed in glance-store:
status: New → Fix Committed

This issue was fixed in the openstack/glance_store 0.28.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers