ceph backend reporting meaningless error when no space left

Bug #1808456 reported by Liang Fang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Invalid
Undecided
Unassigned
glance_store
Fix Released
Low
Unassigned

Bug Description

When uploading image, but there's no space left in ceph(rbd) backend, client(such as glanceclient) will receive a meaningless error:
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

steps to reproduce:
-------------------------------------------------
1.Prepare ceph backend for glance, make the free space small enough, e.g. 10MB.
To be simple, you also can modify ceph's code(function resize), to let it raise errno.ENOSPC. I did this way.
2.uploading image: glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c

logs
-------------------------------------------------
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
> /usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=============================>] 100%
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T02:08:36Z |
| disk_format | raw |
| id | 8c2e48f0-aafc-4744-95b6-fe0b6fbfe975 |
| min_disk | 0 |
| min_ram | 0 |
| name | img2-ceph |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | 3242a198f7044fcd9b756866ec296391 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2018-12-14T02:08:36Z |
| virtual_size | Not available |
| visibility | public |
+------------------+--------------------------------------+
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

expected
-------------------------------------------------
The correct message should be something related "Storage Full", rbd.py should raise glance_store.StorageFull, and this exception will be caught by notifier.py.

Some code snippet in notifier.py:
        except glance_store.StorageFull as e:
            msg = (_("Image storage media is full: %s") %
                   encodeutils.exception_to_unicode(e))
            _send_notification(notify_error, 'image.upload', msg)
            raise webob.exc.HTTPRequestEntityTooLarge(explanation=msg)

After doing this, the expected behavior will be:
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
> /usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=============================>] 100%
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T01:41:36Z |
| disk_format | raw |
| id | 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 |
| min_disk | 0 |
| min_ram | 0 |
| name | img2-ceph |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | 3242a198f7044fcd9b756866ec296391 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2018-12-14T01:41:36Z |
| virtual_size | Not available |
| visibility | public |
+------------------+--------------------------------------+
413 Request Entity Too Large: Image storage media is full: Failed to store image 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 insufficient space available (HTTP 413)

Changed in glance:
status: New → Invalid
Changed in glance-store:
importance: Undecided → Low
Liang Fang (liangfang)
Changed in glance-store:
status: New → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 0.28.0

This issue was fixed in the openstack/glance_store 0.28.0 release.

Changed in glance-store:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.