Glance image create fails with No space left on device error

[Rajat Dhasmana]

When configured to use cinder as glance backend, the image create operation fails with the following error.

91d - a5e795241efa48e882f7feb6748722d2 a5e795241efa48e882f7feb6748722d2] Exception while accessing to cinder volume 6bb6aa01-1ff6-4ae3-853b-4378630d2f07.: IOError: [Errno 28] No space left on device 2021-12-01 18:23:06.228 34 ERROR glance_store._drivers.cinder Traceback (most recent call last): 2021-12-01 18:23:06.228 34 ERROR glance_store._drivers.cinder File "/usr/lib/python2.7/site-packages/glance_store/_drivers/cinder.py", line 583, in _open_cinder_volume 2021-12-01 18:23:06.228 34 ERROR glance_store._drivers.cinder yield f 2021-12-01 18:23:06.228 34 ERROR glance_store._drivers.cinder File "/usr/lib/python2.7/site-packages/glance_store/_drivers/cinder.py", line 765, in add 2021-12-01 18:23:06.228 34 ERROR glance_store._drivers.cinder f.write(buf) 2021-12-01 18:23:06.228 34 ERROR glance_store._drivers.cinder IOError: [Errno 28] No space left on device

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/826290
Committed: https://opendev.org/openstack/glance_store/commit/f3433ed1a5176bff85b3fe04dba2d4c76618a299
Submitter: "Zuul (22348)"
Branch: master

commit f3433ed1a5176bff85b3fe04dba2d4c76618a299
Author: whoami-rajat <email address hidden>
Date: Tue Jan 25 19:24:30 2022 +0530

    Cinder store: Wait for device resize

    When we have an image with size > 1 GB, we follow the following steps
    to accomodate the image:
    1) Detach the volume
    2) extend the volume
    3) Attach the volume
    4) Open the volume device as a file and resume writing the image

    Sometimes due to several reasons (mostly network related), the
    size of the device file could mismatch with the actual volume size
    (or the backend LUN size). This can happen if the extend was performed
    (i.e. the control path) but it takes the time to reflect that into
    the mapped device (i.e. the data path). This mismatch can cause the
    issue "IOError: [Errno 28] No space left on device".
    To avoid this scenario, we check if the device size is less than the
    volume size, we wait for the extended LUN to show up in mapped device
    and then continue the image writing operation.

    Closes-Bug: #1959913

    Change-Id: I206580f6be615ebc5e15b546b9c23728d4116a5d

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 3.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/glance_store/+/829705

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/829705
Committed: https://opendev.org/openstack/glance_store/commit/ea5139be9a76ca8c2e60d12d8dd776bff39c4579
Submitter: "Zuul (22348)"
Branch: stable/xena

commit ea5139be9a76ca8c2e60d12d8dd776bff39c4579
Author: whoami-rajat <email address hidden>
Date: Tue Jan 25 19:24:30 2022 +0530

    Cinder store: Wait for device resize

    When we have an image with size > 1 GB, we follow the following steps
    to accomodate the image:
    1) Detach the volume
    2) extend the volume
    3) Attach the volume
    4) Open the volume device as a file and resume writing the image

    Sometimes due to several reasons (mostly network related), the
    size of the device file could mismatch with the actual volume size
    (or the backend LUN size). This can happen if the extend was performed
    (i.e. the control path) but it takes the time to reflect that into
    the mapped device (i.e. the data path). This mismatch can cause the
    issue "IOError: [Errno 28] No space left on device".
    To avoid this scenario, we check if the device size is less than the
    volume size, we wait for the extended LUN to show up in mapped device
    and then continue the image writing operation.

    Closes-Bug: #1959913

    Change-Id: I206580f6be615ebc5e15b546b9c23728d4116a5d
    (cherry picked from commit f3433ed1a5176bff85b3fe04dba2d4c76618a299)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/glance_store/+/829771

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/829771
Committed: https://opendev.org/openstack/glance_store/commit/5f1cee6fa90268b17c7b6f842854a3b27e44994c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5f1cee6fa90268b17c7b6f842854a3b27e44994c
Author: whoami-rajat <email address hidden>
Date: Tue Jan 25 19:24:30 2022 +0530

    Cinder store: Wait for device resize

    When we have an image with size > 1 GB, we follow the following steps
    to accomodate the image:
    1) Detach the volume
    2) extend the volume
    3) Attach the volume
    4) Open the volume device as a file and resume writing the image

    Sometimes due to several reasons (mostly network related), the
    size of the device file could mismatch with the actual volume size
    (or the backend LUN size). This can happen if the extend was performed
    (i.e. the control path) but it takes the time to reflect that into
    the mapped device (i.e. the data path). This mismatch can cause the
    issue "IOError: [Errno 28] No space left on device".
    To avoid this scenario, we check if the device size is less than the
    volume size, we wait for the extended LUN to show up in mapped device
    and then continue the image writing operation.

    Closes-Bug: #1959913

    Change-Id: I206580f6be615ebc5e15b546b9c23728d4116a5d
    (cherry picked from commit f3433ed1a5176bff85b3fe04dba2d4c76618a299)
    (cherry picked from commit ea5139be9a76ca8c2e60d12d8dd776bff39c4579)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 2.7.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 2.5.1 release.