glance_store

Cinder Scaleio: cannot download image

Bug #2086759 reported by Rajat Dhasmana
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
Fix Released
Undecided
Unassigned

Bug Description

When glance is configured to use cinder as a backend and the cinder backend is powerflex/scaleio. we are able to create the image but when downloading the image, it errors out with corrupt hash.

This issue is reproduced by Jean Pierre.

See original description
Rajat Dhasmana (whoami-rajat)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance_store/+/934237

Changed in glance-store:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/934237
Committed: https://opendev.org/openstack/glance_store/commit/55cefedd169b84d713a8ff080d909483ea51c913
Submitter: "Zuul (22348)"
Branch: master

commit 55cefedd169b84d713a8ff080d909483ea51c913
Author: Rajat Dhasmana <email address hidden>
Date: Wed Nov 6 17:59:03 2024 +0530

    Cinder PowerFlex: Fix issue when downloading image

    When using Glance -> Cinder -> PowerFlex configuration, we fail when
    trying to download an image. This regression was introduced with
    2 changes:
    1. Introduce wait mechanism for extended LUN[1]
    2. Generalize the wait mechanism for all operations involving attach[2]

    Originally the wait logic was introduced for create image operation but
    later was generalized to get image operation as well with the
    refactoring done in [2].

    When extending the volume, the seek pointer is reset after the extend
    so this wasn't an issue but when we try to download the image,
    the file pointer stays to the end of file and we end up reading
    nothing eventually failing the operation.

    The reason for using seek and tell method is:
    fstat doesn't work with block devices exported by the scaleio/powerflex
    protocol as seen in the testing done.

    $ os.fstat(device_file.fileno())

    os.stat_result(st_mode=25008, st_ino=818, st_dev=5, st_nlink=1,
    st_uid=42415, st_gid=6, st_size=0, st_atime=1731055942,
    st_mtime=1731055942, st_ctime=1731055944)

    See that st_size equals 0

    Also the linux stat command returns 0 size:

    Symbolic Link:
    $ stat /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d

      File: /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d -> ../../scinib

      Size: 12 Blocks: 0 IO Block: 4096 symbolic link
    ...

    Original File:
    $ stat /dev/scinib

      File: /dev/scinib

      Size: 0 Blocks: 0 IO Block: 4096 block special file
    ...

    To fix this download issue, we get the original position of file,
    seek to the end to get the device size and eventually reset the
    file pointer to the original position.

    [1] https://review.opendev.org/c/openstack/glance_store/+/826290
    [2] https://review.opendev.org/c/openstack/glance_store/+/843103

    Closes-Bug: #2086759
    Change-Id: I713b24a233e33d2908ac2706c9d5fea1c048810c

Changed in glance-store:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/2024.2)

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/glance_store/+/934722

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/2024.2)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/934722
Committed: https://opendev.org/openstack/glance_store/commit/139693146ec89b15cef396561556237d32593e0f
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit 139693146ec89b15cef396561556237d32593e0f
Author: Rajat Dhasmana <email address hidden>
Date: Wed Nov 6 17:59:03 2024 +0530

    Cinder PowerFlex: Fix issue when downloading image

    When using Glance -> Cinder -> PowerFlex configuration, we fail when
    trying to download an image. This regression was introduced with
    2 changes:
    1. Introduce wait mechanism for extended LUN[1]
    2. Generalize the wait mechanism for all operations involving attach[2]

    Originally the wait logic was introduced for create image operation but
    later was generalized to get image operation as well with the
    refactoring done in [2].

    When extending the volume, the seek pointer is reset after the extend
    so this wasn't an issue but when we try to download the image,
    the file pointer stays to the end of file and we end up reading
    nothing eventually failing the operation.

    The reason for using seek and tell method is:
    fstat doesn't work with block devices exported by the scaleio/powerflex
    protocol as seen in the testing done.

    $ os.fstat(device_file.fileno())

    os.stat_result(st_mode=25008, st_ino=818, st_dev=5, st_nlink=1,
    st_uid=42415, st_gid=6, st_size=0, st_atime=1731055942,
    st_mtime=1731055942, st_ctime=1731055944)

    See that st_size equals 0

    Also the linux stat command returns 0 size:

    Symbolic Link:
    $ stat /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d

      File: /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d -> ../../scinib

      Size: 12 Blocks: 0 IO Block: 4096 symbolic link
    ...

    Original File:
    $ stat /dev/scinib

      File: /dev/scinib

      Size: 0 Blocks: 0 IO Block: 4096 block special file
    ...

    To fix this download issue, we get the original position of file,
    seek to the end to get the device size and eventually reset the
    file pointer to the original position.

    [1] https://review.opendev.org/c/openstack/glance_store/+/826290
    [2] https://review.opendev.org/c/openstack/glance_store/+/843103

    Closes-Bug: #2086759
    Change-Id: I713b24a233e33d2908ac2706c9d5fea1c048810c
    (cherry picked from commit 55cefedd169b84d713a8ff080d909483ea51c913)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/glance_store/+/935027

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/935027
Committed: https://opendev.org/openstack/glance_store/commit/8fbdf992296a7dad9fcf5fc7564e59f9c60da034
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 8fbdf992296a7dad9fcf5fc7564e59f9c60da034
Author: Rajat Dhasmana <email address hidden>
Date: Wed Nov 6 17:59:03 2024 +0530

    Cinder PowerFlex: Fix issue when downloading image

    When using Glance -> Cinder -> PowerFlex configuration, we fail when
    trying to download an image. This regression was introduced with
    2 changes:
    1. Introduce wait mechanism for extended LUN[1]
    2. Generalize the wait mechanism for all operations involving attach[2]

    Originally the wait logic was introduced for create image operation but
    later was generalized to get image operation as well with the
    refactoring done in [2].

    When extending the volume, the seek pointer is reset after the extend
    so this wasn't an issue but when we try to download the image,
    the file pointer stays to the end of file and we end up reading
    nothing eventually failing the operation.

    The reason for using seek and tell method is:
    fstat doesn't work with block devices exported by the scaleio/powerflex
    protocol as seen in the testing done.

    $ os.fstat(device_file.fileno())

    os.stat_result(st_mode=25008, st_ino=818, st_dev=5, st_nlink=1,
    st_uid=42415, st_gid=6, st_size=0, st_atime=1731055942,
    st_mtime=1731055942, st_ctime=1731055944)

    See that st_size equals 0

    Also the linux stat command returns 0 size:

    Symbolic Link:
    $ stat /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d

      File: /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d -> ../../scinib

      Size: 12 Blocks: 0 IO Block: 4096 symbolic link
    ...

    Original File:
    $ stat /dev/scinib

      File: /dev/scinib

      Size: 0 Blocks: 0 IO Block: 4096 block special file
    ...

    To fix this download issue, we get the original position of file,
    seek to the end to get the device size and eventually reset the
    file pointer to the original position.

    [1] https://review.opendev.org/c/openstack/glance_store/+/826290
    [2] https://review.opendev.org/c/openstack/glance_store/+/843103

    Closes-Bug: #2086759
    Change-Id: I713b24a233e33d2908ac2706c9d5fea1c048810c
    (cherry picked from commit 55cefedd169b84d713a8ff080d909483ea51c913)
    (cherry picked from commit 139693146ec89b15cef396561556237d32593e0f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/glance_store/+/935042

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/935042
Committed: https://opendev.org/openstack/glance_store/commit/e69d54a6269c8828a8c1b82ef9fbf4751dad5d8d
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit e69d54a6269c8828a8c1b82ef9fbf4751dad5d8d
Author: Rajat Dhasmana <email address hidden>
Date: Wed Nov 6 17:59:03 2024 +0530

    Cinder PowerFlex: Fix issue when downloading image

    When using Glance -> Cinder -> PowerFlex configuration, we fail when
    trying to download an image. This regression was introduced with
    2 changes:
    1. Introduce wait mechanism for extended LUN[1]
    2. Generalize the wait mechanism for all operations involving attach[2]

    Originally the wait logic was introduced for create image operation but
    later was generalized to get image operation as well with the
    refactoring done in [2].

    When extending the volume, the seek pointer is reset after the extend
    so this wasn't an issue but when we try to download the image,
    the file pointer stays to the end of file and we end up reading
    nothing eventually failing the operation.

    The reason for using seek and tell method is:
    fstat doesn't work with block devices exported by the scaleio/powerflex
    protocol as seen in the testing done.

    $ os.fstat(device_file.fileno())

    os.stat_result(st_mode=25008, st_ino=818, st_dev=5, st_nlink=1,
    st_uid=42415, st_gid=6, st_size=0, st_atime=1731055942,
    st_mtime=1731055942, st_ctime=1731055944)

    See that st_size equals 0

    Also the linux stat command returns 0 size:

    Symbolic Link:
    $ stat /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d

      File: /dev/disk/by-id/emc-vol-01f948a4798c2e0f-e062e68a0000001d -> ../../scinib

      Size: 12 Blocks: 0 IO Block: 4096 symbolic link
    ...

    Original File:
    $ stat /dev/scinib

      File: /dev/scinib

      Size: 0 Blocks: 0 IO Block: 4096 block special file
    ...

    To fix this download issue, we get the original position of file,
    seek to the end to get the device size and eventually reset the
    file pointer to the original position.

    [1] https://review.opendev.org/c/openstack/glance_store/+/826290
    [2] https://review.opendev.org/c/openstack/glance_store/+/843103

    Closes-Bug: #2086759
    Change-Id: I713b24a233e33d2908ac2706c9d5fea1c048810c
    (cherry picked from commit 55cefedd169b84d713a8ff080d909483ea51c913)
    (cherry picked from commit 139693146ec89b15cef396561556237d32593e0f)
    (cherry picked from commit 8fbdf992296a7dad9fcf5fc7564e59f9c60da034)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/glance_store/+/935790

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance_store (stable/2023.1)

Change abandoned by "Dr. Jens Harbott <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/glance_store/+/935790
Reason: stable/2023.1 branch of this repo is about to be deleted.
To be able to do that, all open patches need to be abandoned.
Please cherry pick the patch to unmaintained/2023.1 if you want to
further work on this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 4.9.0

This issue was fixed in the openstack/glance_store 4.9.0 Epoxy release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 4.8.2

This issue was fixed in the openstack/glance_store 4.8.2 Dalmatian release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 4.6.2

This issue was fixed in the openstack/glance_store 4.6.2 Bobcat release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 4.7.1

This issue was fixed in the openstack/glance_store 4.7.1 Caracal release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.