Nova-compute wouldn't retry image download when gets "Corrupt image download" error

Bug #1950657 reported by Stanislav Dmitriev
34
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Stanislav Dmitriev
Victoria
In Progress
Undecided
Unassigned
Wallaby
In Progress
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned

Bug Description

Nova-compute wouldn't retry image download when gets "Corrupt image download" error from glanceclient.

There is a configuration option num_retries (equal 3 by default) in glance section of nova-config file, so nova-compute supposed to retry image download if it failed, but it wouldn't work in case of next exception:

2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in image_chunks:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in self.iterable:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] (computed, hash_value))
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]

It wouldn't work because IOError exception is not in retry_excs list:
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179

        retry_excs = (glanceclient.exc.ServiceUnavailable,
                glanceclient.exc.InvalidEndpoint,
                glanceclient.exc.CommunicationError)

so try-except block doesn't catch and download retry never happens

Changed in nova:
status: New → Confirmed
summary: Nova-compute wouldn't retry image download when gets "Corrupt image
- download" error
+ download" erro
summary: Nova-compute wouldn't retry image download when gets "Corrupt image
- download" erro
+ download" error
Changed in nova:
assignee: nobody → S.A.Dmitriev (sdmitriev1)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/818503

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/818503
Committed: https://opendev.org/openstack/nova/commit/ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Submitter: "Zuul (22348)"
Branch: master

commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Author: sdmitriev1 <email address hidden>
Date: Thu Nov 18 22:05:05 2021 -0500

    Retry image download if it's corrupted

    Adding IOError in list of catching exceptions in order to
    fix behavior when nova-compute wouldn't retry image download
    when got "Corrupt image download" error from glanceclient
    and had num_retries config option set.

    Closes-Bug: #1950657
    Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/819179

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/819180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/819181

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/819179
Committed: https://opendev.org/openstack/nova/commit/b44ec0dc4927046d15525af78ab9d534cea9ce69
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b44ec0dc4927046d15525af78ab9d534cea9ce69
Author: sdmitriev1 <email address hidden>
Date: Thu Nov 18 22:05:05 2021 -0500

    Retry image download if it's corrupted

    Adding IOError in list of catching exceptions in order to
    fix behavior when nova-compute wouldn't retry image download
    when got "Corrupt image download" error from glanceclient
    and had num_retries config option set.

    Closes-Bug: #1950657
    Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0
    (cherry picked from commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.1.0

This issue was fixed in the openstack/nova 24.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 25.0.0.0rc1

This issue was fixed in the openstack/nova 25.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers