Nova-compute wouldn't retry image download when gets "Corrupt image download" error

Bug #1950657 reported by S.A.Dmitriev
34
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
S.A.Dmitriev

Bug Description

Nova-compute wouldn't retry image download when gets "Corrupt image download" error from glanceclient.

There is a configuration option num_retries (equal 3 by default) in glance section of nova-config file, so nova-compute supposed to retry image download if it failed, but it wouldn't work in case of next exception:

2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in image_chunks:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in self.iterable:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] (computed, hash_value))
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]

It wouldn't work because IOError exception is not in retry_excs list:
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179

        retry_excs = (glanceclient.exc.ServiceUnavailable,
                glanceclient.exc.InvalidEndpoint,
                glanceclient.exc.CommunicationError)

so try-except block doesn't catch and download retry never happens

Changed in nova:
status: New → Confirmed
summary: Nova-compute wouldn't retry image download when gets "Corrupt image
- download" error
+ download" erro
summary: Nova-compute wouldn't retry image download when gets "Corrupt image
- download" erro
+ download" error
Changed in nova:
assignee: nobody → S.A.Dmitriev (sdmitriev1)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/818503

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/818503
Committed: https://opendev.org/openstack/nova/commit/ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Submitter: "Zuul (22348)"
Branch: master

commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Author: sdmitriev1 <email address hidden>
Date: Thu Nov 18 22:05:05 2021 -0500

    Retry image download if it's corrupted

    Adding IOError in list of catching exceptions in order to
    fix behavior when nova-compute wouldn't retry image download
    when got "Corrupt image download" error from glanceclient
    and had num_retries config option set.

    Closes-Bug: #1950657
    Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/819179

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/819180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/819181

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers