OpenStack Compute (nova)

Nova-compute wouldn't retry image download when gets "Corrupt image download" error

Bug #1950657 reported by Stanislav Dmitriev
40
This bug affects 8 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Stanislav Dmitriev
Victoria
In Progress
Undecided
Unassigned
Wallaby
In Progress
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned

Bug Description

Nova-compute wouldn't retry image download when gets "Corrupt image download" error from glanceclient.

There is a configuration option num_retries (equal 3 by default) in glance section of nova-config file, so nova-compute supposed to retry image download if it failed, but it wouldn't work in case of next exception:

2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in image_chunks:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in self.iterable:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] (computed, hash_value))
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]

It wouldn't work because IOError exception is not in retry_excs list:
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179

        retry_excs = (glanceclient.exc.ServiceUnavailable,
                glanceclient.exc.InvalidEndpoint,
                glanceclient.exc.CommunicationError)

so try-except block doesn't catch and download retry never happens

Martin Chlumsky (martin-chlumsky)
Changed in nova:
status: New → Confirmed
Benjamin FÜHRMANN (fuhrmannb)
summary: Nova-compute wouldn't retry image download when gets "Corrupt image
- download" error
+ download" erro
summary: Nova-compute wouldn't retry image download when gets "Corrupt image
- download" erro
+ download" error
Stanislav Dmitriev (sdmitriev1)
Changed in nova:
assignee: nobody → S.A.Dmitriev (sdmitriev1)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/818503

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/818503
Committed: https://opendev.org/openstack/nova/commit/ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Submitter: "Zuul (22348)"
Branch: master

commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Author: sdmitriev1 <email address hidden>
Date: Thu Nov 18 22:05:05 2021 -0500

    Retry image download if it's corrupted

    Adding IOError in list of catching exceptions in order to
    fix behavior when nova-compute wouldn't retry image download
    when got "Corrupt image download" error from glanceclient
    and had num_retries config option set.

    Closes-Bug: #1950657
    Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/819179

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/819180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/819181

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/819179
Committed: https://opendev.org/openstack/nova/commit/b44ec0dc4927046d15525af78ab9d534cea9ce69
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b44ec0dc4927046d15525af78ab9d534cea9ce69
Author: sdmitriev1 <email address hidden>
Date: Thu Nov 18 22:05:05 2021 -0500

    Retry image download if it's corrupted

    Adding IOError in list of catching exceptions in order to
    fix behavior when nova-compute wouldn't retry image download
    when got "Corrupt image download" error from glanceclient
    and had num_retries config option set.

    Closes-Bug: #1950657
    Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0
    (cherry picked from commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.1.0

This issue was fixed in the openstack/nova 24.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 25.0.0.0rc1

This issue was fixed in the openstack/nova 25.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/victoria)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/819181
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/wallaby)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/819180
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

Revision history for this message
Yi Song (songyi2k) wrote :

Hi,

I met similar error, but with different exception: BrokenPipeError
Should it also be added to retry_excs list? Or, may retry for any exceptions?

2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 394, in download
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] image_chunks, data, dst_path)
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 459, in _verify_and_write
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] {'path': dst_path, 'exception': ex})
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] self.force_reraise()
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] six.reraise(self.type_, self.value, self.tb)
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] raise value
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 440, in _verify_and_write
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] for chunk in image_chunks:
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/glanceclient/common/utils.py", line 519, in __iter__
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] for chunk in self.iterable:
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] File "/usr/lib/python3.6/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] (computed, hash_value))
2024-03-08 01:40:55.307 7 ERROR nova.compute.manager [instance: 76060d9b-bcee-4906-9b7b-10e020e05828] BrokenPipeError: [Errno 32] Corrupt image download. Hash was 22350f12068085c9d6f03a33b2eed895707d77ef70661750d3310d1347f236771c4403974a104ad4cd3657a4c3e76330c8e11032803106278437d6f31869b584 expected e2b48e14b0aebef3bcc92c1697a6948f4fbcd62cfaa82d193d98d1b6614b8adefbb6bc3f13c11218da34f15e21412694c6cc59ae2a637ba1cfe5660a0cfa58ba

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.