OpenStack Compute (nova)

Wrong exception is expected to retry volume detachment API calls

  1. Series ussuri
  2. Bug #1944043
Bug #1944043 reported by Takashi Kajinami
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Takashi Kajinami
Queens
New
Undecided
Unassigned
Rocky
New
Undecided
Unassigned
Stein
New
Undecided
Unassigned
Train
In Progress
Undecided
Unassigned
Ussuri
In Progress
Undecided
Unassigned
Victoria
Fix Released
Undecided
Unassigned
Wallaby
Fix Released
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned
Yoga
Fix Released
Undecided
Unassigned

Bug Description

Description
===========
The following change introduced the logic to retry cinder API calls to detach volumes.

https://review.opendev.org/c/openstack/nova/+/669674

The logic detects the InternalServerError class from cindreclient.apiclient.exceptions.

However this is wrong and these API calls raises the ClientException class from cinderclient.exceptions instead.

Steps to reproduce
==================
N/A

Actual result
=============
N/A

Environment
===========
N/A

Logs & Configs
==============
N/A

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

The following traceback was captured in stable/train base deployment.
In this case 504 is returned from cinder API instead of 503 but the traceback shows that cinderclient.exceptions.ClientException is raised when API call fails.

~~~
2021-09-09 12:11:14.000 7 INFO nova.compute.manager [req-d4ea1834-53a1-4786-8c9a-f495307ec9d3 99f5628a6a55449fae14d97881a054ce e722c8ab654a440e8d3cdc5ef592f21b - default default] [instance: 3a9b54b5-35b7-4427-bc64-f88b0d8949b3] Detaching volume 35ff73cd-ebe2-40b3-8be6-f91aca202aa8
...
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server [req-d4ea1834-53a1-4786-8c9a-f495307ec9d3 99f5628a6a55449fae14d97881a054ce e722c8ab654a440e8d3cdc5ef592f21b - default default] Exception during message handling: cinderclient.excep
tions.ClientException: Gateway Timeout (HTTP 504)
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
...
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 6237, in _detach_volume
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 486, in detach
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server attachment_id, destroy_bdm)
...
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 223, in delete
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'DELETE', **kwargs)
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 205, in _cs_request
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 191, in request
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Timeout (HTTP 504)
2021-09-09 12:12:14.512 7 ERROR oslo_messaging.rpc.server
~~~

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/809934

Changed in nova:
status: New → In Progress
Takashi Kajinami (kajinamit)
summary: - Wrong exception type is expected to retry volume detachment API calls
+ Wrong exception is expected to retry volume detachment API calls
Takashi Kajinami (kajinamit)
Changed in nova:
assignee: nobody → Takashi Kajinami (kajinamit)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/809934
Committed: https://opendev.org/openstack/nova/commit/f6206269b71d9bcaf65aa0313c21cc6b75a54fb3
Submitter: "Zuul (22348)"
Branch: master

commit f6206269b71d9bcaf65aa0313c21cc6b75a54fb3
Author: Takashi Kajinami <email address hidden>
Date: Sun Sep 19 00:06:56 2021 +0900

    Fix the wrong exception used to retry detach API calls

    When cinderclient receives an error response from API, it raises one
    of the exceptions from the cinderclient.exceptions module instead of
    the cinderclient.apiclient.exceptions module.

    This change fixes the wrong exception used to detect 500 error from
    cinder API, so that API calls to detach volumes are properly retried.

    Closes-Bug: #1944043
    Change-Id: I741cb6b29a67da8c60708c6251c441d778ca74d0

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/829049

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 25.0.0.0rc1

This issue was fixed in the openstack/nova 25.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/866084

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/866086

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/866088

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/866090

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/829049
Committed: https://opendev.org/openstack/nova/commit/513241a7e4f3f3246c75e66a910db6c864c3981e
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 513241a7e4f3f3246c75e66a910db6c864c3981e
Author: Takashi Kajinami <email address hidden>
Date: Sun Sep 19 00:06:56 2021 +0900

    Fix the wrong exception used to retry detach API calls

    When cinderclient receives an error response from API, it raises one
    of the exceptions from the cinderclient.exceptions module instead of
    the cinderclient.apiclient.exceptions module.

    This change fixes the wrong exception used to detect 500 error from
    cinder API, so that API calls to detach volumes are properly retried.

    Closes-Bug: #1944043
    Change-Id: I741cb6b29a67da8c60708c6251c441d778ca74d0
    (cherry picked from commit f6206269b71d9bcaf65aa0313c21cc6b75a54fb3)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/866084
Committed: https://opendev.org/openstack/nova/commit/7168314dd125fca8d1cecac035477ea0314ab828
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7168314dd125fca8d1cecac035477ea0314ab828
Author: Takashi Kajinami <email address hidden>
Date: Sun Sep 19 00:06:56 2021 +0900

    Fix the wrong exception used to retry detach API calls

    When cinderclient receives an error response from API, it raises one
    of the exceptions from the cinderclient.exceptions module instead of
    the cinderclient.apiclient.exceptions module.

    This change fixes the wrong exception used to detect 500 error from
    cinder API, so that API calls to detach volumes are properly retried.

    Closes-Bug: #1944043
    Change-Id: I741cb6b29a67da8c60708c6251c441d778ca74d0
    (cherry picked from commit f6206269b71d9bcaf65aa0313c21cc6b75a54fb3)
    (cherry picked from commit 513241a7e4f3f3246c75e66a910db6c864c3981e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.2.0

This issue was fixed in the openstack/nova 24.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/nova/+/866086
Committed: https://opendev.org/openstack/nova/commit/ba3a6f385512c352374425972791084cb54ab7ee
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit ba3a6f385512c352374425972791084cb54ab7ee
Author: Takashi Kajinami <email address hidden>
Date: Sun Sep 19 00:06:56 2021 +0900

    Fix the wrong exception used to retry detach API calls

    When cinderclient receives an error response from API, it raises one
    of the exceptions from the cinderclient.exceptions module instead of
    the cinderclient.apiclient.exceptions module.

    This change fixes the wrong exception used to detect 500 error from
    cinder API, so that API calls to detach volumes are properly retried.

    Closes-Bug: #1944043
    Change-Id: I741cb6b29a67da8c60708c6251c441d778ca74d0
    (cherry picked from commit f6206269b71d9bcaf65aa0313c21cc6b75a54fb3)
    (cherry picked from commit 513241a7e4f3f3246c75e66a910db6c864c3981e)
    (cherry picked from commit 7168314dd125fca8d1cecac035477ea0314ab828)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/train)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/866090
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/ussuri)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/866088
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova victoria-eom

This issue was fixed in the openstack/nova victoria-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova wallaby-eom

This issue was fixed in the openstack/nova wallaby-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.