OpenStack Compute (nova)

UnboundLocalError: local variable 'instance_uuid' referenced before assignment

Bug #1940812 reported by Balazs Gibizer
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Balazs Gibizer
Xena
Fix Released
Undecided
Unassigned

Bug Description

there is an unbound variable in [1] causing:

Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi [None req-b45811ea-c6f9-4422-8cf1-8970273587bc tempest-DeleteServersTestJSON-1241246976 tempest-DeleteServersTestJSON-1241246976-project] Unexpected exception in API method: UnboundLocalError: local variable 'instance_uuid' referenced before assignment
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/objects/instance.py", line 653, in destroy
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi db_inst = db.instance_destroy(self._context, self.uuid,
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi ectxt.value = e.inner_exc
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi self.force_reraise()
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi raise self.value
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi return f(context, *args, **kwargs)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/main/api.py", line 1294, in instance_destroy
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi raise exception.ConstraintNotMet()
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi nova.exception.ConstraintNotMet: Constraint not met.
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2248, in _delete
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi if self._delete_while_booting(context, instance):
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2188, in _delete_while_booting
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi instance.destroy()
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi return fn(self, *args, **kwargs)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/objects/instance.py", line 658, in destroy
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi raise exception.ObjectActionError(action='destroy',
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi nova.exception.ObjectActionError: Object action destroy failed because: host changed
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 658, in wrapped
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 1072, in delete
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi self._delete(req.environ['nova.context'], req, id)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 884, in _delete
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi self.compute_api.delete(context, instance)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 228, in inner
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 155, in inner
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2575, in delete
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi self._delete_instance(context, instance)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2566, in _delete_instance
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi self._delete(context, instance, 'delete', self._do_delete,
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2290, in _delete
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi self._local_delete_cleanup(context, instance_uuid)
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi UnboundLocalError: local variable 'instance_uuid' referenced before assignment
Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: ERROR nova.api.openstack.wsgi
Aug 20 14:57:42.707787 ubuntu-focal-inap-mtl01-0026012484 <email address hidden>[111846]: INFO nova.api.openstack.wsgi [None req-b45811ea-c6f9-4422-8cf1-8970273587bc tempest-DeleteServersTestJSON-1241246976 tempest-DeleteServersTestJSON-1241246976-project] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

[1] https://github.com/openstack/nova/blob/10b1dc84f47a71061340f8e0ae0fe32dca44061a/nova/compute/api.py#L2290
[2] https://61871398d16bb191f50f-b8e5e06ecccd85e574a730a33d1fddb0.ssl.cf2.rackcdn.com/745605/6/gate/nova-ceph-multistore/805eaaa/controller/logs/screen-n-api.txt

Balazs Gibizer (balazs-gibizer)
Changed in nova:
assignee: nobody → Balazs Gibizer (balazs-gibizer)
tags: added: gate-failure
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

It seems that the previous fix https://review.opendev.org/c/openstack/nova/+/775308 does not corrected every possible cases.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/805605

Changed in nova:
status: New → In Progress
Balazs Gibizer (balazs-gibizer)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/805605
Committed: https://opendev.org/openstack/nova/commit/14e43f385e6d243b6efd11a777d082e63b66367c
Submitter: "Zuul (22348)"
Branch: master

commit 14e43f385e6d243b6efd11a777d082e63b66367c
Author: Balazs Gibizer <email address hidden>
Date: Mon Aug 23 10:56:58 2021 +0200

    Avoid unbound instance_uuid var during delete

    The patch I03cf285ad83e09d88cdb702a88dfed53c01610f8 fixed most of the
    possible cases for this to happen but missed one. An early enough
    exception during _delete() can cause that the instance_uuid never gets
    defined but then we try to use it during the finally block. This patch
    moves the saving of the instance_uuid to the top of the try block to
    avoid the issue.

    Change-Id: Ib3073d7f595c8927532b7c49fc7e5ffe80d508b9
    Closes-Bug: #1940812
    Related-Bug: #1914777

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/816488

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/816488
Committed: https://opendev.org/openstack/nova/commit/00cba396134846dbbcf68cc9cf08e80618322bbe
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 00cba396134846dbbcf68cc9cf08e80618322bbe
Author: Balazs Gibizer <email address hidden>
Date: Mon Aug 23 10:56:58 2021 +0200

    Avoid unbound instance_uuid var during delete

    The patch I03cf285ad83e09d88cdb702a88dfed53c01610f8 fixed most of the
    possible cases for this to happen but missed one. An early enough
    exception during _delete() can cause that the instance_uuid never gets
    defined but then we try to use it during the finally block. This patch
    moves the saving of the instance_uuid to the top of the try block to
    avoid the issue.

    Change-Id: Ib3073d7f595c8927532b7c49fc7e5ffe80d508b9
    Closes-Bug: #1940812
    Related-Bug: #1914777
    (cherry picked from commit 14e43f385e6d243b6efd11a777d082e63b66367c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/828839

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.1.0

This issue was fixed in the openstack/nova 24.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 25.0.0.0rc1

This issue was fixed in the openstack/nova 25.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/828839
Committed: https://opendev.org/openstack/nova/commit/fc20dc2ea23d511a0f741901d7f03135bb269165
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit fc20dc2ea23d511a0f741901d7f03135bb269165
Author: Balazs Gibizer <email address hidden>
Date: Mon Aug 23 10:56:58 2021 +0200

    Avoid unbound instance_uuid var during delete

    The patch I03cf285ad83e09d88cdb702a88dfed53c01610f8 fixed most of the
    possible cases for this to happen but missed one. An early enough
    exception during _delete() can cause that the instance_uuid never gets
    defined but then we try to use it during the finally block. This patch
    moves the saving of the instance_uuid to the top of the try block to
    avoid the issue.

    Change-Id: Ib3073d7f595c8927532b7c49fc7e5ffe80d508b9
    Closes-Bug: #1940812
    Related-Bug: #1914777
    (cherry picked from commit 14e43f385e6d243b6efd11a777d082e63b66367c)
    (cherry picked from commit 00cba396134846dbbcf68cc9cf08e80618322bbe)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.2.2

This issue was fixed in the openstack/nova 23.2.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.