OpenStack Compute (nova)

Snapshots fail with traceback from API

Bug #1915400 reported by Andrew Bonney
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Balazs Gibizer
Victoria
New
Undecided
Unassigned
Wallaby
In Progress
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned
Yoga
Fix Released
Undecided
Unassigned

Bug Description

Description
===========
Having upgraded three OpenStack deployments to Victoria, we have noticed that snapshots are now failing to be created. When a user attempts this via Horizon they receive an error such as:

Error: Unable to create snapshot. Details

Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'AttributeError'> (HTTP 500) (Request-ID: req-7e4b0824-9884-4d50-b8ee-5184e245a11d)

It appears that this can be worked around by adding 'from eventlet import tpool' and adjusting the 'eventlet.tpool.execute' call to match in nova/utils.py

Steps to reproduce
==================
Via Horizon on a Victoria deployment:
- Create an instance
- Suspend the instance
- Create a snapshot

Expected result
===============
A snapshot should have been created

Actual result
=============
Error message as above

Environment
===========
1. Exact version of OpenStack you are running.

Git hash cb4963ba8c4babb7e460d71daf91099fc6da1dc6

2. Which hypervisor did you use?

Libvirt+KVM

2. Which storage type did you use?

Ceph Octopus

3. Which networking type did you use?

Neutron with Linux Bridge

Logs & Configs
==============
The full traceback from the API is as follows:

Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] Unexpected exception in API method: AttributeError: module 'eventlet' has no attribute 'tpool'
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 1244, in _action_create_image
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi metadata)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 156, in inner
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 3374, in snapshot_volume_backed
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return self.image_api.create(context, image_meta)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 1224, in create
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return session.create(context, image_info, data=data)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 555, in create
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi sharing_member_id=sharing_member_id)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 670, in _create_v2
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi image = self._upload_data(context, image_id, data)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 591, in _upload_data
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi args=(image_id, data))
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/utils.py", line 694, in tpool_execute
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi eventlet.tpool.execute(func, *args, **kwargs)
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi AttributeError: module 'eventlet' has no attribute 'tpool'
                                                                         2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi
Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.079 26063 INFO nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
                                                                         <class 'AttributeError'>

Revision history for this message
Lee Yarwood (lyarwood) wrote :

Which version of eventlet is being used in the nova-22.0.0 venv?

I see it's currently pinned to 0.30.1 in openstack/requirements :

$ git grep eventlet upper-constraints.txt
upper-constraints.txt:eventlet===0.30.1

Lee Yarwood (lyarwood)
Changed in nova:
status: New → Incomplete
Revision history for this message
Andrew Bonney (andrewbonney) wrote :

It's 0.26.1. Apologies, should have included that before.

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

Eventually it's pinned to 0.26.1 for Victoria https://opendev.org/openstack/requirements/src/branch/stable/victoria/upper-constraints.txt#L150

Revision history for this message
Andrew Bonney (andrewbonney) wrote :

Here's a simple test case from the virtualenv which shows a failure to call the module using the existing import, and a success using an alternative approach:

root@infra1-nova-api-container-c49119df:~# source /openstack/venvs/nova-22.0.0/bin/activate

(nova-22.0.0) root@infra1-nova-api-container-c49119df:~# python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import eventlet
>>> eventlet.tpool.execute
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'eventlet' has no attribute 'tpool'

>>> from eventlet import tpool
>>> tpool.execute
<function execute at 0x7f512ee14ea0>

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

Changed in nova:
status: Incomplete → Expired
Revision history for this message
Andrew Bonney (andrewbonney) wrote :

This is still an issue as far as I'm aware

Andrew Bonney (andrewbonney)
Changed in nova:
status: Expired → New
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

Yes it seems like that import is bogus. In my environment it did not cause the import error as the wsgi application initialization already imported eventlet.tpool indirectly.

This is the way we get tpool now:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 21, in <module>
    from nova.api.openstack.compute.routes import APIRouterV21 # noqa
  File "/opt/stack/nova/nova/api/openstack/compute/routes.py", line 20, in <module>
    from nova.api.openstack.compute import admin_actions
  File "/opt/stack/nova/nova/api/openstack/compute/admin_actions.py", line 17, in <module>
    from nova.api.openstack import common
  File "/opt/stack/nova/nova/api/openstack/common.py", line 27, in <module>
    from nova.compute import task_states
  File "/opt/stack/nova/nova/compute/task_states.py", line 26, in <module>
    from nova.objects import fields
  File "/opt/stack/nova/nova/objects/fields.py", line 24, in <module>
    from nova.network import model as network_model
  File "/opt/stack/nova/nova/network/model.py", line 23, in <module>
    from nova import utils
  File "/opt/stack/nova/nova/utils.py", line 39, in <module>
    from oslo_concurrency import processutils
  File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 57, in <module>
    from eventlet import tpool

So after that import eventlet.tpool is in scope:

stack@aio:~/devstack$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nova.api.openstack.compute.wsgi import init_application
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
>>> import eventlet
>>> eventlet.tpool
<module 'eventlet.tpool' from '/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py'>
>>>

Still this only works by chance. The import should be fixed.

Changed in nova:
status: New → Confirmed
status: Confirmed → Triaged
importance: Undecided → High
assignee: nobody → Balazs Gibizer (balazs-gibizer)
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

It is broken since https://review.opendev.org/c/openstack/nova/+/734776 So I marked it as affecting nova back until victoria

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/830383

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/830383
Committed: https://opendev.org/openstack/nova/commit/b2d28f890872747d099a262e4a208e146b882f3f
Submitter: "Zuul (22348)"
Branch: master

commit b2d28f890872747d099a262e4a208e146b882f3f
Author: Balazs Gibizer <email address hidden>
Date: Tue Feb 22 12:24:37 2022 +0100

    Fix eventlet.tpool import

    Currently nova.utils.tpool_execute() only works by chance. And as the
    bug report shows there are env where it fails.

    The nova.utils.tpool_execute() call tries to uses eventlet.tpool.execute
    but the tpool module is not imported by the utils module only eventlet.
    In devstack it works by chance as the wsgi init actually imports
    eventlet.tpool indirectly via:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 21, in <module>
        from nova.api.openstack.compute.routes import APIRouterV21 # noqa
      File "/opt/stack/nova/nova/api/openstack/compute/routes.py", line 20, in <module>
        from nova.api.openstack.compute import admin_actions
      File "/opt/stack/nova/nova/api/openstack/compute/admin_actions.py", line 17, in <module>
        from nova.api.openstack import common
      File "/opt/stack/nova/nova/api/openstack/common.py", line 27, in <module>
        from nova.compute import task_states
      File "/opt/stack/nova/nova/compute/task_states.py", line 26, in <module>
        from nova.objects import fields
      File "/opt/stack/nova/nova/objects/fields.py", line 24, in <module>
        from nova.network import model as network_model
      File "/opt/stack/nova/nova/network/model.py", line 23, in <module>
        from nova import utils
      File "/opt/stack/nova/nova/utils.py", line 39, in <module>
        from oslo_concurrency import processutils
      File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 57, in <module>
        from eventlet import tpool

    This was broken since I8dbc579e0037969aab4f2bb500fccfbde4190726. This
    patch adds the correct import statement.

    Change-Id: Ic46345ceeb445164aea6ae9b35c457c6150765f6
    Closes-Bug: #1915400

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/836788

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/nova/+/836788
Committed: https://opendev.org/openstack/nova/commit/a9f444a997cd0a26aae3dac821299adf0e944121
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit a9f444a997cd0a26aae3dac821299adf0e944121
Author: Balazs Gibizer <email address hidden>
Date: Tue Feb 22 12:24:37 2022 +0100

    Fix eventlet.tpool import

    Currently nova.utils.tpool_execute() only works by chance. And as the
    bug report shows there are env where it fails.

    The nova.utils.tpool_execute() call tries to uses eventlet.tpool.execute
    but the tpool module is not imported by the utils module only eventlet.
    In devstack it works by chance as the wsgi init actually imports
    eventlet.tpool indirectly via:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 21, in <module>
        from nova.api.openstack.compute.routes import APIRouterV21 # noqa
      File "/opt/stack/nova/nova/api/openstack/compute/routes.py", line 20, in <module>
        from nova.api.openstack.compute import admin_actions
      File "/opt/stack/nova/nova/api/openstack/compute/admin_actions.py", line 17, in <module>
        from nova.api.openstack import common
      File "/opt/stack/nova/nova/api/openstack/common.py", line 27, in <module>
        from nova.compute import task_states
      File "/opt/stack/nova/nova/compute/task_states.py", line 26, in <module>
        from nova.objects import fields
      File "/opt/stack/nova/nova/objects/fields.py", line 24, in <module>
        from nova.network import model as network_model
      File "/opt/stack/nova/nova/network/model.py", line 23, in <module>
        from nova import utils
      File "/opt/stack/nova/nova/utils.py", line 39, in <module>
        from oslo_concurrency import processutils
      File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 57, in <module>
        from eventlet import tpool

    This was broken since I8dbc579e0037969aab4f2bb500fccfbde4190726. This
    patch adds the correct import statement.

    Change-Id: Ic46345ceeb445164aea6ae9b35c457c6150765f6
    Closes-Bug: #1915400
    (cherry picked from commit b2d28f890872747d099a262e4a208e146b882f3f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/840733

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/840733
Committed: https://opendev.org/openstack/nova/commit/b57fce8d14b94059c65526450468d48dab736983
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b57fce8d14b94059c65526450468d48dab736983
Author: Balazs Gibizer <email address hidden>
Date: Tue Feb 22 12:24:37 2022 +0100

    Fix eventlet.tpool import

    Currently nova.utils.tpool_execute() only works by chance. And as the
    bug report shows there are env where it fails.

    The nova.utils.tpool_execute() call tries to uses eventlet.tpool.execute
    but the tpool module is not imported by the utils module only eventlet.
    In devstack it works by chance as the wsgi init actually imports
    eventlet.tpool indirectly via:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 21, in <module>
        from nova.api.openstack.compute.routes import APIRouterV21 # noqa
      File "/opt/stack/nova/nova/api/openstack/compute/routes.py", line 20, in <module>
        from nova.api.openstack.compute import admin_actions
      File "/opt/stack/nova/nova/api/openstack/compute/admin_actions.py", line 17, in <module>
        from nova.api.openstack import common
      File "/opt/stack/nova/nova/api/openstack/common.py", line 27, in <module>
        from nova.compute import task_states
      File "/opt/stack/nova/nova/compute/task_states.py", line 26, in <module>
        from nova.objects import fields
      File "/opt/stack/nova/nova/objects/fields.py", line 24, in <module>
        from nova.network import model as network_model
      File "/opt/stack/nova/nova/network/model.py", line 23, in <module>
        from nova import utils
      File "/opt/stack/nova/nova/utils.py", line 39, in <module>
        from oslo_concurrency import processutils
      File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 57, in <module>
        from eventlet import tpool

    This was broken since I8dbc579e0037969aab4f2bb500fccfbde4190726. This
    patch adds the correct import statement.

    Change-Id: Ic46345ceeb445164aea6ae9b35c457c6150765f6
    Closes-Bug: #1915400
    (cherry picked from commit b2d28f890872747d099a262e4a208e146b882f3f)
    (cherry picked from commit a9f444a997cd0a26aae3dac821299adf0e944121)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/845838

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 25.0.1

This issue was fixed in the openstack/nova 25.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.1.1

This issue was fixed in the openstack/nova 24.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 26.0.0.0rc1

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/wallaby)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/845838
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.