nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | OpenStack Compute (nova) |
Critical
|
Lee Yarwood | ||
| | Train |
High
|
Lee Yarwood | ||
| | Ussuri |
High
|
Lee Yarwood | ||
| | Victoria |
High
|
Lee Yarwood | ||
Bug Description
Our gate jobs running python 3.6 and libvirt-python 6.8.0 are currently failing because when we call libvirt methods, the following error is raised [1]:
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.454426 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.456615 ubuntu-
Oct 23 12:12:21.459391 ubuntu-
The libvirt-python package version was recently bumped in upper-constraints from version 6.6.0 to 6.8.0:
https:/
and the tempest job on that change ^ passed running python 3.8.
But then our jobs running python 3.6 immediately began to fail.
There appears to be some kind of incompatibility between python 3.6 and libvirt-python 6.8.0.
I had a look through the libvirt-python repo to see what changed between 6.6.0 and 6.8.0 and can't tell what's the root cause. We get the TypeError raised in nova-compute when we call the libvirt openAuth method but nothing in this diff [2] looks problematic?
Not sure what we can do other than revert the upper-constraints bump for libvirt-python and go back to 6.6.0.
[1] https:/
[2] https:/
| Changed in nova: | |
| status: | New → Confirmed |
| assignee: | nobody → melanie witt (melwitt) |
| Daniel Berrange (berrange) wrote : | #1 |
| Daniel Berrange (berrange) wrote : | #2 |
The error message comes from the Python standard library typing.py module, however, the specific error message was removed in python 3.7.0 - only exists in python 3.6.x series
| Daniel Berrange (berrange) wrote : | #3 |
I've done some tests with openAuth() and confirmed that use of tpool proxy breaks type hinting on python 3.6.
| Daniel Berrange (berrange) wrote : | #4 |
I believe the problem is actually this bit of code:
https:/
It attempts to get a list of libvirt classes via inspection. With the introduction of type checking to libvirt, there are a bunch of private global variables that are pointing to classes. This means Nova is extracting a bunch of classes that it shouldn't ever look at.
Check this code:
classes = inspect.
for cls in classes:
print("Seen %s" % str(cls))
It prints:
Seen ('Callable', typing.Callable)
Seen ('Dict', typing.Dict)
Seen ('List', typing.List)
Seen ('TracebackType', <class 'traceback'>)
Seen ('Tuple', typing.Tuple)
Seen ('Type', typing.Type)
Seen ('TypeVar', typing.TypeVar)
Seen ('_BlkioParameter', typing.Dict[str, typing.Any])
Seen ('_DomainCB', typing.
Seen ('_EventAddHand
Seen ('_EventAddTime
Seen ('_EventCB', typing.
Seen ('_EventRemoveH
Seen ('_EventRemoveT
Seen ('_EventUpdateH
Seen ('_EventUpdateT
Seen ('_MemoryParame
Seen ('_SchedParameter', typing.Dict[str, typing.Any])
Seen ('_TimerCB', typing.
Seen ('_TypedParameter', typing.Dict[str, typing.Any])
Seen ('libvirtError', <class 'libvirt.
Seen ('virConnect', <class 'libvirt.
Seen ('virDomain', <class 'libvirt.
Seen ('virDomainChec
Seen ('virDomainSnap
Seen ('virInterface', <class 'libvirt.
Seen ('virNWFilter', <class 'libvirt.
Seen ('virNWFilterBi
Seen ('virNetwork', <class 'libvirt.
Seen ('virNetworkPort', <class 'libvirt.
Seen ('virNodeDevice', <class 'libvirt.
Seen ('virSecret', <class 'libvirt.
Seen ('virStoragePool', <class 'libvirt.
Seen ('virStorageVol', <class 'libvirt.
Seen ('virStream', <class 'libvirt.
Nova should only be looking at wrapping classes beginning with "vir".
I believe if nova changes:
return tuple([cls[1] for cls in classes if cls[0] != 'libvirtError'])
to
return tuple([cls[1] for cls in classes if cls.startswith(
then the bug should go away.
| Kashyap Chamarthy (kashyapc) wrote : | #5 |
Thanks for the analysis, Dan.
Seems like the original change of extracting the libvirt classes was introduced as part of this change:
https:/
libvirt: Fix service-wide pauses caused by un-proxied libvirt calls
(2019-08-21)
Fix proposed to branch: master
Review: https:/
| Changed in nova: | |
| assignee: | melanie witt (melwitt) → Lee Yarwood (lyarwood) |
| status: | Confirmed → In Progress |
Reviewed: https:/
Committed: https:/
Submitter: Jens Harbott (frickler) (<email address hidden>)
Branch: master
commit 0d2ca53bb86b8e4
Author: Lee Yarwood <email address hidden>
Date: Tue Oct 27 08:44:59 2020 +0000
libvirt: Only ask tpool.Proxy to autowrap vir* classes
I668643c836
pauses by providing a more complete list of classes to tpool.Proxy.
While this excluded libvirtError it can include internal libvirt-python
classes pointed to by private globals that have been introduced with the
use of type checking within the module.
Any attempt to wrap these internal classes will result in the failure
seen in bug #1901383. As a result this change simply ignores any class
found during inspection that doesn't start with the `vir` string, used
by libvirt to denote public methods and classes.
Closes-Bug: #1901383
Co-Authored-By: Daniel Berrange <email address hidden>
Change-Id: I568b0c4fd6069b
| Changed in nova: | |
| status: | In Progress → Fix Released |
Fix proposed to branch: stable/victoria
Review: https:/
Fix proposed to branch: stable/ussuri
Review: https:/
Fix proposed to branch: stable/train
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/victoria
commit 048a3337a8c98ec
Author: Lee Yarwood <email address hidden>
Date: Tue Oct 27 08:44:59 2020 +0000
libvirt: Only ask tpool.Proxy to autowrap vir* classes
I668643c836
pauses by providing a more complete list of classes to tpool.Proxy.
While this excluded libvirtError it can include internal libvirt-python
classes pointed to by private globals that have been introduced with the
use of type checking within the module.
Any attempt to wrap these internal classes will result in the failure
seen in bug #1901383. As a result this change simply ignores any class
found during inspection that doesn't start with the `vir` string, used
by libvirt to denote public methods and classes.
Closes-Bug: #1901383
Co-Authored-By: Daniel Berrange <email address hidden>
Change-Id: I568b0c4fd6069b
(cherry picked from commit 0d2ca53bb86b8e4
| tags: | added: in-stable-victoria |
| melanie witt (melwitt) wrote : | #12 |
This issue was fixed in the openstack/nova 21.2.0 release.
This issue was fixed in the openstack/nova 20.6.0 release.
This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.
The big change in 6.8.0 was that libvirt added python type hinting for almost all its APIs.
For the openAuth method, the change is this:
-def openAuth(uri, auth, flags=0):
+def openAuth(uri: str, auth: List, flags: int = 0) -> 'virConnect':
but nova should be matching that typing signature from what I see.
I wonder if there's a problem with eventlet tpool proxy stuff causing bad interactions with typing ?