OK, let me wrap here a few findings.
First, the number of occurences is way lower than what I said in c#25 (fortunately).
Something around 19 occurrences around the last 20 days (which is still pretty big tho)

Secondly, I think I identified the real exception leading to a failed test condition :


2023-01-24 18:29:03.537384 | ubuntu-jammy | During handling of the above exception, another exception occurred:
2023-01-24 18:29:03.537389 | ubuntu-jammy |
2023-01-24 18:29:03.537394 | ubuntu-jammy | Traceback (most recent call last):
2023-01-24 18:29:03.537400 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10619, in _live_migration
2023-01-24 18:29:03.537405 | ubuntu-jammy |     self.live_migration_abort(instance)
2023-01-24 18:29:03.537411 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10087, in live_migration_abort
2023-01-24 18:29:03.537416 | ubuntu-jammy |     guest = self._host.get_guest(instance)
2023-01-24 18:29:03.537422 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/host.py", line 673, in get_guest
2023-01-24 18:29:03.537427 | ubuntu-jammy |     return libvirt_guest.Guest(self._get_domain(instance))
2023-01-24 18:29:03.537432 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/guest.py", line 112, in __init__
2023-01-24 18:29:03.537438 | ubuntu-jammy |     libvirt = importutils.import_module('libvirt')
2023-01-24 18:29:03.537482 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_utils/importutils.py", line 73, in import_module
2023-01-24 18:29:03.537494 | ubuntu-jammy |     __import__(import_str)
2023-01-24 18:29:03.537500 | ubuntu-jammy |   File
2023-01-24 18:29:03.537505 | ubuntu-jammy |     "/home/zuul/src/opendev.org/openstack/nova/nova/tests/fixtures/nova.py", line 1829, in find_spec
2023-01-24 18:29:03.537511 | ubuntu-jammy |     raise ImportError(fullname)
2023-01-24 18:29:03.537516 | ubuntu-jammy | ImportError: libvirt
2023-01-24 18:29:03.537521 | ubuntu-jammy |
2023-01-24 18:29:03.537527 | ubuntu-jammy | During handling of the above exception, another exception occurred:
2023-01-24 18:29:03.537532 | ubuntu-jammy |
2023-01-24 18:29:03.537537 | ubuntu-jammy | Traceback (most recent call last):
2023-01-24 18:29:03.537543 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 8854, in _do_live_migration
2023-01-24 18:29:03.537548 | ubuntu-jammy |     self.driver.live_migration(context, instance, dest,
2023-01-24 18:29:03.537553 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10076, in live_migration
2023-01-24 18:29:03.537559 | ubuntu-jammy |     self._live_migration(context, instance, dest,
2023-01-24 18:29:03.537564 | ubuntu-jammy |   File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10620, in _live_migration
2023-01-24 18:29:03.537569 | ubuntu-jammy |     except libvirt.libvirtError:
2023-01-24 18:29:03.537575 | ubuntu-jammy | AttributeError: 'NoneType' object has no attribute 'libvirtError'


There are false positives like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aea/850501/15/check/nova-tox-functional-py38/aea02af/job-output.txt that show that most of the times the eventlet threads are missing the DB, but the overall test isn't failing, unless a specific thread above is running *after* we cleaned up the poisoned libvirt object.

Corresponding logsearch query :
    logsearch log --job-group nova-functional --result FAILURE  "ImportError: This test imports the 'libvirt' module, which it should not in the test environment. Please add appropriate mocking to this test."  --days 20

    Builds with matching logs 19/143:

    +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+

    | uuid                             | finished            | project        | pipeline | review                            | branch | job                       |

    +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+

    | 9f2e63e091e74ba99151a5c8e30890d1 | 2023-02-07T09:21:45 | openstack/nova | check    | https://review.opendev.org/870938 | master | nova-tox-functional-py39  |

    | 5a44ba30aec545de867f2a81c2fb35b2 | 2023-02-07T08:51:52 | openstack/nova | check    | https://review.opendev.org/870932 | master | nova-tox-functional-py38  |

    | d00d1ffaa9b34afba264c32b0bdf6a7c | 2023-02-06T16:24:57 | openstack/nova | check    | https://review.opendev.org/868237 | master | nova-tox-functional-py38  |

    | 679799857eee48e799dd3fdc80712cf0 | 2023-02-03T23:18:23 | openstack/nova | gate     | https://review.opendev.org/872432 | master | nova-tox-functional-py38  |

    | 2f8f21db817f4488a24cbc72f8922a6c | 2023-02-02T18:53:13 | openstack/nova | check    | https://review.opendev.org/860287 | master | nova-tox-functional-py38  |

    | 3888abbc3b444916875966eda45abe95 | 2023-02-02T12:50:31 | openstack/nova | check    | https://review.opendev.org/860288 | master | nova-tox-functional-py310 |

    | 003e39baf11248cca9d4ad857228d776 | 2023-02-02T10:36:21 | openstack/nova | check    | https://review.opendev.org/872413 | master | nova-tox-functional-py310 |

    | efd50b6ce8c744089ba3023408449cc1 | 2023-01-31T20:36:40 | openstack/nova | check    | https://review.opendev.org/872348 | master | nova-tox-functional-py310 |

    | 27395778c2da42fbbef27fb4d1ae175d | 2023-01-30T19:22:13 | openstack/nova | check    | https://review.opendev.org/863920 | master | nova-tox-functional-py310 |

    | 8025cac32927455e826448d5165800d2 | 2023-01-25T12:42:42 | openstack/nova | check    | https://review.opendev.org/854824 | master | nova-tox-functional-py39  |

    | 92daaa41e4404efdac9186d10dabc2e0 | 2023-01-25T11:44:18 | openstack/nova | check    | https://review.opendev.org/833090 | master | nova-tox-functional-py38  |

    | 466e0d714c6d4d72b566cdf8fe21d650 | 2023-01-24T18:38:42 | openstack/nova | check    | https://review.opendev.org/860282 | master | nova-tox-functional-py310 |

    | 5e37af39cbeb40f7b50977d35a96e2e9 | 2023-01-24T15:27:31 | openstack/nova | check    | https://review.opendev.org/871612 | master | nova-tox-functional-py38  |

    | 4b02df6e2a92485194e47859ea92480c | 2023-01-23T15:27:17 | openstack/nova | check    | https://review.opendev.org/869950 | master | nova-tox-functional-py38  |

    | ef9ef691ad2f43b3bc40800d95c8eefa | 2023-01-20T23:00:13 | openstack/nova | check    | https://review.opendev.org/863920 | master | nova-tox-functional-py310 |

    | d926c0602cdf4f94a67ef342b63fc81c | 2023-01-19T15:20:11 | openstack/nova | check    | https://review.opendev.org/867324 | master | nova-tox-functional-py39  |

    | 889b15a8573c4aed9d93c71e5e22bef7 | 2023-01-19T13:43:52 | openstack/nova | check    | https://review.opendev.org/870932 | master | nova-tox-functional-py310 |

    | fb32a7becc0043e283cc00c5504ebc9b | 2023-01-19T13:33:02 | openstack/nova | check    | https://review.opendev.org/870933 | master | nova-tox-functional-py310 |

    | c763a465e27e4b84bcdc7211fa8d61d3 | 2023-01-19T13:12:54 | openstack/nova | check    | https://review.opendev.org/870932 | master | nova-tox-functional-py39  |

    +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+