| 2022-11-10 11:51:15 |
Christian Ehrhardt |
bug |
|
|
added bug |
| 2022-11-10 11:51:46 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Kinetic |
|
| 2022-11-10 11:51:46 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Kinetic) |
|
| 2022-11-10 11:51:46 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-11-10 11:51:46 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Jammy) |
|
| 2022-11-10 11:51:56 |
Christian Ehrhardt |
tags |
|
libvirt-23.04 server-todo |
|
| 2022-11-10 11:52:04 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Server |
| 2022-11-16 16:14:51 |
Christian Ehrhardt |
tags |
libvirt-23.04 server-todo |
libvirt-23.04 |
|
| 2022-11-17 08:46:33 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
New |
Triaged |
|
| 2022-11-17 08:46:35 |
Christian Ehrhardt |
libvirt (Ubuntu Jammy): status |
New |
Triaged |
|
| 2022-11-17 08:46:36 |
Christian Ehrhardt |
libvirt (Ubuntu Kinetic): status |
New |
Triaged |
|
| 2022-11-17 08:46:51 |
Christian Ehrhardt |
tags |
libvirt-23.04 |
libvirt-23.04 server-todo |
|
| 2022-11-17 08:47:12 |
Christian Ehrhardt |
libvirt (Ubuntu): assignee |
|
Christian Ehrhardt (paelzer) |
|
| 2022-11-17 08:47:14 |
Christian Ehrhardt |
libvirt (Ubuntu Jammy): assignee |
|
Christian Ehrhardt (paelzer) |
|
| 2022-11-17 08:47:16 |
Christian Ehrhardt |
libvirt (Ubuntu Kinetic): assignee |
|
Christian Ehrhardt (paelzer) |
|
| 2022-11-22 09:49:20 |
Christian Ehrhardt |
description |
Situation:
1. libvirt runs fine
2. something (could be something completely else) triggers udev events
3. one of those udev events has a massive content size
4. systemd libudev delivers EINVAL due to failing on the size
5. libvirt udev listener dies due to that
6. later on many thing in libvirt do a "push event, wait for handler"
which appears to be a hot loop burning a full cpu all the time.
As outlined in the upstream bug [2] there is an issue between
a) systemd before a fix for [1] (=anything before ~now) fixing #4 of the above
b) libvirt before a fix for [2] fixing #5 of the above
c) bios data (or anything else) creating rather huge udev content
By now systemd was fixed upstream avoiding the issue in the long run.
But we should consider to harden at least the most recent LTS by backporting the fix [3] to avoid this issue. If we get a real report by anyone on older releases we can even consider that.
[1]: https://github.com/systemd/systemd/issues/24987
[2]: https://gitlab.com/libvirt/libvirt/-/issues/245
[3]: https://listman.redhat.com/archives/libvir-list/2022-November/235723.html |
[ Impact ]
* udev rules can carry a lot of data. In a particular case
that we have found it was a somewhat overeager bios that
delivered more than 8k content in a single event. This
triggered an issue in systemds libudev which we fixed
in systemd upstream and will find its way into stable
releases as well.
* But at the same time it showed that libvirt is crashing
rather hard on an event like this and that led to a
libvirt thread to essentially
a) become a busy loop burnign cpu
b) missing events as the handler died
* Upstream accepted my fix to harden the event handler for
that situation which allows it to survive working with
the older broken systemd as well as any other situation
that might cause this.
This SRU will backport that to Ubuntu releases.
[ Test Plan ]
* There might be artificial ways to test this, but we know
that the internal system diglett has one of the bad bios
that trigger this error.
* On such a system one just needs to run these steps
1. libvirt runs fine
2. something (could be something completely else) triggers udev events
3. one of those udev events has a massive content size
4. systemd libudev delivers EINVAL due to failing on the size
5. libvirt udev listener dies due to that
6. later on many thing in libvirt do a "push event, wait for handler"
which appears to be a hot loop burning a full cpu all the time.
With the fix the busy loop will not be triggered.
[ Where problems could occur ]
* The udev event handler of libvirt will now survive EINVAL
return codes. I consider it unlikely but there might be a
chance that other situations cause those which now would
no more leading to an exit of the handler.
Nevertheless the consequence of silently having that handler
dying is bad, and IMHO with the fix - even in this unlikely
environment - it is better to have the handler continue working.
[ Other Info ]
* n/a
----
Situation:
1. libvirt runs fine
2. something (could be something completely else) triggers udev events
3. one of those udev events has a massive content size
4. systemd libudev delivers EINVAL due to failing on the size
5. libvirt udev listener dies due to that
6. later on many thing in libvirt do a "push event, wait for handler"
which appears to be a hot loop burning a full cpu all the time.
As outlined in the upstream bug [2] there is an issue between
a) systemd before a fix for [1] (=anything before ~now) fixing #4 of the above
b) libvirt before a fix for [2] fixing #5 of the above
c) bios data (or anything else) creating rather huge udev content
By now systemd was fixed upstream avoiding the issue in the long run.
But we should consider to harden at least the most recent LTS by backporting the fix [3] to avoid this issue. If we get a real report by anyone on older releases we can even consider that.
[1]: https://github.com/systemd/systemd/issues/24987
[2]: https://gitlab.com/libvirt/libvirt/-/issues/245
[3]: https://listman.redhat.com/archives/libvir-list/2022-November/235723.html |
|
| 2022-11-22 10:25:34 |
Christian Ehrhardt |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/433429 |
|
| 2022-11-24 07:18:40 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
Triaged |
In Progress |
|
| 2022-11-24 07:18:42 |
Christian Ehrhardt |
libvirt (Ubuntu Jammy): status |
Triaged |
In Progress |
|
| 2022-11-24 07:18:43 |
Christian Ehrhardt |
libvirt (Ubuntu Kinetic): status |
Triaged |
In Progress |
|
| 2022-11-29 02:36:24 |
Launchpad Janitor |
libvirt (Ubuntu): status |
In Progress |
Fix Released |
|
| 2022-12-05 20:34:47 |
Brian Murray |
libvirt (Ubuntu Kinetic): status |
In Progress |
Fix Committed |
|
| 2022-12-05 20:34:49 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-12-05 20:34:51 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2022-12-05 20:34:54 |
Brian Murray |
tags |
libvirt-23.04 server-todo |
libvirt-23.04 server-todo verification-needed verification-needed-kinetic |
|
| 2022-12-06 20:39:05 |
Brian Murray |
libvirt (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2022-12-06 20:39:11 |
Brian Murray |
tags |
libvirt-23.04 server-todo verification-needed verification-needed-kinetic |
libvirt-23.04 server-todo verification-needed verification-needed-jammy verification-needed-kinetic |
|
| 2022-12-08 15:08:52 |
Christian Ehrhardt |
tags |
libvirt-23.04 server-todo verification-needed verification-needed-jammy verification-needed-kinetic |
libvirt-23.04 server-todo verification-done-jammy verification-needed verification-needed-kinetic |
|
| 2022-12-12 10:26:41 |
Christian Ehrhardt |
tags |
libvirt-23.04 server-todo verification-done-jammy verification-needed verification-needed-kinetic |
libvirt-23.04 server-todo verification-done verification-done-jammy verification-done-kinetic |
|
| 2023-01-03 21:18:34 |
Launchpad Janitor |
libvirt (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2023-01-03 21:18:44 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2023-01-03 21:20:28 |
Launchpad Janitor |
libvirt (Ubuntu Kinetic): status |
Fix Committed |
Fix Released |
|
