cpu might go into a hot loop on systems with uncommon udev data
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
libvirt (Ubuntu) |
Fix Released
|
Undecided
|
Christian Ehrhardt | ||
Jammy |
Fix Released
|
Undecided
|
Christian Ehrhardt | ||
Kinetic |
Fix Released
|
Undecided
|
Christian Ehrhardt |
Bug Description
[ Impact ]
* udev rules can carry a lot of data. In a particular case
that we have found it was a somewhat overeager bios that
delivered more than 8k content in a single event. This
triggered an issue in systemds libudev which we fixed
in systemd upstream and will find its way into stable
releases as well.
* But at the same time it showed that libvirt is crashing
rather hard on an event like this and that led to a
libvirt thread to essentially
a) become a busy loop burnign cpu
b) missing events as the handler died
* Upstream accepted my fix to harden the event handler for
that situation which allows it to survive working with
the older broken systemd as well as any other situation
that might cause this.
This SRU will backport that to Ubuntu releases.
[ Test Plan ]
* There might be artificial ways to test this, but we know
that the internal system diglett has one of the bad bios
that trigger this error.
* On such a system one just needs to run these steps
1. libvirt runs fine
2. something (could be something completely else) triggers udev events
3. one of those udev events has a massive content size
4. systemd libudev delivers EINVAL due to failing on the size
5. libvirt udev listener dies due to that
6. later on many thing in libvirt do a "push event, wait for handler"
which appears to be a hot loop burning a full cpu all the time.
With the fix the busy loop will not be triggered.
[ Where problems could occur ]
* The udev event handler of libvirt will now survive EINVAL
return codes. I consider it unlikely but there might be a
chance that other situations cause those which now would
no more leading to an exit of the handler.
Nevertheless the consequence of silently having that handler
dying is bad, and IMHO with the fix - even in this unlikely
environment - it is better to have the handler continue working.
[ Other Info ]
* n/a
----
Situation:
1. libvirt runs fine
2. something (could be something completely else) triggers udev events
3. one of those udev events has a massive content size
4. systemd libudev delivers EINVAL due to failing on the size
5. libvirt udev listener dies due to that
6. later on many thing in libvirt do a "push event, wait for handler"
which appears to be a hot loop burning a full cpu all the time.
As outlined in the upstream bug [2] there is an issue between
a) systemd before a fix for [1] (=anything before ~now) fixing #4 of the above
b) libvirt before a fix for [2] fixing #5 of the above
c) bios data (or anything else) creating rather huge udev content
By now systemd was fixed upstream avoiding the issue in the long run.
But we should consider to harden at least the most recent LTS by backporting the fix [3] to avoid this issue. If we get a real report by anyone on older releases we can even consider that.
[1]: https:/
[2]: https:/
[3]: https:/
Related branches
- Christian Ehrhardt (community): Approve
- Canonical Server Reporter: Pending requested
- git-ubuntu import: Pending requested
-
Diff: 218 lines (+177/-0)5 files modifieddebian/changelog (+15/-0)
debian/patches/series (+3/-0)
debian/patches/ubuntu/lp-1993304-apparmor-allow-getattr-on-usb-devices.patch (+49/-0)
debian/patches/ubuntu/lp-1996176-nodedev-ignore-EINVAL-from-libudev-in-udevEventHandl.patch (+58/-0)
debian/patches/ubuntu/lp-1997269-fix-swtpm-pid-duplication.patch (+52/-0)
tags: | added: libvirt-23.04 server-todo |
tags: | removed: server-todo |
description: | updated |
Changed in libvirt (Ubuntu): | |
status: | Triaged → In Progress |
Changed in libvirt (Ubuntu Jammy): | |
status: | Triaged → In Progress |
Changed in libvirt (Ubuntu Kinetic): | |
status: | Triaged → In Progress |
This has completed upstream now /listman. redhat. com/archives/ libvir- list/2022- November/ 235723. html /gitlab. com/libvirt/ libvirt/ -/commit/ 33a38492b75acb7 dbec9b64c41a5db a4acde4240
https:/
and is applied as:
https:/
There is another bug 1993304 which I'd like to combine in one upload.
So there is a bit more delay to get this fixed.