| 2019-12-04 21:28:05 |
Jamie Strandboge |
bug |
|
|
added bug |
| 2019-12-04 21:29:04 |
Jamie Strandboge |
description |
With 0.7.4+ds-1 from 19.10, usbguard may stop responding to events when recvmsg fails with ENOBUFS. To reproduce:
while /bin/true ; do
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo udevadm settle --timeout=3
done
Eventually, this pops out in the journald logs:
usbguard-daemon[12488]: ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[12488]: UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
and usbguard will no longer process events until restarted (sudo systemctl restart usbguard).
I then installed usbguard 0.7.5 from the desktop team's PPA[1], ran the loop and saw the equivalent error pop out:
usbguard-daemon[5958]: [1575487887.438] (E) ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[5958]: [1575487887.438] (E) UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
This is coming from https://github.com/USBGuard/usbguard/blob/master/src/Library/UEventDeviceManager.cpp#L528 where recvmsg() is returning -1 with 'No buffer space available' (ENOBUFS). Looking at sysdeps/gnu/errlist.c in glibc, ENOBUFS is documented as:
#ifdef ENOBUFS
/*
TRANS The kernel's buffers for I/O operations are all in use. In GNU, this
TRANS error is always synonymous with @code{ENOMEM}; you may get one or the
TRANS other from network operations. */
[ERR_REMAP (ENOBUFS)] = N_("No buffer space available"),
It seems that usbguard should be able to handle transient ENOBUFS scenarios with uevent storms and recover when the kernel's buffers are back in order. In my testing, I treated ENOBUFS similarly to EAGAIN/EWOULDBLOCK by logging a warning and simply returning[2]:
usbguard-daemon[2405]: ueventProcessRead: failed to read pending uevent (returning): rc=-1 errno=105
It appears usbguard continues to function (eg, if I do the loop for a long time, usbguard remains responsive and processes other uevents, but I'm not sure this is the correct approach since, for example, UEventDeviceManager::ueventOpen() speaks to this error condition by saying: "Set a 1MiB receive buffer on the netlink socket to avoid ENOBUFS error in recvmsg."
[1]https://launchpad.net/~ubuntu-desktop/+archive/ubuntu/usbguard
[2]exploratory patch (do not commit in Ubuntu without upstream confirmation)
Index: usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
===================================================================
--- usbguard-0.7.4+ds.orig/src/Library/UEventDeviceManager.cpp
+++ usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
@@ -468,6 +468,12 @@ namespace usbguard
<< "reading from uevent source would block thread execution";
return;
}
+ else if (saved_errno == ENOBUFS) {
+ USBGUARD_LOG(Error) << "ueventProcessRead: "
+ << "failed to read pending uevent (returning): "
+ << "rc=" << rc << " errno=" << saved_errno;
+ return;
+ }
else {
USBGUARD_LOG(Error) << "ueventProcessRead: "
<< "failed to read pending uevent: " |
With 0.7.4+ds-1 from 19.10, usbguard may stop responding to events when recvmsg fails with ENOBUFS. To reproduce:
while /bin/true ; do
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo udevadm settle --timeout=3
done
Eventually, this pops out in the journald logs:
usbguard-daemon[12488]: ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[12488]: UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
and usbguard will no longer process events until restarted (sudo systemctl restart usbguard).
I then installed usbguard 0.7.5 from the desktop team's PPA[1], ran the loop and saw the equivalent error pop out:
usbguard-daemon[5958]: [1575487887.438] (E) ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[5958]: [1575487887.438] (E) UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
This is coming from https://github.com/USBGuard/usbguard/blob/master/src/Library/UEventDeviceManager.cpp#L528 where recvmsg() is returning -1 with 'No buffer space available' (ENOBUFS). Looking at sysdeps/gnu/errlist.c in glibc, ENOBUFS is documented as:
#ifdef ENOBUFS
/*
TRANS The kernel's buffers for I/O operations are all in use. In GNU, this
TRANS error is always synonymous with @code{ENOMEM}; you may get one or the
TRANS other from network operations. */
[ERR_REMAP (ENOBUFS)] = N_("No buffer space available"),
It seems that usbguard should be able to handle transient ENOBUFS scenarios with uevent storms and recover when the kernel's buffers are back in order. In my testing, I treated ENOBUFS similarly to EAGAIN/EWOULDBLOCK by logging a warning and simply returning[2]:
usbguard-daemon[2405]: ueventProcessRead: failed to read pending uevent (returning): rc=-1 errno=105
It appears usbguard continues to function (eg, if I do the loop for a long time, usbguard remains responsive and processes other uevents, but I'm not sure this is the correct approach since, for example, UEventDeviceManager::ueventOpen() speaks to this error condition by saying: "Set a 1MiB receive buffer on the netlink socket to avoid ENOBUFS error in recvmsg").
[1]https://launchpad.net/~ubuntu-desktop/+archive/ubuntu/usbguard
[2]exploratory patch (do not commit in Ubuntu without upstream confirmation)
Index: usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
===================================================================
--- usbguard-0.7.4+ds.orig/src/Library/UEventDeviceManager.cpp
+++ usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
@@ -468,6 +468,12 @@ namespace usbguard
<< "reading from uevent source would block thread execution";
return;
}
+ else if (saved_errno == ENOBUFS) {
+ USBGUARD_LOG(Error) << "ueventProcessRead: "
+ << "failed to read pending uevent (returning): "
+ << "rc=" << rc << " errno=" << saved_errno;
+ return;
+ }
else {
USBGUARD_LOG(Error) << "ueventProcessRead: "
<< "failed to read pending uevent: " |
|
| 2019-12-04 21:36:15 |
Jamie Strandboge |
bug watch added |
|
https://github.com/USBGuard/usbguard/issues/349 |
|
| 2019-12-04 21:36:15 |
Jamie Strandboge |
bug task added |
|
usbguard |
|
| 2019-12-04 21:44:03 |
Jamie Strandboge |
usbguard (Ubuntu): status |
New |
Confirmed |
|
| 2019-12-04 22:53:01 |
Bug Watch Updater |
usbguard: status |
Unknown |
New |
|
| 2020-04-03 08:33:53 |
Arjen Lentz |
bug |
|
|
added subscriber Arjen Lentz |
| 2020-04-30 18:52:58 |
Marcos Alano |
bug |
|
|
added subscriber Marcos Alano |
| 2020-05-06 17:58:20 |
Jamie Strandboge |
usbguard (Ubuntu): status |
Confirmed |
Triaged |
|
| 2020-07-08 12:43:58 |
Sebastien Bacher |
usbguard (Ubuntu): status |
Triaged |
Fix Released |
|
| 2024-03-28 12:38:31 |
Heitor Alves de Siqueira |
nominated for series |
|
Ubuntu Focal |
|
| 2024-03-28 12:38:31 |
Heitor Alves de Siqueira |
bug task added |
|
usbguard (Ubuntu Focal) |
|
| 2024-03-28 12:38:38 |
Heitor Alves de Siqueira |
usbguard (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2024-03-28 12:38:40 |
Heitor Alves de Siqueira |
usbguard (Ubuntu Focal): status |
New |
In Progress |
|
| 2024-03-28 12:38:42 |
Heitor Alves de Siqueira |
usbguard (Ubuntu Focal): assignee |
|
Heitor Alves de Siqueira (halves) |
|
| 2024-03-28 12:38:49 |
Heitor Alves de Siqueira |
bug |
|
|
added subscriber Heitor Alves de Siqueira |
| 2024-03-28 13:03:49 |
Heitor Alves de Siqueira |
description |
With 0.7.4+ds-1 from 19.10, usbguard may stop responding to events when recvmsg fails with ENOBUFS. To reproduce:
while /bin/true ; do
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo udevadm settle --timeout=3
done
Eventually, this pops out in the journald logs:
usbguard-daemon[12488]: ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[12488]: UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
and usbguard will no longer process events until restarted (sudo systemctl restart usbguard).
I then installed usbguard 0.7.5 from the desktop team's PPA[1], ran the loop and saw the equivalent error pop out:
usbguard-daemon[5958]: [1575487887.438] (E) ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[5958]: [1575487887.438] (E) UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
This is coming from https://github.com/USBGuard/usbguard/blob/master/src/Library/UEventDeviceManager.cpp#L528 where recvmsg() is returning -1 with 'No buffer space available' (ENOBUFS). Looking at sysdeps/gnu/errlist.c in glibc, ENOBUFS is documented as:
#ifdef ENOBUFS
/*
TRANS The kernel's buffers for I/O operations are all in use. In GNU, this
TRANS error is always synonymous with @code{ENOMEM}; you may get one or the
TRANS other from network operations. */
[ERR_REMAP (ENOBUFS)] = N_("No buffer space available"),
It seems that usbguard should be able to handle transient ENOBUFS scenarios with uevent storms and recover when the kernel's buffers are back in order. In my testing, I treated ENOBUFS similarly to EAGAIN/EWOULDBLOCK by logging a warning and simply returning[2]:
usbguard-daemon[2405]: ueventProcessRead: failed to read pending uevent (returning): rc=-1 errno=105
It appears usbguard continues to function (eg, if I do the loop for a long time, usbguard remains responsive and processes other uevents, but I'm not sure this is the correct approach since, for example, UEventDeviceManager::ueventOpen() speaks to this error condition by saying: "Set a 1MiB receive buffer on the netlink socket to avoid ENOBUFS error in recvmsg").
[1]https://launchpad.net/~ubuntu-desktop/+archive/ubuntu/usbguard
[2]exploratory patch (do not commit in Ubuntu without upstream confirmation)
Index: usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
===================================================================
--- usbguard-0.7.4+ds.orig/src/Library/UEventDeviceManager.cpp
+++ usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
@@ -468,6 +468,12 @@ namespace usbguard
<< "reading from uevent source would block thread execution";
return;
}
+ else if (saved_errno == ENOBUFS) {
+ USBGUARD_LOG(Error) << "ueventProcessRead: "
+ << "failed to read pending uevent (returning): "
+ << "rc=" << rc << " errno=" << saved_errno;
+ return;
+ }
else {
USBGUARD_LOG(Error) << "ueventProcessRead: "
<< "failed to read pending uevent: " |
[Impact]
usbguard-daemon will no longer process device events until restarted
[Test Plan]
1. Spin up a Focal VM with usbguard enabled
2. Run the script below:
while /bin/true ; do
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo udevadm settle --timeout=3
done
3. journald logs will indicate the following errors:
usbguard-daemon[12488]: ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[12488]: UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
[Where problems could occur]
The fix introduces an additional check in the usbguard event handler, so we should properly test this path. Potential regressions could cause USB devices to not be probed correctly, or usbguard dropping device events. We should also make sure the usbguard daemon is recovering from ENOBUFS and other errors correctly without needing to be restarted.
[Other Info]
The fix has been introduced upstream in version usbguard-0.7.7:
$ git describe --contains 5f297079b843
usbguard-0.7.7~2
As such, releases starting with Jammy already contain the fix:
$ rmadison usbguard
usbguard | 0.7.2+ds-1 | bionic/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
usbguard | 0.7.6+ds-1build1 | focal/universe | source, amd64, arm64, armhf, ppc64el, riscv64, s390x
usbguard | 1.1.1+ds-3 | jammy/universe | source, amd64, arm64, armhf, ppc64el, riscv64, s390x
usbguard | 1.1.2+ds-4 | mantic/universe | source, amd64, arm64, armhf, ppc64el, riscv64, s390x
usbguard | 1.1.2+ds-6 | noble/universe | source, amd64, arm64, armhf, ppc64el, riscv64, s390x
--
[Original description]
With 0.7.4+ds-1 from 19.10, usbguard may stop responding to events when recvmsg fails with ENOBUFS. To reproduce:
while /bin/true ; do
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo udevadm settle --timeout=3
done
Eventually, this pops out in the journald logs:
usbguard-daemon[12488]: ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[12488]: UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
and usbguard will no longer process events until restarted (sudo systemctl restart usbguard).
I then installed usbguard 0.7.5 from the desktop team's PPA[1], ran the loop and saw the equivalent error pop out:
usbguard-daemon[5958]: [1575487887.438] (E) ueventProcessRead: failed to read pending uevent: rc=-1 errno=105
usbguard-daemon[5958]: [1575487887.438] (E) UEventDeviceManager thread: UEvent device manager: recvmsg: No buffer space available
This is coming from https://github.com/USBGuard/usbguard/blob/master/src/Library/UEventDeviceManager.cpp#L528 where recvmsg() is returning -1 with 'No buffer space available' (ENOBUFS). Looking at sysdeps/gnu/errlist.c in glibc, ENOBUFS is documented as:
#ifdef ENOBUFS
/*
TRANS The kernel's buffers for I/O operations are all in use. In GNU, this
TRANS error is always synonymous with @code{ENOMEM}; you may get one or the
TRANS other from network operations. */
[ERR_REMAP (ENOBUFS)] = N_("No buffer space available"),
It seems that usbguard should be able to handle transient ENOBUFS scenarios with uevent storms and recover when the kernel's buffers are back in order. In my testing, I treated ENOBUFS similarly to EAGAIN/EWOULDBLOCK by logging a warning and simply returning[2]:
usbguard-daemon[2405]: ueventProcessRead: failed to read pending uevent (returning): rc=-1 errno=105
It appears usbguard continues to function (eg, if I do the loop for a long time, usbguard remains responsive and processes other uevents, but I'm not sure this is the correct approach since, for example, UEventDeviceManager::ueventOpen() speaks to this error condition by saying: "Set a 1MiB receive buffer on the netlink socket to avoid ENOBUFS error in recvmsg").
[1]https://launchpad.net/~ubuntu-desktop/+archive/ubuntu/usbguard
[2]exploratory patch (do not commit in Ubuntu without upstream confirmation)
Index: usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
===================================================================
--- usbguard-0.7.4+ds.orig/src/Library/UEventDeviceManager.cpp
+++ usbguard-0.7.4+ds/src/Library/UEventDeviceManager.cpp
@@ -468,6 +468,12 @@ namespace usbguard
<< "reading from uevent source would block thread execution";
return;
}
+ else if (saved_errno == ENOBUFS) {
+ USBGUARD_LOG(Error) << "ueventProcessRead: "
+ << "failed to read pending uevent (returning): "
+ << "rc=" << rc << " errno=" << saved_errno;
+ return;
+ }
else {
USBGUARD_LOG(Error) << "ueventProcessRead: "
<< "failed to read pending uevent: " |
|
| 2024-04-10 14:51:22 |
Robie Basak |
usbguard (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2024-04-10 14:51:24 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2024-04-10 14:51:27 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2024-04-10 14:51:29 |
Robie Basak |
tags |
|
verification-needed verification-needed-focal |
|
| 2024-04-16 17:49:30 |
Heitor Alves de Siqueira |
tags |
verification-needed verification-needed-focal |
verification-done verification-done-focal |
|
| 2024-05-01 10:30:05 |
Launchpad Janitor |
usbguard (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2024-05-01 10:30:08 |
Robie Basak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
