journal shows "systemd-udevd[2837]: nvme0n1: Process ... failed with exit code 1."

[Jimmy]

Configuration:
 OS:jammy-live-server20220320-amd64.iso
 CPU:AMD EPYC 7702 64-Core Processor
 UEFI Version:D8E119A
 BMC Version:D8BT19I
 SSD:Intel 1.60TB NVMe SSD
 Boot mode:legacy
Reproduce Steps:
 1.Boot into BIOS and set boot mode to legacy
 2.install ubuntu 22.04 on NVMe SSD
 3.Check syslog log
Current behaviors:
syslog shows systemd-udevd errors in Ubuntu 22.04
Feb 9 10:16:19 len systemd-udevd[2837]: nvme0n1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1' failed with exit code 1.
Feb 9 10:16:19 len systemd-udevd[2877]: nvme0n1p3: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1p3' failed with exit code 1.
Feb 9 10:16:19 len systemd-udevd[2876]: nvme0n1p2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1p2' failed with exit code 1.
Feb 9 10:16:19 len systemd-udevd[2837]: nvme0n1p1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1p1' failed with exit code 1.
Feb 9 10:16:19 len systemd-udevd[2828]: sr0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sr0' failed with exit code 1.
Feb 9 10:16:19 len systemd-udevd[2850]: dm-0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-0' failed with exit code 1.

[Jiwei Sun (sunjw10)]

According to our investigation, I found that the issue is caused by the "mount" system call can not be allowed to execute.

And after checking the upstream commit, the issue is introduced by the following patch,

commit ee8f26180d01e3ddd4e5f20b03b81e5e737657ae
Author: Lennart Poettering <email address hidden>
Date: Thu Apr 19 11:04:17 2018 +0200

    units: switch from system call blacklist to whitelist

    This is generally the safer approach, and is what container managers
    (including nspawn) do, hence let's move to this too for our own
    services. This is particularly useful as this this means the new
    @system-service system call filter group will get serious real-life
    testing quickly.

    This also switches from firing SIGSYS on unexpected syscalls to
    returning EPERM. This would have probably been a better default anyway,
    but it's hard to change that these days. When whitelisting system calls
    SIGSYS is highly problematic as system calls that are newly introduced
    to Linux become minefields for services otherwise.

    Note that this enables a system call filter for udev for the first time,
    and will block @clock, @mount and @swap from it. Some downstream
    distributions might want to revert this locally if they want to permit
    unsafe operations on udev rules, but in general this shiuld be mostly
    safe, as we already set MountFlags=shared for udevd, hence at least
    @mount won't change anything.

Before the patch，systemd-udevd.service has not set “SystemCallFilter=”，but after the patch，the “SystemCallFilter=@system-service @module @raw-io” was added

diff --git a/units/systemd-udevd.service.in b/units/systemd-udevd.service.in
index 2b9fa69d9b..6a3814e5d9 100644
--- a/units/systemd-udevd.service.in
+++ b/units/systemd-udevd.service.in
@@ -29,6 +29,8 @@ PrivateMounts=yes
 MemoryDenyWriteExecute=yes
 RestrictRealtime=yes
 RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6
+SystemCallFilter=@system-service @module @raw-io
+SystemCallErrorNumber=EPERM
 SystemCallArchitectures=native
 LockPersonality=yes
 IPAddressDeny=any

Because there is no "SystemCallFilter=@system-service @module @raw-io" in the /usr/lib/systemd/system/systemd-udevd.service on UBuntu20.04, we can not experience the same issue, but can reproduce the issue on UBuntu22.04.
If we add "@mount" to "SystemCallFilter=" on UBuntu22.04, the issue can be fixed.

Thanks,
Regards,
Jiwei

[Jeff Lane  (bladernr)]

Can this be reproduced in EFI mode, or is it only when in Legacy Mode?

ALso, what is the mount command you are using? Just basic like "mount /dev/device /mountpoint"? Or is there a different mount command you are using that is failing?

[Nick Rosbrook (enr0n)]

These errors appear to be coming from the udev rules in /lib/udev/rules.d/66-snapd-autoimport.rules. I also see the errors on my system, but I have not experienced any other problems as a result.

Jimmy or Jiwei, can you please tell us how you came across these logs? What other problem(s) did you experience that seem to be associated?

[Jimmy (sujj5)]

The error messages are also shown under UEFI mode

[Nick Rosbrook (enr0n)]

After speaking with snapd folks, it was determined that these udev rules should not be triggered on desktop or server (and in fact, should be limited to removable media). The proper solution here is to improve the snapd auto-import udev rules.

Given that, we do not need to make any changes to systemd-udevd's syscall filtering.

[Jiwei Sun (sunjw10)]

Hi @Nick Rosbrook (enr0n)

Will any modification(regarding snapd udev rule) be included in the Ubuntu 22.04 release version?

Thanks,
Regards,
Jiwei

[Nick Rosbrook (enr0n)]

I am not sure (I am not directly involved with snapd), but I can ask someone to take a look.

[Tuxo (tuxoholic)]

Rule is triggered by all block devices, I switched it off with !="block" but that can only be a temporary fix.

[Johannes Weber (jhweber)]

I just ran into this bug also on Ubuntu 20 LTS after updating the kernel a week ago (multiboot laptop with Ubuntu and Windows 10). After figuring out that it's not a hardware issue (since windows still booted normally), I made the switch to Jammy, installed Ubuntu and all my software from scratch, and it worked during the first boot cycle. Rebooting a week later after living on standby, I am screwed again. Why is this neither fixed nor is there a set of instructions how to deal with it for the time being, if it is HIGH priority?
Could you please provide simple, pedestrian instructions (including full paths, etc.) for a user who is used to working in bash, but not to the technical details of the inner workings of Ubuntu. I need a robust workaround until this bug is eventually fixed, and have a hard time believing that the bug still shows after the last post is 4 weeks old.
Any help is appreciated.

[Nick Rosbrook (enr0n)]

Hi Johannes,

> Rebooting a week later after living on standby, I am screwed again.

Can you please explain what problems you experienced as a result of this bug? My previous understanding was that there was no real effect, and just a failed udev rule (which should not be running anyways) and noise in the logs.

To work around this for the time being, you can simply remove the offending udev rule:

# rm /lib/udev/rules.d/66-snapd-autoimport.rules

[Alberto Mardegan (mardy)]

Thanks Nick for pointing me to this bug; I was working on the same issue, as reported in a duplicate bug.

As far as the classic desktop is concerned, this bug occurs not because of the failed mount, but because of "unshare" failing (see the discussion in bug 1971955); in any case, the analysis by Jiwei Sun is correct: it's happening because of the added SystemCallFilter entry to the udev unit file.

[Alberto Mardegan (mardy)]

Some updates, while I'm still investigating this on a QEMU running Ubuntu Core 22.

The "snap auto-import" command is only needed on Ubuntu Core systems, so the plan is to remove the /lib/udev/rules.d/66-snapd-autoimport.rules from the snapd deb package (and consequently also from the snapd snap).

We do still need it in UC, but as a matter of fact the script is already provided by the Ubuntu Core repositories:

- UC22: https://github.com/snapcore/core-base/blob/main/static/usr/lib/udev/rules.d/66-snapd-autoimport.rules
- UC20: https://github.com/snapcore/core20/blob/master/static/usr/lib/udev/rules.d/66-snapd-autoimport.rules
- UC18: https://github.com/snapcore/core18/blob/master/static/lib/udev/rules.d/66-snapd-autoimport.rules
- UC16: **missing**

The file in UC22 seems already correct, in that it does not call unshare. So the only one left out is UC16, where we will add the udev rule that we currently have in snapd; after that, we will remove the rule from snapd and this bug for the desktop will be gone for good.

There is a remaining problem, however: the systemd-udevd.service in Ubuntu 22 disallows the mount operation, so we'll have to update the unit file to allow it. I think we'll also need to update the auto-import command itself, because it appears that all errors while accessing the devices are silenced, so we would have never known of this problem, if it wasn't for this bug report.

[Neoon (microlxc)]

@Nick

Your mentioned fix, doesn't work for me even after a reboot.
LXD with snapd still refuses to start the associated VM, but with no error messages anymore in syslog.

I tried this on Ubuntu 22.04, Kernel 5.15 with LXD 4.x LTS which previously worked fine, after a while it seems like something broke it.

[Sergey Ivanov (icegood1980)]

still present in 22.10. Newest ubuntu still work against old version of snap.

# probe for assertions, must run before udisks2
ACTION=="add", SUBSYSTEM=="block", KERNEL!="loop*", KERNEL!="ram*" \
    RUN+="/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/%k"

in my rule file

[Neoon (microlxc)]

"Fix" as suggested by the dev's would be upgrading to 5.x LTS with:

"snap refresh lxd --channel=5.0/stable"

Same error, however doesn't seem to affect anything anymore.

[Bernhard Riegler (riegler-b)]

Hi there, I see this with "lunar" Xubuntu 23.04

bernie@prod:~/Desktop$ journalctl -b |grep 'exit code 1'
Mai 30 17:16:45 prod systemd-udevd[246]: sda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda' failed with exit code 1.
Mai 30 17:16:45 prod systemd-udevd[234]: sr0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sr0' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[230]: sda2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda2' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[243]: sda11: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda11' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[232]: sda9: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda9' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[237]: sda7: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda7' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[227]: sda4: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda4' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[245]: sda10: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda10' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[236]: sda6: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda6' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[231]: sda3: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda3' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[246]: sda1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda1' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[234]: sda5: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda5' failed with exit code 1.
Mai 30 17:16:46 prod systemd-udevd[235]: sda8: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda8' failed with exit code 1.
bernie@prod:~/Desktop$ cd /etc
bernie@prod:/etc$ ls *rel*
lsb-release os-release
bernie@prod:/etc$ cat lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=23.04
DISTRIB_CODENAME=lunar
DISTRIB_DESCRIPTION="Ubuntu 23.04"
bernie@prod:/etc$ cat os-release
PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"
VERSION_CODENAME=lunar
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=lunar
LOGO=ubuntu-logo
bernie@prod:/etc$ uname -a
Linux prod 6.2.0-20-generic #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 6 07:48:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[Seth Arnold (seth-arnold)]

I have to imagine mardy isn't going to bother investigating further, so lets unsubscribe him. I'm hoping that will be enough for the snap team to see this bug again.

Thanks

[Seth Arnold (seth-arnold)]

(why is this file installed in desktop systems anyway?)

[Nathan Stratton Treadway (nathanst)]

Seeing this warning message in Jammy, with snapd 2.58+22.04.1 installed.

[ventsy velev (ventsyv)]

I saw this as well.

Linux kythera-Precision-5570 6.2.0-31-generic #31~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Aug 16 13:45:26 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

I deleted /lib/udev/rules.d/66-snapd-autoimport.rules and the issues seems to have resolved itself.

[Bernhard Riegler (riegler-b)]

good catch

I renamed the file '66-snapd-autoimport.rules' to 'OFF.66-snapd-autoimport.rules.OFF'
and rebooted.

root@prod:/lib/udev/rules.d# journalctl -b |grep 'exit code 1'
root@prod:/lib/udev/rules.d# echo $?
1

[Laurent Bonnaud (laurent-bonnaud)]

The error messages are still there in mantic:

# journalctl | grep unshare
Sep 24 10:14:08 vougeot (udev-worker)[1373]: nvme0n1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1' failed with exit code 1.
Sep 24 10:14:08 vougeot (udev-worker)[1373]: nvme0n1p1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1p1' failed with exit code 1.
Sep 24 10:14:08 vougeot (udev-worker)[1367]: nvme0n1p2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1p2' failed with exit code 1.

[Penny Chiu (pchiu)]

We also encountered this issue in Ubuntu 22.04 on NVIDIA Jetson Platform. What is the official fix and when will the fix be landed on Ubuntu 22.04?

[Curtis Nixon (mandle)]

Xubuntu 24.04

May 04 18:28:06 xxx (udev-worker)[494]: nvme0n1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1' failed with exit code 1.

trying
sudo mv /lib/udev/rules.d/66-snapd-autoimport.rules /lib/udev/rules.d/OFF.66-snapd-autoimport.rules

[Curtis Nixon (mandle)]

that worked.