sytemd-udev package upgrade breaks if udev is masked

[Simon Fels]

The Anbox Cloud LXD images are based of the standard Ubuntu cloud images for LXD but have the systemd-udevd.service unit masked (`systemctl mask systemd-udevd.service`) as we do not need udev.

With the latest security patch on Ubuntu 22.04 upgrading the udev package fails:

...
Mar 07 21:33:48 ams-cg3qqttdjmctnqegcqag sh[753]: left to upgrade {'udev', 'libudev1'}
Mar 07 21:33:48 ams-cg3qqttdjmctnqegcqag sh[753]: applying set ['udev', 'libudev1']
Mar 07 21:33:48 ams-cg3qqttdjmctnqegcqag sh[753]: Log ended: 2023-03-07 21:33:47
Mar 07 21:33:48 ams-cg3qqttdjmctnqegcqag sh[753]: Log started: 2023-03-07 21:33:48
Mar 07 21:33:48 ams-cg3qqttdjmctnqegcqag sh[753]: [614B blob data]
Mar 07 21:33:48 ams-cg3qqttdjmctnqegcqag sh[753]: Preparing to unpack .../udev_249.11-0ubuntu3.7_arm64.deb ...
Mar 07 21:33:48 ams-cg3qqttdjmctnqegcqag sh[753]: Unpacking udev (249.11-0ubuntu3.7) over (249.11-0ubuntu3.6) ...
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag sh[753]: Preparing to unpack .../libudev1_249.11-0ubuntu3.7_arm64.deb ...
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag sh[753]: Unpacking libudev1:arm64 (249.11-0ubuntu3.7) over (249.11-0ubuntu3.6) ...
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag sh[753]: Setting up libudev1:arm64 (249.11-0ubuntu3.7) ...
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag sh[753]: Setting up udev (249.11-0ubuntu3.7) ...
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag systemd[1]: Reloading.
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag sh[753]: Failed to restart udev.service: Unit udev.service failed to load properly, please adjust/correct and reload service manager: File exists
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag sh[753]: See system logs and 'systemctl status udev.service' for details.
Mar 07 21:33:49 ams-cg3qqttdjmctnqegcqag sh[753]: invoke-rc.d: initscript udev, action "restart" failed.

We can workaround this by doing a patch release to our users next week but it would be a lot better if the package scripts check if the service unit is masked and if it is avoid restarting udev.service

The problem isn't introduced with the 249.11-0ubuntu3.7 but should exist for longer already. It just became a problem for us due to the security patch rolling in via unattended-upgrades

[Mauricio Faria de Oliveira (mfo)]

Hi Simon,

Thanks for the detailed report!

For testing purposes, this can be reproduced this with standard LXD images by masking the systemd-udev.service unit, as you described.

Could you please confirm if from the Anbox perspective, for a workaround, is it OK to temporatily unmask/start the unit (due to restart), then upgrade the package, and stop/mask again?

For example:

lxc launch ubuntu:jammy jammy-udev-masked
lxc exec jammy-udev-masked -- su - ubuntu

$ dpkg -s udev | grep Version:
Version: 249.11-0ubuntu3.6

sudo systemctl mask systemd-udevd.service
sudo systemctl stop systemd-udevd.service

sudo apt update && sudo apt install -y udev

...
Setting up udev (249.11-0ubuntu3.7) ...
Failed to restart udev.service: Unit udev.service failed to load properly, please adjust/correct and reload service manager: File exists
See system logs and 'systemctl status udev.service' for details.
invoke-rc.d: initscript udev, action "restart" failed.
○ udev.service
     Loaded: error (Reason: Unit udev.service failed to load properly, please adjust/correct and reload service manager: File exists)
     Active: inactive (dead)
Warning: The unit file, source configuration file or drop-ins of udev.service changed on disk. Run 'systemctl daemon-reload' to reload units.
dpkg: error processing package udev (--configure):
 installed udev package post-installation script subprocess returned error exit status 1
dmesg: read kernel buffer failed: Operation not permitted
...

sudo systemctl unmask systemd-udevd.service
sudo apt install -f

sudo systemctl mask systemd-udevd.service
sudo systemctl stop systemd-udevd.service

[Simon Fels (morphis)]

Hey Mauricio,

> Could you please confirm if from the Anbox perspective, for a
> workaround, is it OK to temporatily unmask/start the unit
> (due to restart), then upgrade the package, and stop/mask again?

It's not that simple. We have prebuilt images we ship to users which run through an automated bootstrap process (see https://anbox-cloud.io/docs/exp/applications#bootstrap). As part of that process we apply pending security patches to ensure the images are up to date. This process fails with the latest udev update but it should have failed with older udev versions as well. It only became a problem now due to bad timing (us not having released a new image and the security patch being rolled out). We will ship a new patch release next Wednesday (Mar 15) which will then include the updated udev package and also a tweak to the automatic process to prevent this from happening again.

For our users we have a simple knob in our management service to turn off the process of applying pending updates to mitigate the situation till the we roll out new images next week.

[Mauricio Faria de Oliveira (mfo)]

Hey Simon. Understood, thanks for the clarification!