udev rule causing boot delay when rsyslog is not installed

[Gauthier Jolly]

This udev rule: /lib/udev/rules.d/67-azure-console.rules is wrong because it refers to the syslog group that doesn't exist on our images. This wasn't causing any particular issue before Lunar but know it's causing "systemd daemon-reload" to take 1min to run.

This is a big issue because "systemd daemon-reload" is called during boot and when snaps are installed.

Reproduce:
 1. start a minimal Lunar VM on Azure: az vm create --ssh-key-value $SSH_KEY_PATH --admin-username ubuntu -n jammy-minimal -g test-minimal --image 'Canonical:0001-com-ubuntu-minimal-jammy:minimal-22_04-lts-gen2:latest' --size Standard_D2as_v5
 2. run `time systemctl daemon-reload`: it will take 1min
 3. run `groupadd syslog`
 4. run `time systemctl daemon-reload`: it will take less than a second to run

It's not a problem on non-minimal images because rsyslog is installed and the group is present.

[Gauthier Jolly (gjolly)]

I don't think this udev rule is doing anything even on systems where syslog is present. I checked and /dev/console still belongs to "root:tty".

[Gauthier Jolly (gjolly)]

This issue seems to be directly related to this bug: https://bugs.launchpad.net/netplan/+bug/1999178.
Running "udevadm trigger" or "udevadm test /dev/console" just throws a warning message but does not block of a minute:

  /usr/lib/udev/rules.d/67-azure-console.rules:2 Unknown group 'syslog', ignoring

However, running "systemctl daemon-reload" with systemd debug logs enabled clearly shows the one minute delay being triggered by the netplan generator:

  Apr 12 11:15:24 lunar-minimal systemd[724]: /usr/lib/systemd/system-generators/systemd-sysv-generator succeeded.
  Apr 12 11:16:24 lunar-minimal systemd[724]: /usr/lib/systemd/system-generators/netplan succeeded.

My conclusion for now is that the bug in netplan is only triggered if a udev rule cannot be satisfied. This is why we are only seeing this issue with Minimal (no syslog group) Azure (udev rule present) Lunar (bad version of netplan) images.

[Gauthier Jolly (gjolly)]

We need to discuss that a bit further but I think it is safe to simply remove this udev rule.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "walinuxagent.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Łukasz Zemczak (sil2100)]

Sponsored for lunar. Will accept it - should migrate by itself shortly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package walinuxagent - 2.2.46-0ubuntu7

---------------
walinuxagent (2.2.46-0ubuntu7) lunar; urgency=medium

  * Remove udev rule for serial console (LP: #2016012)

 -- Gauthier Jolly <email address hidden> Thu, 13 Apr 2023 13:58:02 +0200