Ubuntu Desktop boot hangs absent zeroconf packets and after avahi-daemon purge

[ebsf]

Our install procedures customarily airgap machines during installation, purge unnecessary and irrelevant packages such as avahi-daemon, ufw, netplan.io, ModemManager, network-manager, fprintd, etc., configure networking via systemd-networkd, and enable an iptables firewall to be in place before the network is up, all before exposing any machine to a network for further configuration.

The iptables rules drop all zeroconf and broadcast traffic for obvious security reasons. The kernel is typically configured to forward packets, so these DROP rules are in the mangle table's PREROUTING chain, to scrub them before reaching the filter table's INPUT or FORWARD chains. Presumably, this also scrubs such traffic to the loopback interface. INPUT, OUTPUT, and FORWARD rules ACCEPT suitable traffic before a default DROP rule in each case.

We have found the Ubuntu desktop 22.04 boot process to be especially fragile. System boot hangs every time, typically presenting as either an ordering cycle or a failure of partition mounts, which may be related. Occasionally, we note that dmesg.service and networkd-dispatcher.service fail by reason of time-out. Often, the last message in tty1 is Reached target Host and Network Name Lookups.

Over the course of weeks, we have debugged our install scripts and packet filtering and although the modality is unclear, the cause is an absence of zeroconf network traffic or the purge of the avahi-daemon package.

Curiously, none of this configuration has any effect and the boot process proceeds normally and as expected so long as a machine's Ethernet cables are unplugged. Once connected, attempts to upgrade a system (# apt update && apt upgrade) themselves hang, the machine reboots successfully, and then after dpkg --reconfigure -a, the attempted reboot hangs as before.

Ubuntu desktop 20.04 machines do not exhibit this behavior.

The expectation is that local processes would utilize d-bus or, if ip traffic somehow was necessary for local interprocess communication, that those processes would rely on name resolution other than broadcast traffic.

Alternatively, the expectation is that the necessity of deliberately opening this security vulnerability would be well and conspicuously documented, including identifying the processes, ports, protocols, sources, destinations, interfaces, sockets, and any IP or MAC address so that the traffic can be suitably filtered.

Persisting avahi-daemon and zeroconf is a non-starter.

Release: Ubuntu desktop 22.04 LTS

Version: gnome-shell 42.0-2ubuntu1

Expected behavior: Boot to gdm3 login prompt.

Actual behavior: Consistent boot hang.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: gnome-shell 42.0-2ubuntu1
ProcVersionSignature: Ubuntu 5.15.0-33.34-generic 5.15.30
Uname: Linux 5.15.0-33-generic x86_64
ApportVersion: 2.20.11-0ubuntu82
Architecture: amd64
CasperMD5CheckResult: pass
Date: Tue Jun 7 10:00:18 2022
DisplayManager: gdm3
GsettingsChanges:

InstallationDate: Installed on 2022-05-24 (14 days ago)
InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220419)
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 TERM=xterm-256color
 PATH=(custom, no user)
RelatedPackageVersions: mutter-common 42.0-3ubuntu2
SourcePackage: gnome-shell
UpgradeStatus: No upgrade log present (probably fresh install)

[Steve Beattie (sbeattie)]

Thanks for reporting this issue. I'm opening up it publicly since it would be useful for the people who work on the installer to see this.

[Daniel van Vugt (vanvugt)]

Thanks for the bug report.

Please:

1. Reproduce the hang again.

2. Reboot into recovery mode. You need to be quick to press Esc at the right time at the end of the BIOS screen.

3. Run:

   journalctl -b-1 > prevboot1.txt
   journalctl -b-2 > prevboot2.txt
   journalctl -b-3 > prevboot3.txt

4. Attach the resulting text files here.

[ebsf (eb-9)]

Sorry, just saw the e-mail notifications of the last two posts.

You will note if you read the bug report that the problem is a boot hang. No means exists to run any commands whatsoever, including journalctl.

Besides, the machine is long gone. I've probably installed and wiped Ubuntu 22.04 (chiefly Desktop but also Server) several dozens of times over a period of weeks. I haven't been too sentimental about retaining failed software.

I've given you enough breadcrumbs to both (a) lead you to the problem (which should be evident from the previously-attached dependency tree and other files) and (b) configure a machine to reproduce it. In fact, I would expect the dependency tree would be largely tantamount to what the logs would reveal.

My focus is on getting a production machine running, which has yet to occur, nearly three months after release. I really need to get back to my own business. If I have a few hours if ever Ubuntu 22.04 installs, I'll see about re-creating the problem. This has put my business back three months so far, and I have a lot of catch-up, so don't hold your breath. You're probably better off reading the attached files and recreating the failure on your end.

[ebsf (eb-9)]

The first question to ask, one would imagine, is what the file system mounts depend on or are ordered after, and of those antecedents, which depend on avahi-daemon or zeroconf traffic.

Another question would be whether the problem is caused by an inelegant or otherwise incomplete unmount during the preceding session's shutdown. If so, then the question becomes how to remove any consequent dependencies from the next session's boot order so the system boots in spite of any such inelegant shutdown.

[ebsf (eb-9)]

You're in luck. One of my partitions briefly exhibited the boot hang and I was able to capture the prevboot files. See attached.

The system booted to emergency mode, not rescue mode, and I remounted storage as rw. Curiously, without doing anything other than piping the journalctl output to the text files, the partition was able to reboot without hang. Those prevboot files also are attached.

On other 22.04 installations on other partitions, I have observed non-fatal failures cropping up on boot and shutdown/reboot/poweroff. The solution seems to be running `# depmod`, after which the failures and shutdown timeouts cease.

I suspect because of this, that the boot hangs I have documented in connection with avahi-daemon and fprintd here and in other posts and bug reports may be attributable to misordered kernel dependencies. I have no idea how to test this hypothesis or establish whether it is a consequence of corruption or a failure to update but perhaps this avenue may lead you to a cause.

[ebsf (eb-9)]

The interface permits only one attachment per comment, so the preceding all are related to the next preceding substantive comment.

[Launchpad Janitor (janitor)]

[Expired for Ubuntu because there has been no activity for 60 days.]