Boot stalls for 90 seconds on mdadm

[Johan Ehnberg]

On a fresh Ubuntu Xenial 16.04 LTS Server install, the boot stalls for exactly 90 seconds when using a fairly complex partitioning setup. After this, everything works as expected.

The partitioning is as follows:
/sda GPT
/sda1 ESP (/boot/efi)
/sda2 raid1 md0 ext4 (/boot)
/sda3 raid1 md1 luks,passphrase ext4 (/)
/sda4 raid1 md2 luks,key zfs
/sdb GPT
/sdb1 ESP (copy of /boot/efi)
/sdb2 raid1 md0 ext4 (/boot)
/sdb3 raid1 md1 luks,passphrase ext4 (/)
/sdb4 raid1 md2 luks,key zfs
/sdc GPT
/sdc1 luks,key xfs
/sdd GPT
/sdd1 luks,key xfs
/sdd2 luks,random swap

This may be an installer or mdadm bug instead - I have not been able to exactly determine the cause of the bug.

I have several identical servers, all showing the same symptom, so it should not be due to broken hardware (although design flaws are on the table).

The key seems to be in these lines from journalctl:
...
May 16 14:06:08 molnix kernel: bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
May 16 14:06:08 molnix kernel: device eno1 entered promiscuous mode
May 16 14:06:08 molnix systemd-udevd[689]: Process '/lib/udev/hdparm' failed with exit code 5.
May 16 14:06:08 molnix systemd-udevd[701]: Process '/lib/udev/hdparm' failed with exit code 5.
May 16 14:06:08 molnix systemd-udevd[707]: Process '/lib/udev/hdparm' failed with exit code 5.
May 16 14:06:08 molnix systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
May 16 14:06:08 molnix systemd-udevd[701]: Process '/sbin/mdadm --incremental /dev/sda2 --offroot' failed with exit code 1.
May 16 14:06:08 molnix systemd-udevd[707]: Process '/sbin/mdadm --incremental /dev/sda3 --offroot' failed with exit code 1.
May 16 14:06:08 molnix systemd[1]: Found device Samsung_SSD_850_EVO_250GB EFI\x20System\x20Partition.
May 16 14:06:08 molnix systemd-udevd[708]: Process '/sbin/mdadm --incremental /dev/sda4 --offroot' failed with exit code 1.
May 16 14:06:08 molnix systemd-udevd[689]: Process '/sbin/mdadm --incremental /dev/sdb3 --offroot' failed with exit code 1.
May 16 14:06:08 molnix systemd[1]: Reached target Local File Systems (Pre).
May 16 14:06:08 molnix systemd-udevd[701]: Process '/sbin/mdadm --incremental /dev/sdb4 --offroot' failed with exit code 1.
May 16 14:06:08 molnix systemd-udevd[699]: Process '/sbin/mdadm --incremental /dev/sdb2 --offroot' failed with exit code 1.
May 16 14:06:08 molnix systemd-udevd[706]: Process '/lib/udev/hdparm' failed with exit code 5.
May 16 14:06:08 molnix kernel: ipmi_si IPI0001:00: The BMC does not support clearing the recv irq bit, compensating, but the BMC needs to be fixed.
May 16 14:06:08 molnix kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x002a7c, prod_id: 0x086d, dev_id: 0x20)
...

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: systemd 229-4ubuntu5
ProcVersionSignature: Ubuntu 4.4.0-22.39-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
Date: Mon May 16 14:17:31 2016
InstallationDate: Installed on 2016-05-15 (0 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
MachineType: Supermicro Super Server
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic.efi.signed root=UUID=0ca65308-3f70-4231-8f5f-e68583a6d5b3 ro
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /lib/systemd/system/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
 [EXTENDED] /lib/systemd/system/rc-local.service → /lib/systemd/system/rc-local.service.d/debian.conf

 2 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/04/2016
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.0c
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: X10SDV-4C-TLN4F
dmi.board.vendor: Supermicro
dmi.board.version: 2.00
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 0123456789
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1.0c:bd01/04/2016:svnSupermicro:pnSuperServer:pvr0123456789:rvnSupermicro:rnX10SDV-4C-TLN4F:rvr2.00:cvnSupermicro:ct17:cvr0123456789:
dmi.product.name: Super Server
dmi.product.version: 0123456789
dmi.sys.vendor: Supermicro

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[jhansonxi (jhansonxi)]

May be related to bug #1568097 which had the same messages.