Rebooting bionic fails with hardware error on ProLiant DL360 Gen9

[Andres Rodriguez]

From an ephemeral environment in an HPGen9, the machine fails to reboot when the ephemeral environment is Bionic (it does not fail in Xenial).

Full log from the moment the 'sudo reboot' was executed: https://pastebin.ubuntu.com/p/hyD6VY7CbW/

The actual failure is:

[ OK ] Stopped Remount Root and Kernel File Systems.
[ OK ] Reached target Shutdown.
[ OK ] Reached target Final Step.
         Starting Reboot...
[ OK ] Stopped Monitoring of LVM2 mirrors,…sing dmeventd or progress polling.
         Stopping LVM2 metadata daemon...
[ OK ] Stopped LVM2 metadata daemon.
[ 257.160527] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 1
[ 257.160530] {1}[Hardware Error]: event severity: fatal
[ 257.160533] {1}[Hardware Error]: Error 0, type: fatal
[ 257.160534] {1}[Hardware Error]: section_type: PCIe error
[ 257.160536] {1}[Hardware Error]: port_type: 4, root port
[ 257.160541] {1}[Hardware Error]: version: 1.16
[ 257.160544] {1}[Hardware Error]: command: 0x6010, status: 0x0143
[ 257.160546] {1}[Hardware Error]: device_id: 0000:00:01.0
[ 257.160547] {1}[Hardware Error]: slot: 0
[ 257.160548] {1}[Hardware Error]: secondary_bus: 0x03
[ 257.160550] {1}[Hardware Error]: vendor_id: 0x8086, device_id: 0x6f02
[ 257.160552] {1}[Hardware Error]: class_code: 040600
[ 257.160554] {1}[Hardware Error]: bridge: secondary_status: 0x2000, control: 0x0003
[ 257.160556] {1}[Hardware Error]: Error 1, type: fatal
[ 257.160557] {1}[Hardware Error]: section_type: PCIe error
[ 257.160559] {1}[Hardware Error]: port_type: 4, root port
[ 257.160560] {1}[Hardware Error]: version: 1.16
[ 257.160562] {1}[Hardware Error]: command: 0x6010, status: 0x0143
[ 257.160564] {1}[Hardware Error]: device_id: 0000:00:01.0
[ 257.160565] {1}[Hardware Error]: slot: 0
[ 257.160566] {1}[Hardware Error]: secondary_bus: 0x03
[ 257.160567] {1}[Hardware Error]: vendor_id: 0x8086, device_id: 0x6f02
[ 257.160568] {1}[Hardware Error]: class_code: 040600
[ 257.160569] {1}[Hardware Error]: bridge: secondary_status: 0x2000, control: 0x0003
[ 257.160570] Kernel panic - not syncing: Fatal hardware error!
[ 257.200091] Kernel Offset: 0x25400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 257.815850] ERST: [Firmware Warn]: Firmware does not respond in time.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 33 Apr 6 17:40 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=username/amd64/ga-18.04/bionic/daily/boot-kernel nomodeset ro root=squash:http://10.245.136.6:5248/images/username/amd64/ga-18.04/bionic/daily/squashfs ip=::::hostname:BOOTIF ip6=off overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:{datasource_list: [MAAS]}end_cc cloud-config-url=http://10.245.136.6:5240/MAAS/metadata/latest/by-id/hqsxdc/?op=get_preseed apparmor=0 log_host=10.245.136.6 log_port=514 --- console=ttyS1,115200n8 BOOTIF=01-94:18:82:7a:82:c8
ProcVersionSignature: User Name 4.15.0-13.14-generic 4.15.10
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-13-generic N/A
 linux-backports-modules-4.15.0-13-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic uec-images
Uname: Linux 4.15.0-13-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 09/13/2016
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.board.name: ProLiant DL360 Gen9
dmi.board.vendor: HP
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP89:bd09/13/2016:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL360 Gen9
dmi.sys.vendor: HP
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 6 17:53 seq
 crw-rw---- 1 root audio 116, 33 Apr 6 17:53 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-13-generic.efi.signed root=UUID=40d154cc-a366-4fb0-840e-01067637050e ro console=ttyS1,115200n8
ProcVersionSignature: User Name 4.15.0-13.14-generic 4.15.10
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-13-generic N/A
 linux-backports-modules-4.15.0-13-generic N/A
 linux-firmware 1.173
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic uec-images
Uname: Linux 4.15.0-13-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 09/13/2016
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.board.name: ProLiant DL360 Gen9
dmi.board.vendor: HP
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP89:bd09/13/2016:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL360 Gen9
dmi.sys.vendor: HP
---
ApportVersion: 2.20.9-0ubuntu4
Architecture: amd64
DistroRelease: Ubuntu 18.04
Package: linux (not installed)
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
Tags: bionic uec-images
Uname: Linux 4.16.0-041600-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True
---
ApportVersion: 2.20.9-0ubuntu4
Architecture: amd64
DistroRelease: Ubuntu 18.04
Package: linux (not installed)
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
Tags: bionic uec-images
Uname: Linux 4.16.0-041600-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1761796

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.16 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

Test both -proposed & 4.16 kernels, and the issue persists.

[Andres Rodriguez (andreserl)]

collecting logs with mainline kernel.

[Andres Rodriguez (andreserl)]

apport information

[Andres Rodriguez (andreserl)]

apport information

[Joseph Salisbury (jsalisbury)]

Can you see if this bug happens with the Artful kernel? It is available from:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/14537150

Since this bug does not happen with Xenial, we can perform a kernel bisect to identify the commit that introduced the bug. We first need to identify the last kernel version that did not have bug and the first kernel version that does.

[Andres Rodriguez (andreserl)]

This issue is not present in Artful. This issue is only present in Bionic.

[Joseph Salisbury (jsalisbury)]

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue.

Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug:

4.14-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc1/
4.14 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14/
4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/

You don't have to test every kernel, just up until the kernel that first has this bug.

Thanks in advance!

[Andres Rodriguez (andreserl)]

Finally got around to test this. ANd this are my findings:

1. Installed 4.15-rc1 above, rebooted the machine, and did not see the issue.

ubuntu@just-pika:~$ uname -a
Linux just-pika 4.15.0-041500rc1-generic #201711262030 SMP Mon Nov 27 01:32:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@just-pika:~$ sudo reboot

2. I rebooted for a second time to see if the issue was present, but did not see the issue.

3. On the third attempt. I installed [1]. After I installed, I proceeded to reboot the machine. ANd this is when I saw the issue, even though the [1] kernel wasn't the running kernel.

[1]: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15/

[Joseph Salisbury (jsalisbury)]

Just to confirm, 4.15-rc1 does not have the issue, but 4.15 final does? If that is the case, we can try to narrow it down further. Could you test v4.15-rc9:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc9/

[Andres Rodriguez (andreserl)]

So I've tested all the way from rc2 -> rc7, and confirm that rc7 is the issue. In two different machines:

machine1: running kernel was rc3
machine2: running kernel was rc4

In both cases, after install of rc7, and reboot, I see the issue. Logs for the latest:

https://pastebin.ubuntu.com/p/yVTWBRqXSC/

[Joseph Salisbury (jsalisbury)]

I started a kernel bisect between v4.15-rc6 and v4.15-rc7. The kernel bisect will require testing of about 6 test kernels.

I built the first test kernel, up to the following commit:
d8887f1c7289848e74c92bd4322789a9cd7de699

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1761796

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Andres Rodriguez (andreserl)]

@Joseph:
Issue still exists:

ubuntu@superb-whale:~$ uname -a
Linux superb-whale 4.15.0-041500rc6-generic #201805171959 SMP Thu May 17 20:02:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@superb-whale:~$ sudo reboot

===

Stopping LVM2 metadata daemon... [14/5352]
[ OK ] Stopped LVM2 metadata daemon.
[ 107.570384] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 1
[ 107.570386] {1}[Hardware Error]: event severity: fatal
[ 107.570388] {1}[Hardware Error]: Error 0, type: fatal
[ 107.570389] {1}[Hardware Error]: section_type: PCIe error
[ 107.570391] {1}[Hardware Error]: port_type: 4, root port

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
abb62c46d4949d44979fa647740feff3f7538799

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1761796

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Andres Rodriguez (andreserl)]

Hi Joseph,

Sorry for such a late reply, but with all the recent business travel I've been unable to continue working on this.

That said, now that things are calmer I can continue, however, I don't have easy access the hardware in question and it is very time consuming getting access to this while blocking all the other work that happens with this hardware.

Is there a way to have all those kernels available so I can test things in one go?

[Joseph Salisbury (jsalisbury)]

The problem with having the kernels pre-built is the bisect needs to know whether a test kernel has the bug or not. Based on that input, the bisect decides the next SHA1 to test.

Since it's been some time, maybe it would be best to first test the latest mainline kernel again.
 That will tell us if the issue has already been fixed upstream. The latest mainline kernel is:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18/

[Andres Rodriguez (andreserl)]

Maybe I did something wrong, but this kernel (4.18) causes a kernel panic. The process I followed was same as always, to wget the *.deb, dpkg -i *.deb && update-grub at the end to ensure this kernel is used:

[ 6.538003] evm: security.SMACK64
[ 6.539108] evm: security.SMACK64EXEC
[ 6.540387] evm: security.SMACK64TRANSMUTE
[ 6.541798] evm: security.SMACK64MMAP
[ 6.543023] evm: security.apparmor
[ 6.544209] evm: security.ima
[ 6.545253] evm: security6.668405] usb 2-1: New USB device found, idVendor=8087, idProduct=8002, bcdDevice= 0.05
[ 6.885734] evm: HMAC attrs: 0x1
[ 6.886472] Magic number: 14:225:302
[ 7.048969] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 7.050067] tty tty31: hash matches
[ 7.054948] memory memory160: hash matches
[ 7.056460] rtc_cmos 00:00: setting system clock to 2018-08-14 23:16:59 UTC (1534288619)
[ 7.056552] hub 2-1:1.0: USB hub found
[ 7.060431] hub 2-1:1.0: 8 ports detected
[ 7.063051] md: Waiting for all devices to be available before autodetect
[ 7.093858] md: If you don't use raid, use raid=noautodetect
[ 7.118684] md: Autodetecting RAID arrays.
[ 7.137860] md: autorun ...
[ 7.150409] md: ... autorun DONE.
[ 7.165346] VFS: Cannot open root device "sda1" or unknown-block(0,0): error -6
[ 7.197757] Please append a correct "root=" boot option; here are the available partitions:
[ 7.235391] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 7.273379] CPU: 6 PID: 1 Comm: swapper/0 Not tainted 4.18.0-041800-generic #201808122131
[ 7.310122] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 01/22/2018
[ 7.544633] Call Trace:
[ 7.555562] dump_stack+0x63/0x83
[ 7.570510] panic+0xe7/0x247
[ 7.583414] mount_block_root+0x231/0x2d8
[ 7.601354] ? do_early_param+0x95/0x95
[ 7.618598] mount_root+0x7c/0x7f
[ 7.634231] prepare_namespace+0x139/0x171
[ 7.653611] kernel_init_freeable+0x236/0x261
[ 7.673689] ? rest_init+0xb0/0xb0
[ 7.688884] kernel_init+0xe/0x104
[ 7.704165] ret_from_fork+0x35/0x40
[ 7.720293] Kernel Offset: 0x29e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 7.772913] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
[ 7.815580] ------------[ cut here ]------------
[ 7.836276] sched: Unexpected reschedule of offline CPU#0!
[ 7.860432] WARNING: CPU: 6 PID: 1 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3a/0x40
[ 7.902510] Modules linked in:
[ 7.916373] CPU: 6 PID: 1 Comm: swapper/0 Not tainted 4.18.0-041800-generic #201808122131
[ 7.953258] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 01/22/2018
[ 8.089242] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 8.113868] Code: cf 62 01 73 17 48 8b 05 94 8c 17 01 be fd 00 00 00 48 8b 40 30 e8 06 a1 ba 00 5d c3 89 fe 48 c7 c7 58 c1 ed ab e8 48 41 03 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 20 65 48
[ 8.198752] RSP: 0000:ffff8cc61f383e08 EFLAGS: 00010086
[ 8.222224] RAX: 0000000000000000 RBX: 0000000000000003 RCX:...

[Joseph Salisbury (jsalisbury)]

Thanks for testing. The panic was not caused from anything you did. That is a new bug in the 4.18 kernel. We will have to monitor this new bug, since Cosmic will eventually be 4.18 based. We should also test the latest Cosmic kernel, which is 4.17 based. It is available from:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+build/15227897

[Andres Rodriguez (andreserl)]

Tested the 4.17 kernel, and this is the issue I now see:

[...]
[ 17.340040] raid6: sse2x1 xor() 2828 MB/s
[ 17.408036] raid6: sse2x2 gen() 7445 MB/s
[ 17.472037] raid6: sse2x2 xor() 5425 MB/s
[ 17.540032] raid6: sse2x4 gen() 10259 MB/s
[ 17.604037] raid6: sse2x4 xor() 6286 MB/s
[ 17.668034] raid6: avx2x1 gen() 14089 MB/s
[ 17.732034] raid6: avx2x1 xor() 9346 MB/s
[ 17.796035] raid6: avx2x2 gen() 16758 MB/s
[ 17.860033] raid6: avx2x2 xor() 9779 MB/s
[ 17.924034] raid6: avx2x4 gen() 19079 MB/s
[ 17.988036] raid6: avx2x4 xor() 11383 MB/s
[ 18.007313] raid6: using algorithm avx2x4 gen() 19079 MB/s
[ 18.031519] raid6: .... xor() 11383 MB/s, rmw enabled
[ 18.054148] raid6: using avx2x2 recovery algorithm
[ 18.076927] xor: automatically using best checksumming function avx
[ 18.109754] async_tx: api initialized (async)
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... [ 18.199553] Btrfs loaded, crc32c=crc32c-intel
Scanning for Btrfs filesystems
done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... mdadm: No devices listed in conf file were found.
done.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: error opening /dev/md?*: No such file or directory
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
done.
Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=bd0dbca3-60b8-474e-ac82-96f1b9ff9cc2 does not exist. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell ...

[Joseph Salisbury (jsalisbury)]

I think our best bet is to perform the bisect. When you have a chance, can you test the kernel posted in comment #34? I'll build the next test kernel based on that test result.