grub lvmid disk not found using mdraid+lvm

[Derek Nicol]

Doing a clean install of ubuntu server 18.10 alt install, with gpt -> mdraid -> lvm -> btrfs. Installer completes successfully, grub-install completes successfully, but upon first boot I receive a grub "lvmid disk not found" error and thrown into the recovery console.
I know this setup is complex and perhaps not ideal, that aside, I've had it work in the past without issues or modification after installer.

I have tried the following, all resulting in the same lvmid not found error in grub:
- making a separate /boot LV with ext4 instead of btrfs
- setting root=UUID=xxx in /etc/default grub
- preloading lvm/mdraid modules in /etc/default/grub
- not using btrfs subvolumes in root (although should be an issue as ubuntu creates them by default)
- using raid5 only (not mixing raid1/5 for PV)
- running update-initramfs from chroot

Also, a little background as to why I'm doing the partitions/raid/lvm as I am can be found here: https://serverfault.com/questions/272776/how-to-do-the-equivalent-of-synology-hybrid-raid-on-linux-myself

Setup is as follows:
=== /ets/fstab
/dev/mapper/vg1-root / btrfs noatime,subvol=@ 0 1
/dev/mapper/vg1-root /home btrfs noatime,subvol=@home 0 2
/dev/mapper/vg1-swap none swap sw 0 0

=== blkid
/dev/sda1: PARTUUID="32e4d29c-844f-4793-8eca-f5656432ead5"
/dev/sda2: UUID="ad79a45d-4609-b2f3-b2d2-e97802031740" UUID_SUB="806397cd-bb32-3e05-ad3a-6888c075f326" LABEL="jarvis:0" TYPE="linux_raid_member" PARTUUID="22b13abe-db68-4bc3-b1b3-ce1786bbee4d"
/dev/sda3: UUID="649390e4-1cae-f25c-0792-89590d5f1853" UUID_SUB="de6a5629-6dc2-3f02-f826-9dc830897997" LABEL="jarvis:1" TYPE="linux_raid_member" PARTUUID="dd2048a5-ea24-4d9e-a5cf-d66dc808cebc"
/dev/sda4: UUID="b6863d97-ea04-ab1c-b0d5-aa2d44eed9b1" UUID_SUB="c9c3072b-3d6e-8b6e-3079-770f2d0be910" LABEL="jarvis:2" TYPE="linux_raid_member" PARTUUID="f040a326-c446-4330-ba79-5c42d7aee6f2"
/dev/sdc1: PARTUUID="2c8cd1b1-d237-4bb4-940d-496634fc16a7"
/dev/sdc2: UUID="ad79a45d-4609-b2f3-b2d2-e97802031740" UUID_SUB="39df4b32-dd92-754c-ecf2-9bccb1985fe1" LABEL="jarvis:0" TYPE="linux_raid_member" PARTUUID="291de3df-58be-4054-ab90-e65af7d9ae03"
/dev/sdc3: UUID="649390e4-1cae-f25c-0792-89590d5f1853" UUID_SUB="4bcf6186-4284-3254-503f-9a937ccc1a70" LABEL="jarvis:1" TYPE="linux_raid_member" PARTUUID="cdf51368-faf0-4455-a286-8ab3330d720e"
/dev/sdc4: UUID="b6863d97-ea04-ab1c-b0d5-aa2d44eed9b1" UUID_SUB="a7e1157a-69d1-708f-b28b-b01ec43f5de7" LABEL="jarvis:2" TYPE="linux_raid_member" PARTUUID="1eff3990-5a32-4c9e-a622-b410f2285b51"
/dev/sdd1: PARTUUID="7c247a87-1177-4da6-b52b-612fdc7df0b2"
/dev/sdd2: UUID="649390e4-1cae-f25c-0792-89590d5f1853" UUID_SUB="3bd9c6a8-54d9-5fbe-509b-885c5655cf9c" LABEL="jarvis:1" TYPE="linux_raid_member" PARTUUID="9c61a3d2-ecb7-4f80-87b6-f590183ab898"
/dev/sde1: PARTUUID="3a3342c7-366f-41ef-b3d3-0d0c3a349014"
/dev/sde2: UUID="ad79a45d-4609-b2f3-b2d2-e97802031740" UUID_SUB="f2441bfc-7a4a-0595-43f9-bfb0695ba35a" LABEL="jarvis:0" TYPE="linux_raid_member" PARTUUID="8ae879b1-b880-428c-9967-14615346382f"
/dev/md127: UUID="H10Yrl-SAkb-0bAE-OYpG-pBhQ-iz0U-dKq427" TYPE="LVM2_member"
/dev/md126: UUID="Sm53JC-Iz2u-eAf9-NYbb-GFYI-7DKh-al5p6v" TYPE="LVM2_member"
/dev/md125: UUID="vhvdNY-Mvgv-QM7s-EcD7-Wzr9-I1D4-abcZ9z" TYPE="LVM2_member"
/dev/mapper/vg1-swap: UUID="91c45d4d-6f9e-49ab-8a6d-476f18afa133" TYPE="swap"
/dev/mapper/vg1-root: LABEL="root" UUID="aa02e081-d12c-4b5e-9dbd-8613ba77ce26" UUID_SUB="55f643de-4203-41d5-b042-f765e57744f0" TYPE="btrfs"

=== lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 819.5M 1 loop
sda 8:0 0 3.7T 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1.8T 0 part
│ └─md127 9:127 0 3.7T 0 raid5
│ ├─vg1-swap 253:0 0 3.7G 0 lvm
│ └─vg1-root 253:1 0 6.4T 0 lvm /
├─sda3 8:3 0 931.2G 0 part
│ └─md125 9:125 0 1.8T 0 raid5
│ └─vg1-root 253:1 0 6.4T 0 lvm /
└─sda4 8:4 0 931.2G 0 part
  └─md126 9:126 0 931.1G 0 raid1
    └─vg1-root 253:1 0 6.4T 0 lvm /
sdb 8:16 1 7.5G 0 disk
├─sdb1 8:17 1 871M 0 part
└─sdb2 8:18 1 1.4M 0 part
sdc 8:32 0 3.7T 0 disk
├─sdc1 8:33 0 1M 0 part
├─sdc2 8:34 0 1.8T 0 part
│ └─md127 9:127 0 3.7T 0 raid5
│ ├─vg1-swap 253:0 0 3.7G 0 lvm
│ └─vg1-root 253:1 0 6.4T 0 lvm /
├─sdc3 8:35 0 931.2G 0 part
│ └─md125 9:125 0 1.8T 0 raid5
│ └─vg1-root 253:1 0 6.4T 0 lvm /
└─sdc4 8:36 0 931.2G 0 part
  └─md126 9:126 0 931.1G 0 raid1
    └─vg1-root 253:1 0 6.4T 0 lvm /
sdd 8:48 0 931.5G 0 disk
├─sdd1 8:49 0 1M 0 part
└─sdd2 8:50 0 931.2G 0 part
  └─md125 9:125 0 1.8T 0 raid5
    └─vg1-root 253:1 0 6.4T 0 lvm /
sde 8:64 0 1.8T 0 disk
├─sde1 8:65 0 1M 0 part
└─sde2 8:66 0 1.8T 0 part
  └─md127 9:127 0 3.7T 0 raid5
    ├─vg1-swap 253:0 0 3.7G 0 lvm
    └─vg1-root 253:1 0 6.4T 0 lvm /

=== mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active (auto-read-only) raid5 sdc3[1] sda3[0] sdd2[2]
      1952665600 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
        resync=PENDING
      bitmap: 8/8 pages [32KB], 65536KB chunk

md126 : active (auto-read-only) raid1 sdc4[1] sda4[0]
      976332800 blocks super 1.2 [2/2] [UU]
        resync=PENDING
      bitmap: 8/8 pages [32KB], 65536KB chunk

md127 : active (auto-read-only) raid5 sde2[2] sdc2[1] sda2[0]
      3905789952 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
        resync=PENDING
      bitmap: 8/15 pages [32KB], 65536KB chunk

=== dmsetup ls
vg1-swap (253:0)
vg1-root (253:1)

=== pvs
  PV VG Fmt Attr PSize PFree
  /dev/md125 vg1 lvm2 a-- <1.82t 0
  /dev/md126 vg1 lvm2 a-- 931.10g 0
  /dev/md127 vg1 lvm2 a-- <3.64t 0

=== vgs
  VG #PV #LV #SN Attr VSize VFree
  vg1 3 2 0 wz--n- <6.37t 0

=== lvs
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  root vg1 -wi-ao---- 6.36t
  swap vg1 -wi-a----- 3.72g

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: grub2-common 2.02+dfsg1-5ubuntu8.2
Uname: Linux 4.19.24-1-lts x86_64
NonfreeKernelModules: cfg80211 rfkill 8021q garp mrp stp llc btrfs dm_mod raid456 async_raid6_recov async_memcpy libcrc32c crc32c_generic async_pq async_xor xor snd_hda_codec_realtek snd_hda_codec_generic powernow_k8 edac_mce_amd async_tx kvm_amd ccp rng_core ppdev snd_hda_intel snd_hda_codec kvm raid6_pq raid1 md_mod joydev mousedev input_leds snd_hda_core snd_hwdep evdev snd_pcm pcspkr irqbypass snd_timer parport_pc sp5100_tco parport mac_hid snd k8temp i2c_piix4 soundcore ip_tables x_tables overlay squashfs loop isofs uas usb_storage hid_microsoft hid_generic usbhid hid sd_mod radeon ata_generic pata_acpi i2c_algo_bit drm_kms_helper pata_atiixp syscopyarea sysfillrect ahci sysimgblt fb_sys_fops libahci ttm r8169 drm realtek libphy libata agpgart ohci_pci ehci_pci scsi_mod ehci_hcd ohci_hcd
ApportVersion: 2.20.10-0ubuntu13
Architecture: amd64
Date: Mon Mar 11 11:25:02 2019
InstallationDate: Installed on 2019-03-11 (0 days ago)
InstallationMedia: Ubuntu-Server 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.2)
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: grub2
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John Paul Morrison (jmorrison-w)]

Same problem in 18.04.2 and the super grub disk I had on hand.

ls is rescue mode doesn't show the boot lvm but it detects other drives