Comment 18 for bug 1797581

As Ryan I can not reproduce locally - hrm.

The crash in your log is the root-fs mount.

[ 22.524541] VFS: Cannot open root device "squash:http://172.16.99.2:5248/images/ubuntu/amd64/generic/bion" or unknown-block(0,0): error -6
[ 22.575588] Please append a correct "root=" boot option; here are the available partitions:
[ 22.583909] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Also we have to stick to exactly your values (one of the repros had a slightly different value)
  <memory unit='KiB'>2096128</memory>
  <currentMemory unit='KiB'>2096128</currentMemory>

I tried with the exact numbers above but "normal" cloud image boot is still ok.

I wonder if the kernel has an off by one error e.g. aligning the squash at the lowest 2G but with just this amount of memory choosing a place it would not fit.

We'd need to set up a local http and serve squashfs, to boot into that.
With some luck we can reproduce there and then eliminate libvirt and maas out of the equation.

Repro:
- I started off as Ryan did with a Cloud Image test via UVTool.
- Next I extracted the kernel+initrd from the guest to provide those from the host (as you do via PXE)
- installed nginx
- made initrd available on /var/www/html/boot-initrd (initrd.img-4.15.0-36-generic)
- made kernel available on /var/www/html/boot-kernel (vmlinuz-4.15.0-36-generic)
- The address of the Host on the libvirt net is 192.168.122.1, verify the guest can http from there
- get matching squash (see below for details)
- get an empty qemu disk via qemu-img like the type raw that maas uses
  sudo qemu-img create -f raw /var/lib/libvirt/images/empty-root.img 10G
- With that, modify the guest to use these kernel/initrd/sqashfs/empty-root

XML of the guest: http://paste.ubuntu.com/p/PhBn6n8VYH/

I have a dependency issue for my repro, that is IP being configured in /scripts/init-bottom after trying to mount squashfs in what seems /scripts/local-premount.
http://paste.ubuntu.com/p/RscHmQqFyY/
I can even fetch the squashfs from the initramfs, wouldn't you be affected by the same ordering issue? I need to find how you usually get around that to continue the repro that hopefully eventually helps to focus on the root cause.

I experimented a bit more and asked around on IRC.
But so far I can't get past the ordering issue that IP is initialized to late and due to that squash is failing.

-- Appendix --

Get Squash:
To continue I'd need the current squashfs instead of the disk image.
My uvtool spawned this for me:
$ uvt-simplestreams-libvirt --verbose query
release=bionic arch=amd64 label=daily (20181012)
So lets get the mathcing suqash URL and fetch that.
$ sstream-query --output-format="%(item_url)s" --no-verify http://cloud-images.ubuntu.com/daily arch=amd64 release=bionic label=daily ftype=squashfs version_name=20181012
http://cloud-images.ubuntu.com/daily/server/bionic/20181012/bionic-server-cloudimg-amd64.squashfs
$ sudo wget -O /var/www/html/squashfs http://cloud-images.ubuntu.com/daily/server/bionic/20181012/bionic-server-cloudimg-amd64.squashfs

Note: That setup is available on server horsea