Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to kernel panic

Bug #1797581 reported by Andres Rodriguez on 2018-10-12
46
This bug affects 6 people
Affects Status Importance Assigned to Milestone
MAAS
Undecided
Unassigned
linux (Ubuntu)
Undecided
Unassigned
qemu (Ubuntu)
Undecided
Unassigned

Bug Description

Using latest MAAS master, I'm unable to compose a VM over the UI successfully when composed with 2048 MB of RAM. By that I mean that the VM is created, but it fails with a kernel panic.

summary: - [2.5, UI] Composing a VM over the UI is broken
+ [2.5, UI] Composing a VM over the UI is broken - VM has kernel panic
Changed in maas:
milestone: none → 2.5.0rc1
importance: Undecided → Critical
status: New → Triaged
tags: added: ui
Anthony Dillon (ya-bo-ng) wrote :

This doesn't appear to be a UI issue. Steve, has taken a look into the templates but it all seems to be displaying correctly.

Mike Pontillo (mpontillo) wrote :

I can't reproduce this crash. Does the kernel panic happen at commissioning time or later?

Can you attach the output of `virsh dumpxml <vm-name>` for the both the working VM composed over the API, and the non-working VM composed with the UI?

Changed in maas:
status: Triaged → Incomplete

Adding the linux package; we should narrow down the issue to see if it's in kernel space or user space.

summary: - [2.5, UI] Composing a VM over the UI is broken - VM has kernel panic
+ [2.5] Composing a VM with 2048 MB RAM causes kernel panic
Changed in maas:
status: Incomplete → Invalid
Changed in linux (Ubuntu):
status: New → Incomplete

To be clear, the kernel panic is seen when a VM is composed in MAAS with exactly 2048 MB of RAM. Composing with 2047 or 2049 MB RAM results in a working VM.

Mike Pontillo (mpontillo) wrote :

We can't run apport-collect since the machine doesn't boot, but this was seen with a non-tainted 4.15.0.36-generic kernel in my environment.

Changed in linux (Ubuntu):
status: Incomplete → New
summary: - [2.5] Composing a VM with 2048 MB RAM causes kernel panic
+ Composing a VM in MAAS with exactly 2048 MB RAM causes kernel panic
tags: removed: ui
summary: - Composing a VM in MAAS with exactly 2048 MB RAM causes kernel panic
+ Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to kernel
+ panic
Ryan Harper (raharper) wrote :

Can you attach the guest xml and host kernel/qemu/libvirt packages?

Changed in linux (Ubuntu):
status: New → Incomplete
Mike Pontillo (mpontillo) wrote :

Here's an example of working XML (with 2047 MB RAM) that MAAS generated:

https://paste.ubuntu.com/p/mkF6Kp4hx8/

And here's an example of non-working XML (with 2048 MB RAM) that MAAS generated:

https://paste.ubuntu.com/p/27HHDrzwm8/

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Mike Pontillo (mpontillo) wrote :

Here's the version information.

libvirt-bin:
  Installed: 4.0.0-1ubuntu8.5
  Candidate: 4.0.0-1ubuntu8.5
  Version table:
 *** 4.0.0-1ubuntu8.5 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     4.0.0-1ubuntu8.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     4.0.0-1ubuntu8 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

qemu-kvm:
  Installed: 1:2.11+dfsg-1ubuntu7.6
  Candidate: 1:2.11+dfsg-1ubuntu7.6
  Version table:
 *** 1:2.11+dfsg-1ubuntu7.6 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2.11+dfsg-1ubuntu7.3 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     1:2.11+dfsg-1ubuntu7 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

qemu-system-x86:
  Installed: 1:2.11+dfsg-1ubuntu7.6
  Candidate: 1:2.11+dfsg-1ubuntu7.6
  Version table:
 *** 1:2.11+dfsg-1ubuntu7.6 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2.11+dfsg-1ubuntu7.3 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     1:2.11+dfsg-1ubuntu7 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

linux-image-4.15.0-34-generic:
  Installed: 4.15.0-34.37
  Candidate: 4.15.0-34.37
  Version table:
 *** 4.15.0-34.37 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     4.15.0-34.37~16.04.1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages

Ryan Harper (raharper) wrote :

And /var/log/libvirt/qemu/<guestname>.log ?

Mike Pontillo (mpontillo) wrote :

Here's the log from the failing VM. Doesn't look too unusual to me...

https://paste.ubuntu.com/p/JpmSxmwjfM/

description: updated
Ryan Harper (raharper) wrote :

The backing image:

/var/lib/libvirt/maas-images/e5d185a9-8ccb-4ca6-959a-bd8eff0ee184

What boot image is that? Can I get a copy of that from maas-images? or how is it created?

On the node with the vm that fails, can you:

virsh start <vm-name> --console

Assuming it's a normal ubuntu image which has normal console= settings, it should dump the boot console to the terminal so we can capture the full boot to panic.

Ryan Harper (raharper) wrote :

I'm unable to recreate with a daily bionic cloud-image on a bionic host with the same versions.

% sudo apt install uvtool libvirt
% uvt-simplestreams-libvirt -vv sync --source http://cloud-images.ubuntu.com/daily 'supported=True' arch=amd64 release=bionic
% uvt-kvm create --memory 2048 --cpu 1 --disk 10 rharper-b1 label=daily release=bionic
% virsh dumpxml rharper-b1 | grep Mem
  <currentMemory unit='KiB'>2097152</currentMemory>

Mike Pontillo (mpontillo) wrote :

It's an empty image - MAAS PXE boots the VM.

Could you give it a try with MAAS? I can help you with the setup if needed - just ping me on IRC.

Mike Pontillo (mpontillo) wrote :

Here's a full console log from the failure.

https://paste.ubuntu.com/p/zmS7CP7NKr/

As Ryan I can not reproduce locally - hrm.

The crash in your log is the root-fs mount.

[ 22.524541] VFS: Cannot open root device "squash:http://172.16.99.2:5248/images/ubuntu/amd64/generic/bion" or unknown-block(0,0): error -6
[ 22.575588] Please append a correct "root=" boot option; here are the available partitions:
[ 22.583909] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Also we have to stick to exactly your values (one of the repros had a slightly different value)
  <memory unit='KiB'>2096128</memory>
  <currentMemory unit='KiB'>2096128</currentMemory>

I tried with the exact numbers above but "normal" cloud image boot is still ok.

I wonder if the kernel has an off by one error e.g. aligning the squash at the lowest 2G but with just this amount of memory choosing a place it would not fit.

We'd need to set up a local http and serve squashfs, to boot into that.
With some luck we can reproduce there and then eliminate libvirt and maas out of the equation.

Repro:
- I started off as Ryan did with a Cloud Image test via UVTool.
- Next I extracted the kernel+initrd from the guest to provide those from the host (as you do via PXE)
- installed nginx
- made initrd available on /var/www/html/boot-initrd (initrd.img-4.15.0-36-generic)
- made kernel available on /var/www/html/boot-kernel (vmlinuz-4.15.0-36-generic)
- The address of the Host on the libvirt net is 192.168.122.1, verify the guest can http from there
- get matching squash (see below for details)
- get an empty qemu disk via qemu-img like the type raw that maas uses
  sudo qemu-img create -f raw /var/lib/libvirt/images/empty-root.img 10G
- With that, modify the guest to use these kernel/initrd/sqashfs/empty-root

XML of the guest: http://paste.ubuntu.com/p/PhBn6n8VYH/

I have a dependency issue for my repro, that is IP being configured in /scripts/init-bottom after trying to mount squashfs in what seems /scripts/local-premount.
http://paste.ubuntu.com/p/RscHmQqFyY/
I can even fetch the squashfs from the initramfs, wouldn't you be affected by the same ordering issue? I need to find how you usually get around that to continue the repro that hopefully eventually helps to focus on the root cause.

I experimented a bit more and asked around on IRC.
But so far I can't get past the ordering issue that IP is initialized to late and due to that squash is failing.

-- Appendix --

Get Squash:
To continue I'd need the current squashfs instead of the disk image.
My uvtool spawned this for me:
$ uvt-simplestreams-libvirt --verbose query
release=bionic arch=amd64 label=daily (20181012)
So lets get the mathcing suqash URL and fetch that.
$ sstream-query --output-format="%(item_url)s" --no-verify http://cloud-images.ubuntu.com/daily arch=amd64 release=bionic label=daily ftype=squashfs version_name=20181012
http://cloud-images.ubuntu.com/daily/server/bionic/20181012/bionic-server-cloudimg-amd64.squashfs
$ sudo wget -O /var/www/html/squashfs http://cloud-images.ubuntu.com/daily/server/bionic/20181012/bionic-server-cloudimg-amd64.squashfs

Note: That setup is available on server horsea

After DHCP is up it works just fine.

(initramfs) wget http://192.168.122.1:80/squashfs
Connecting to 192.168.122.1:80 (192.168.122.1:80)
squashfs 100% |*******************************| 174M 0:00:00 ETA
(initramfs) mount -t squashfs squashfs /root
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1007984k,nr_inodes=251996,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=204128k,mode=755)
tmpfs-root on /media/root-rw type tmpfs (rw,relatime)
copymods on /root/lib/modules type tmpfs (rw,relatime)
/dev/loop0 on /root type squashfs (ro,relatime)

So if anyone has a good hint how to get out of the ip/squash-root ordering issue let me know.
That would - as mentioned - help to most likely exclude maas and libvirt from the suspects to be able to debug further.

Working on this I found by accident that I actually can reproduce:
  VFS: Cannot open root device "squash:http://192.168.122.1:80/squashfs" or unknown-block(0,0): error -6

But the way I got there lets assume some more potential reasons.
I got there by breaking my initramfs :-)

After realizing this I removed the initramfs from the guest definition and got to just the same error.

The reason is that without initramfs the kernel is responsible to handle root= and it has no idea of squashfs.

With that knowledge I re-checked your log at: https://paste.ubuntu.com/p/mkF6Kp4hx8/
It also has no entry like:
  Loading, please wait...
  starting version 237
Which you'd see if systemd in the initrd would take over.
So your bad case also fails to load the initramfs!

That said, why could that be special for just this memory size?

Theories related to the guest size impacting this:
- initrd is placed explicit in PXE config now conflicting with kernel allocations
- initrd is misplaced/misread by PXE code in qemu

@Mike - I'd want to know your exact PXE config
@Mike - It would be great to attach your kernel+initrd+squashfs+rootdisk files to the bug.

Hopefully we can reproduce by providing kernel+initrd via PXE and varying the guest size.

Changed in qemu (Ubuntu):
status: New → Incomplete
Ryan Harper (raharper) wrote :

[ 0.943808] Unpacking initramfs...
[ 20.690329] Initramfs unpacking failed: junk in compressed archive
[ 20.703673] Freeing initrd memory: 56612K

Looks like the initrd was compromised, possibly a networking hiccup? Can you confirm the checksums on the source and attempt to download the URL ?

I can't see why the size of the VMs ram makes a difference though. But I don't think this is a qemu issue any more.

Since it is reproducible maybe not a networking hiccup.
But maybe a hiccup of the PXE setup (thats why I asked for it).
Or a hiccup of the quite complex shim loading if signed kernels are used.

@Mike - in addition to my questions above could you use a non signed/shim load pattern as well to check if that might be part of the reason?

Changed in maas:
milestone: 2.5.0rc1 → none
Mike Pontillo (mpontillo) wrote :

I agree that this doesn't look like a QEMU issue, and I agree that it doesn't look like a [general] networking hiccup.

@paelzer, the easiest way to get the exact configs you need would be to install MAAS similar to how I've described on our discourse forum[1]. The PXE config will be /similar/ to this[2] (copied/pasted from /usr/lib/maas/maas-test-enlistment on my MAAS server).

[1]: MAAS setup instructions
https://discourse.maas.io/t/setting-up-a-flexible-virtual-maas-test-environment/142

[2]: PXE config
https://paste.ubuntu.com/p/KBsCHCfKBD/

Changed in qemu (Ubuntu):
status: Incomplete → Invalid
Ryan Harper (raharper) wrote :

Does this fail with other releases? like trusty? I was wondering if initrd size plays a factor here:

precise/hwe-t: 25M
trusty/hwe-x: 35M
xenial/ga: 39M
xenial/hwe: 53M
xenial/edge: 53M
bionic/ga: 55M
cosmic/ga: 57M

That might be faster for you to test than for us to replicate setup.

Mike Pontillo (mpontillo) wrote :

Good idea @rharper. It's easy for MAAS to attempt commissioning on Xenial or Bionic, so I gave Xenial a try. It works fine![1]

[1]: Console log -
https://paste.ubuntu.com/p/58hfBX6BhY/

Mike Pontillo (mpontillo) wrote :

I just tried Xenial with the HWE kernel - same result (success). FYI.

https://paste.ubuntu.com/p/WpqHnzbsS7/

Mike Pontillo (mpontillo) wrote :

... it's interesting that [practically?] the same kernel version that fails consistently with Bionic works just fine with Xenial.

I must admit, compared to you just sharing your kernel/initrd/squash asking to install an own dev maas in comment #23 consumes quite some time :-/
Waiting for a sync here, setting up users there, sorting out how to make us provide PXE and such on the "maas" virtual network, ... - it just isn't one click and ready.

Initially things worked other than the unexpected "wait for image sync".
I've got a Pod registered and working to get to libvirt data.

But compose blocks at:
"Pod unable to compose machine: Please add a 'default' or 'maas' network whose bridge is on a MAAS DHCP enabled VLAN. Ensure that libvirt DHCP is not enabled."

Well that is clear, but a link where/how to get MAAS dhcp/pxe to own that vlan would be nice.
Maybe something small and easy for you that would help to be added.

When I go to subnets to create one (to have maas own it) for 172.16.99.0/24 that I created following your link it tells me the subnet already exists "Error: Subnet with this Cidr already exists." - but it isn't in the subnet overview so I can't enable DHCP/PXE on it :-/

My overview suffers a bit from the links you had adding sample-data, I cleaned up a lot of demo-BS and can now see more clearly. I'd want to add a subnet to my fabric that I called "libvirt-maas" being the virbr1 "maas" definition by libvirt.
Afterwards I still get "Error: Subnet with this Cidr already exists."

Well it might conflict with the "172.16.99.1" set up by.
Lets try to set up another subnet, that worked.
Also added a range of IPs in the hope that this would switch DHCP on, but it didn't

So on the subnet's own page "managed allocation" is enabled.
But when clicking on the top menu subnets the column for DHCP in the table is "disabled"

Anyway, lets try to compose something ...
No, still blocked at "Pod unable to compose machine: Please add a 'default' or 'maas' network whose bridge is on a MAAS DHCP enabled VLAN. Ensure that libvirt DHCP is not enabled."

That exceeds what I can find on [2] for DHCP enabling, but still refuses me.
Since reusing maas DHCP/PXE setup is exactly what I wanted to debug for/with you I'm stuck, lost time and we are not a bit further on this bug here :-/

Now trying to squeeze your PXE config into tftp/libvirt manually without MAAS - lets see if I can reproduce via that.

[1]: https://discourse.maas.io/t/setting-up-a-flexible-virtual-maas-test-environment/142
[2]: https://docs.maas.io/devel/en/installconfig-network-subnet-management

I didn't let me calm down - I found in the doc [1] that the switch might be on the VLAN and not the subnet (Hrm why?)

It was on the vlan
http://horsea:5240/MAAS/#/vlan/5021
Not on the subnet
http://horsea:5240/MAAS/#/subnet/11

There I found the "provide dhcp" entry \o/
That unlocked some understanding of what the range allocations would be like on vland/subnets.
I reconfigured the range allocation and hit the "provide DHCP" switch.

Looking back all is reasonable, just some stumbling on my first maas setup from scratch :-/
Must be funny for you seeing me struggle doing so :-)

With that in place I created three pods with 2047/2048/2049 MB memory.
ubuntu@node-horsea:~/maas$ virsh dumpxml horsea-kvm-pod-2047 | grep emor
  <memory unit='KiB'>2096128</memory>
  <currentMemory unit='KiB'>2096128</currentMemory>
ubuntu@node-horsea:~/maas$ virsh dumpxml horsea-kvm-pod-2048 | grep emor
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
ubuntu@node-horsea:~/maas$ virsh dumpxml horsea-kvm-pod-2049 | grep emor
  <memory unit='KiB'>2098176</memory>
  <currentMemory unit='KiB'>2098176</currentMemory>

They all went into the commissioning phase, but who would be surprised got stuck somewhere in there :-/
OTOH the bug says "composing crashes" so we might be at the right place, lets take a look at these three guests.

[1]: https://docs.maas.io/devel/en/installconfig-network-dhcp#enabling-dhcp

I didn't see any guests crashing, instead they all just hang finding nothing to boot.

I reduced this to just qemu (for debugging later on) but it reliably breaks finding nothing from PXE.

Guest Console:
  iPXE (PCI 00:03.0) starting execution...ok
  iPXE initialising devices...ok
  iPXE 1.0.0+git-20180124.fbe8c52d-0ubuntu2.1 -- Open Source Network Boot Firmware
   -- http://ipxe.org
  Features: DNS HTTP HTTPS iSCSI NFS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT

  net0: 52:54:00:3b:72:0a using virtio-net on 0000:00:03.0 (open)
    [Link:up, TX:0 TXE:0 RX:0 RXE:0]
  Configuring (net0 52:54:00:3b:72:0a).................. No configuration methods
  succeeded (http://ipxe.org/040ee119)
  No more network devices

Corresponding command to start qemu:
#!/bin/bash
/usr/bin/qemu-system-x86_64 -name guest=horsea-kvm-pod-2047,debug-threads=on -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -m 2047 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -cpu kvm64 -rtc base=utc -no-shutdown -curses \
-chardev socket,id=charmonitor,path=/tmp/horsea-kvm-pod-2047.monitor.sock,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-chardev stdio,mux=on,id=charserial0,logfile=/tmp/horsea-kvm-pod-2047.log \
-device isa-serial,chardev=charserial0,id=serial0 \
-drive file=/var/lib/uvtool/libvirt/images/d9658e2f-d4c7-4a55-8ecc-b216612fe410,format=raw,if=none,id=drive-virtio-disk0 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,serial=d9658e2f-d4c7-4a55-8ecc-b216612fe410 \
 \
-boot order=n,strict=on \
 \
-net bridge,br=virbr1,helper=/usr/lib/qemu/qemu-bridge-helper -net nic,model=virtio,macaddr=52:54:00:3b:72:0a

This is the mac that is also in the libvirt XML.
The bridge is the one served by MAAS and it seems there is just no PXE responding to it.

Since the two other guests as started by maas+libvirt hang just the same way it most likely is a general MAAS setup issue.
Please help me to get MAAS to reply the way you would usually expect it, to then be able to go on with debugging.

Download full text (3.3 KiB)

Taking the x86 pxelinux.0 from /usr/lib/PXELINUX/pxelinux.0 and otherwise doing the same tftp setup as in [1] and switching from the "maas" to the "default" network where I have set up dhcp to netboot by libvirt as in [1]. I can see netboot happening.

With that instead of pushing things from libvirt via kernel/initrd tags I now provide it via PXE similar to your config.

TL;DR
- tftp serves: lpxelinux.0 + pxe modules + PXEconfig
- nginx serves: kernel/initrd/squashfs
- qemu started directly without libvirt

Still gets the inintial kernel booting fine and then failing to mount the squash:
mount: mounting squash:http://192.168.122.1:80/squashfs on /root failed: No such device

I can from the initramfs mount it:
(initramfs) wget http://192.168.122.1:80/squashfs
Connecting to 192.168.122.1:80 (192.168.122.1:80)
squashfs 100% |*******************************| 174M 0:00:00 ETA
(initramfs) mount -t squashfs squashfs /root/
It is mounted just fine:
/dev/loop0 on /root type squashfs (ro,relatime)

It is possible that I'm back at the same ordering issue that I had of the IP coming up too late to mount the squashfs, but later works fine.

I mounted it with 2047/2048/2049 mb gusts without a problem.

Despite all work on this I'd still need a better reproducer :-/
I might take a look at rebuilding a more verbose initramfs for that after lunch.

--- config details ---

$ find /srv/tftp/
/srv/tftp/
/srv/tftp/squashfs
/srv/tftp/boot-kernel
/srv/tftp/ldlinux.c32
/srv/tftp/lpxelinux.0
/srv/tftp/pxelinux.cfg
/srv/tftp/pxelinux.cfg/01-52-54-00-3b-72-0a
/srv/tftp/boot-initrd

Kernel/initrd/squash as described in comment #18

# modified to get scrollable direct console (but no iPXE VGA output)
#!/bin/bash
/usr/bin/qemu-system-x86_64 -name guest=horsea-kvm-pod-2047,debug-threads=on -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -m 2047 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -cpu kvm64 -rtc base=utc -no-shutdown \
 \
-nographic -serial mon:stdio \
 \
-drive file=/var/lib/uvtool/libvirt/images/d9658e2f-d4c7-4a55-8ecc-b216612fe410,format=raw,if=none,id=drive-virtio-disk0 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,serial=d9658e2f-d4c7-4a55-8ecc-b216612fe410 \
 \
-boot order=n,strict=on \
 \
-net bridge,br=virbr0,helper=/usr/lib/qemu/qemu-bridge-helper -net nic,model=virtio,macaddr=52:54:00:3b:72:0a
# For access to the iPXE / lpxelinux graphical output
#-curses \
#-chardev stdio,mux=on,id=charserial0,logfile=/tmp/test.log \
#-serial chardev:charserial0 \
#-mon chardev=charserial0,mode=readline \
# For access to a scrollable direct console and monitor
# -nographic -serial mon:stdio

$ cat /srv/tftp/pxelinux.cfg/default
DEFAULT execute

LABEL execute
  SAY Booting under MY direction...
  KERNEL http://192.168.122.1:80/boot-kernel
  INITRD http://192.168.122.1:80/boot-initrd
  APPEND console=ttyS0 nomodeset ro root=squash:http://192.168.122.1:80/squashfs ip=::::horsea-kvm-pod-2047:BOOTIF ip6=off overlayroot=tmpfs...

Read more...

There actually is no need to debug further on my non-maas PXE setup.

As identified before - your setup breaks due to
[ 20.690329] Initramfs unpacking failed: junk in compressed archive
Everything else is a follow on issue, with eventually:
[ 22.524541] VFS: Cannot open root device "squash:http://172.16.99.2:5248/images/ubuntu/amd64/generic/bion" or unknown-block(0,0): error -6

But since my setup as outlined in comment #31 PXE-boots the kernel+initramfs just fine I don't have to debug the rest.
I already passed the point where it breaks for you.

Can you use the above hints to get your breaking setup to also one-by-one remove components.
I'd assume that removing libvirt makes no difference to your case.
But you could maybe end up with:
- PXE-served by MAAS = bad
- PXE serverd by tftpd = good
Or anything like it.

So please follow the hints above (or get in touch if you need me to do so) and convert your setup until you can identify which component makes the decision for good/bad case.

Also still I'd be open to get your kernel/initramfs/squash/pxe/.... files to implant it into my setup for a test if any of them is the trigger.

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Changed in maas:
status: Invalid → Incomplete
Changed in qemu (Ubuntu):
status: Invalid → Incomplete
Mike Pontillo (mpontillo) wrote :

After triaging this again on a call with Andres (who originally reported this) this morning, we determined that this issue is no longer reproducible with MAAS. The only way for me to explain it at this point was that there was something wrong with the daily image MAAS was using last week, and a subsequent update fixed it. (It looks like the images were updated yesterday.)

Sorting my /var/lib/maas/boot-resources/cache directory by last-modified, I see the following:

https://paste.ubuntu.com/p/mc76rffz5k/

@paelzer, if you'd like to test with any of those images, I've made copies. (Not sure where to put them, though - I wonder if your MAAS synced them as well?)

Still very weird that it worked in all cases we tested except when we allocated 2048 MB RAM.

I'd like to thank Ryan and Christian for their efforts on this "heisenbug". We should think about how to better handle issues like this, so that it's easier to peel back the layers and get to a point where everyone's environment can be consistent without this much effort. And we'll reopen this if it returns.

Mike Pontillo (mpontillo) wrote :

Looking again at the date stamps, I don't see any squashfs filesystems older than October 15th. The kernels are all from ~September 25th. So I feel like this must have been an interaction between the kernel September 25th and whatever the previous squashfs was.

Hi,
thanks for your feedback.
We are at least much closer to what happened and thereby should be faster if it reoccurs.
I don't think it is an interaction of the Kernel and Squashfs as we found that already initramfs unpack was broken. That is before Squash comes into play.

I'll keep my test setup until I need to re-deploy the host, which usually is about every 1-2 weeks.
If you ever find old kernel/initrd combinations or any new one to trigger it again - please share them via e.g. internal private fileshare - I sent you some details on IRC.

Lets see if it comes up again.

Vladimir Grevtsev (vlgrevtsev) wrote :
Changed in maas:
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
status: Incomplete → New
Changed in qemu (Ubuntu):
status: Incomplete → New
tags: added: cpe-onsite
tags: added: field-medium
Changed in maas:
status: Confirmed → Incomplete
Vladimir Grevtsev (vlgrevtsev) wrote :

Subscribing field-medium as workaround (e.g not to use 2048mb ram VMs) exists, but solution still need to be found.

Also, before yesterday (e.g on libvirt 1:2.11+dfsg-1ubuntu7.8 + maas 2.5rc2) worked fine.

Andres Rodriguez (andreserl) wrote :

Setting this back to incomplete for MAAS because there is not enough information to determine this is a MAAS Bug. That said, given that this work with any VM other than one with 2048 of RAM, the issue may appear to be the kernel or libvirt.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu):
status: New → Confirmed
Changed in qemu (Ubuntu):
status: New → Confirmed

Again it seems to break on initramfs being bad:
[ 0.840153] Initramfs unpacking failed: no cpio magic

@Mike/Andres - can you reproduce it on your own test env this time?

@Vladimir - could you internally share exactly your kernel and initrd that is tried to be booted in this case? Last time we wondered if the issue could be "in there" so it would be great to have exactly those that fail for you shared to try reproducing on them.

One other oddity in the xml is the cgroup construction in the "bad" case.

<resource>
    <partition>/machine</partition>
  </resource>

>
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1797581/+subscriptions
>

@Ryan - the resource and other bits being different is because in the bad case the domain was up and in the good case down. That in itself is not a problem.

Reproduced today on MAAS 2.5 latest bionic

It is not reproduced if switch from ga-18.04 kernel to ga--18.04-low-latency kernel.

Thiago Martins (martinx) wrote :

I'm also facing this problem.

My workaround is to compose VMs using Virt-Manager, Firmware = UEFI for the VM and then, refreshing the MaaS Pod.

There is a need to install ovmf

sudo apt install ovmf

On MaaS Pod.

Then, no more Kernel Panic!

Details:

https://discourse.maas.io/t/maas-2-5-bionic-pod-composing-guests-kernel-panic-no-init/302

Ryan Harper (raharper) wrote :

@Thiago,
Can you attach your guest XML that's working successfully with 2048MB?

Thiago Martins (martinx) wrote :

@Ryan,

 The working XML file is attached here, with 2048 MB of RAM.

 NOTE: This XML was created using Virt-Manager, then, MaaS took it over after being "refreshed".

Cheers!
Thiago

Ryan Harper (raharper) wrote :

On Mon, Jan 14, 2019 at 1:55 PM Thiago Martins <email address hidden>
wrote:

> @Ryan,
>
> The working XML file is attached here, with 2048 MB of RAM.
>
> NOTE: This XML was created using Virt-Manager, then, MaaS took it over
> after being "refreshed".
>

Thanks!

If you drop the <os> section (which is what tells libvirt to boot via UEFI)
does your VM still work?

<os>
<type arch="x86_64" machine="pc-i440fx-bionic">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/vunft-1_VARS.fd</nvram>
</os>

Note that MAAS boots UEFI images with grub2[1], not pxeboot/ipxe/seabios;
so I think we can narrow down the
error to the non-uefi case which may help find the issue.

1.
http://images.maas.io/ephemeral-v3/daily/bootloaders/uefi/amd64/20181123.0/

>
> Cheers!
> Thiago
>
> ** Attachment added: "vunft-1.xml"
>
> https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5229030/+files/vunft-1.xml
>
> --
> You received this bug notification because you are subscribed to qemu in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1797581
>
> Title:
> Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
> kernel panic
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1797581/+subscriptions
>

Thiago Martins (martinx) wrote :

Ryan,

 If I remove the UEFI line (back to BIOS), the machine enters in Kernel Panic during the boot / commissioning.

 That was actually, the very test that I executed in first place (add UEFI to see)! Then, I came with this UEFI workaround, which is even better for me.

Cheers!
Thiago

Changed in maas:
importance: Critical → Undecided
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers