wrong boot order lxd vm with more than one disk

Bug #1943444 reported by Sam Richards
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
High
Alexsander de Souza

Bug Description

When composing, deploying an LXD virtual machine like:

maas vm-host compose 11 storage=‘10(default),11(default),12(default)

I expect the first disk (10gb) to be the boot disk and the device order to be:

sda 10 boot
sdb 11
sdc 12

But I see

sda 11 boot
sdb 12
sdc 10

I think there's a few things happening here.

1. The boot.priority: 0 is set incorrectly in the LXD config for the root disk. I believe this may have initially been a workaround due to a bootIndex bug in LXD for network devices which looks like it has been fixed: https://github.com/lxc/lxd/commit/debbf8adf8cbf20f69be36a3fd8974b234539a1f
to ensure the "hotplugged" nic does indeed always boot first. Given that the bootIndex for a nic is now respected, we can _correct_ the boot.priority in our generated LXD config. I've attached a simple patch to do this.
The issue I see with this patch is that for the set of LXD versions where nics didn't boot first unless the disk was set to boot priority 0 (I assume that was the reason this was set in the first place), between LXD 4.3 (multiple disk support) and 4.15 (nic bootindex fix) that we'd regress. It may be worth making the minimal LXD version requirement 4.15?

Reference:
LXD sets the "root" device's boot priority to 1, but we override it to 0 which matches all the other disk devices (default is 0):
 (https://github.com/lxc/lxd/blob/master/lxd/instance/drivers/driver_qemu.go#L2324).

2. When I make the above patch I get the correct disk order in the resulting VM, I also get the nic booting first. Great! The next problem is that maas appears to have incorrectly partitioned the wrong disk device during deployment.
What I have observed is that the wrong disk ordering is added to maas (lxd_root last) upon compose, then later after deployment the order returns to normal. My assumption here is that the initial discovered VM is using the LXD config order read back from
the LXD API to determine the blockdevice order.
I'm suspicious right now that the order in which this code https://git.launchpad.net/maas/tree/src/provisioningserver/drivers/pod/lxd.py#n711 , reads the disk devices back from the maas API, is used as the device order and that the index isn't
respected for the inital partition layout.

I haven't had a chance to work through it as I don't have a full LXD/maas dev stack up but I'd initially try to sort the DiscoveredMachineBlockDevice list by index before returning it.

3. Lastly and this is really an LXD anomaly but a potentially useful observation is that the scsi bus id of the disks currently reflects the qemu bootIndex (https://github.com/lxc/lxd/blob/master/lxd/instance/drivers/driver_qemu_templates.go#L450)
This leaves a "hole" where the nic correctly has the bootIndex=0 (though that config isn't written out to the qemu.conf file). It's slightly ugly, but we shouldn't really care what the scsi-id is as long as the bootIndex is correct.

Related branches

Revision history for this message
Sam Richards (smallsam) wrote :
Revision history for this message
Thomas Parrott (tomparrott) wrote :

RE 1. I don't think the reason for MAAS setting boot.priority=0 is due to https://github.com/lxc/lxd/commit/debbf8adf8cbf20f69be36a3fd8974b234539a1f as this issue was only introduced 4 days earlier via
https://github.com/lxc/lxd/commit/30996190af7de85d87b0d0c93d6549c6275bf5fb#diff-87040b4a9b898c058e0f889f869c7a0448135f0a7f02db851824f48290c3a2f5 and so they didn't make it into separate releases.

Before we started adding NICs via QMP the boot.priority setting should still have been respected as we were adding NIC config to the statically generated config file.

I'll reply about point 3. on the original forum post (https://discuss.linuxcontainers.org/t/wrong-boot-order-for-maas-generated-lxd-config/12117/) as as you say its not directly relevant to this bug.

Revision history for this message
Alberto Donato (ack) wrote :

I don't think we should change the way we set boot.priority. We want MAAS-created VMs to always boot from the first NIC.

WRT device order, by default MAAS picks the first disk as boot disk (unless a different one is later configured by the user).

For VMs composed by MAAS, we should probably explicitly set the boot disk when refreshing VM information to the LXD root disk, unless the user has already set a boot disk.

Revision history for this message
Sam Richards (smallsam) wrote :

I agree with the NIC always booting first, my patch only intended to correct the device order in the VM because this is influenced by the boot.priority.

Perhaps I can simplify my bug report to be:

When I compose and deploy (with no further storage config)
maas vm-host compose 11 storage=‘10(default),11(default),12(default)

The boot + root disk should be the first device specified in this list with 10Gb. The issue is that the second device in this list ends up as the root/boot device with 11Gb.

Alberto Donato (ack)
Changed in maas:
milestone: none → next
importance: Undecided → High
status: New → Triaged
assignee: nobody → Alberto Donato (ack)
Alberto Donato (ack)
Changed in maas:
assignee: Alberto Donato (ack) → nobody
Changed in maas:
status: Triaged → Fix Committed
Bill Wear (billwear)
Changed in maas:
milestone: next → 3.1.0-beta2
Alberto Donato (ack)
Changed in maas:
assignee: nobody → Alexsander de Souza (alexsander-souza)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers