Simultaneous Trusty deployments to VMs with SATA virtual disks on same physical drive fail sometimes - drops to Grub rescue shell

Bug #1467995 reported by Larry Michel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Undecided
Unassigned
qemu-kvm
New
Undecided
Unassigned

Bug Description

I have 7 VMs configured as following:

Processors: 4 vcpus
Memory: 4 GB
SATA Disk 1: 50 GB
SATA Disk 2: 30 GB
NIC0 :
Source device: "Host device br0: macvtap"
Device model: virtio
MAC address: xx:xx:xx:xx:xx:xx
Source mode: Bridge
NIC2:
Source device:
                      Bridge name: br1
Device model: virtio
MAC address: xx:xx:xx:xx:xx:xx

Everything else was default.

2 of the VMs deployed and the other 5 succeeded installation but upon reboot, I see following errors:

On some VMs:

Boot local disk ...
Booting...
error: file '/boot/grub/i386 pc/normal.mod' not found
Entering rescue mode...
grub rescue>

On another VM:
Booting local disk...
Booting...
error: invalid arch-independent ELF magic
Entering rescue mode...
grub rescue>

I then redeployed the 5 remaining VMs and 3 deployed succesfully and the the last 2 dropped to grub. Finally, I re-deployed these last 2 and they deployed successfully.

One thing to note is that I am creating those VMs by creating the first one and cloning it 6 times to get the remaining VMs. However, I don't think this would be a problem since all the NICs get new MAC addresses.

I am attaching the logs and screen captures.

The PXE mac addresses are:
52:54:00:06:d6:94 (PXE)
52:54:00:9c:42:1e (PXE)
52:54:00:d2:8c:f1 (PXE)
52:54:00:ed:17:1c (PXE)
52:54:00:7a:ba:09 (PXE)
52:54:00:d9:b8:e7 (PXE)
52:54:00:96:58:d7 (PXE)

Tags: oil
Revision history for this message
Larry Michel (lmic) wrote :
summary: - 1.8rc3: Deploying trusty to KVM vms fail sometimes during local boot -
- drops to grub rescue mode
+ 1.8rc3: Deploying trusty to PowerKVM vms fail sometimes during local
+ boot - drops to grub rescue mode
Revision history for this message
Larry Michel (lmic) wrote : Re: 1.8rc3: Deploying trusty to PowerKVM vms fail sometimes during local boot - drops to grub rescue mode

Switched from SATA to IDE disks and I couldn't recreate. So, this looks to be IO related during the Grub steps.

Revision history for this message
Larry Michel (lmic) wrote :

Trying to power off/on system gave same errors during reboot, further evidence that boot sectors for the faulty VMs likely got corrupted during write.

Larry Michel (lmic)
summary: - 1.8rc3: Deploying trusty to PowerKVM vms fail sometimes during local
- boot - drops to grub rescue mode
+ Simultaneous Trusty deployments to VMs with SATA virtual disks on same
+ physical drive fail sometimes - drops to Grub rescue shell
Revision history for this message
Ricardo Bánffy (rbanffy) wrote :

This doesn't seem to be related to MAAS. It's more likely something with KVM and the way it manages the virtual SATA interfaces.

Changed in maas:
status: New → Triaged
Changed in maas:
status: Triaged → Invalid
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Please show the full kvm command we can use to reproduce this.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.