instances fails to register GPT partition entries on virtio devices

[Ben Howard]

When booting via KVM and using the VirtIO interface, GPT partition entries are not shown properly.

ubuntu@gpt-test-4:~$ sudo parted /dev/vda -s -- print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
127 1049kB 2097kB 1049kB BIOS boot partition bios_grub
128 2097kB 89.1MB 87.0MB fat32 EFI System boot
    1 89.1MB 10.7GB 10.6GB ext4 Linux filesystem

Booting with "-hda <disk>.img"
[ 2.612975] sr 1:0:0:0: Attached scsi generic sg1 type 5
[ 2.617252] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.644685] sda: sda1 sda127 sda128
[ 2.664314] sd 0:0:0:0: [sda] Attached SCSI disk

Booting with "-drive if=virtio,file=<disk>.img" or "-drive if=scsi,file=<disk>.img":
[ 2.411933] brd: module loaded
[ 2.422616] loop: module loaded
[ 2.442178] vda: vda1
[ 2.455559] scsi0 : ata_piix
[ 2.456868] scsi1 : ata_piix

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: udev 204-5ubuntu6
ProcVersionSignature: Ubuntu 3.12.0-5.13-generic 3.12.2
Uname: Linux 3.12.0-5-generic x86_64
ApportVersion: 2.12.7-0ubuntu1
Architecture: amd64
Date: Fri Dec 6 18:49:57 2013
Ec2AMI: ami-00000632
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: aki-00000548
Ec2Ramdisk: ari-00000548
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.12.0-5-generic root=LABEL=cloudimg-rootfs console=ttyS0
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2007:svnOpenStackFoundation:pnOpenStackNova:pvr2013.1.3:cvnBochs:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2013.1.3
dmi.sys.vendor: OpenStack Foundation

[Ben Howard (darkmuggle-deactivatedaccount)]

For everyone's convience, you can find a Cloud Image configured for BIOS/GPT and UEFI boot at:
http://people.canonical.com/~ben/lp_1258631.img

[Ben Howard (darkmuggle-deactivatedaccount)]

From VMware Workstation with LSI disk:
[ 2.376392] sd 32:0:0:0: [sda] 4194304 512-byte logical blocks: (2.14 GB/2.00 GiB)
[ 2.378247] sd 32:0:0:0: [sda] Write Protect is off
[ 2.379394] sd 32:0:0:0: [sda] Cache data unavailable
[ 2.380505] sd 32:0:0:0: [sda] Assuming drive cache: write through
[ 2.382008] sd 32:0:0:0: Attached scsi generic sg0 type 0
[ 2.383680] sd 32:0:0:0: [sda] Cache data unavailable
[ 2.384999] sd 32:0:0:0: [sda] Assuming drive cache: write through
[ 2.388112] sda: sda1 sda127 sda128
[ 2.389666] sd 32:0:0:0: [sda] Cache data unavailable
[ 2.390782] sd 32:0:0:0: [sda] Assuming drive cache: write through
[ 2.392173] sd 32:0:0:0: [sda] Attached SCSI disk

Which further illustrates my strong suspicion this is a VirtIO issue.

[Joseph Salisbury (jsalisbury)]

Is this issue new in Trusty? Do you happen to know if this also happens with the latest mainline kernel:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc3-trusty/

[Brad Figg (brad-figg)]

This change was made by a bot.

[Ben Howard (darkmuggle-deactivatedaccount)]

Joseph,

This problem does happen with the latest upstream kernel as well.

ubuntu@gpt-test-4:~$ uname -a
Linux gpt-test-4 3.13.0-031300rc3-generic #201312061335 SMP Fri Dec 6 18:37:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@gpt-test-4:~$ dmesg | grep vda
[ 1.039698] vda: vda1
[ 1.879747] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
[ 2.544984] EXT4-fs (vda1): re-mounted. Opts: (null)

And testing the 3.8 line in Saucy:
ubuntu@gpt-test-4:~$ uname -a
Linux gpt-test-4 3.8.13-03081311-generic #201310151335 SMP Tue Oct 15 17:36:06 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@gpt-test-4:~$ dmesg | grep vda
[ 0.789770] vda: vda1
[ 1.618881] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
[ 2.226267] EXT4-fs (vda1): re-mounted. Opts: (null)

So this appears to be a long-running issue.

[Stefan Bader (smb)]

Funny enough the boot dmesg contains some indication about the problem, while the serial output does now (or so it seems):

[ 0.627017] GPT:Primary header thinks Alt. header is not at the end of the disk.
[ 0.628500] GPT:4194303 != 20971519
[ 0.629167] GPT:Alternate GPT header not at the end of the disk.
[ 0.630291] GPT:4194303 != 20971519
[ 0.630951] GPT: Use GNU Parted to correct GPT errors.
[ 0.632002] vda: vda1

I have to get the image to check locally. Would be interesting to look at the reported values and whether breaking into pre-mount allows to reread the partition table (check whether its a race).

[Ben Howard (darkmuggle-deactivatedaccount)]

The messages:
[ 0.627017] GPT:Primary header thinks Alt. header is not at the end of the disk.
[ 0.628500] GPT:4194303 != 20971519
[ 0.629167] GPT:Alternate GPT header not at the end of the disk.
[ 0.630291] GPT:4194303 != 20971519
[ 0.630951] GPT: Use GNU Parted to correct GPT errors.
[ 0.632002] vda: vda1

Are expected. This image was built as a 2GB volume and then expanded to 10GB on OpenStack. When you reboot, the message goes away and the grow-rootfs fixes the problem. Also, you'll notice that the physical layout has partition 127 and 128 at the begining of the drive, before partition one. I would argue that this message is a red herring.

[Stefan Bader (smb)]

Oh dear, the answer is so simple after one finally realizes what is going on... The problem here is that the virtio_blk driver has not been changed to support more than 15 partitions per device. The partition scanning is looking at the GPT partition table but only at the first 15 elements. Since the other two partitions are 127 and 128, they never get added if the device is a virtio disk.

So this isn't exactly a bug but a limitation in virtio right now. I am not sure whether the images could use number 14 and 15 for those two partitions. Then they should be visible for emulated and virtual devices... Otherwise this would be some development work which one had to discuss upstream.

[Stefan Bader (smb)]

Won't fix sounds harsh but as this is not really a bug but a limitation of the driver, this would be an enhancement and no fix of a bug. Which means this would have to be done (and the need for it accepted) upstream.