Ubuntu can try to boot from unbootable partition if UUID conflicts

[Garth Kidd]

I don't think this is as specific to AWS/EC2 as it might initially appear, but here goes:

To reproduce:

* Create AWS EC2 instance using official Ubuntu 14.04 AMI
* Use AWS EBS snapshot to clone root EBS volume to new EBS volume
* Mount it from EC2 console
* e2label to ensure label unique
* fdisk to ensure partition not bootable
* vi /etc/fstab to ensure mounted in appropriate place
* forget to use tunefs -U to change the partition UUID
* reboot a few times, perhaps upgrading kernel between reboots

Expected behaviour:

* Warnings during or after boot process that two partitions have the same UUID
* Consistently booting from the partition labelled cloudimg-rootfs despite the UUID conflict
* Consistently booting from the partition with the boot flag set despite the UUID conflict

Actual behaviour:

* No apparent warnings during or after boot process that two partitions have the same UUID
* First few boots turned out fine
* Around a month later, after a kernel upgrade from 3.13.0-92 or 105 to 106, started consistently booting from non-bootable partition labelled var-lib

Console log output on problem:

    Begin: Loading essential drivers ... done.
    Begin: Running /scripts/init-premount ... done.
    Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
    Begin: Running /scripts/local-premount ... done.
    [ 2.518466] EXT4-fs (xvdf1): mounted filesystem with ordered data mode. Opts: (null)
    Begin: Running /scripts/local-bottom ... done.
    done.
    Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory
    done.
    mount: mounting /sys on /root/sys failed: No such file or directory
    mount: mounting /proc on /root/proc failed: No such file or directory
    Target filesystem doesn't have requested /sbin/init.
    No init found. Try passing init= bootarg.

Ops remediation of affected hosts:

* Remount secondary EBS volume to spare host
* tunefs /dev/xvdf1 -U `uuidgen`
* Remount it back to the host with the outage

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-106-generic 3.13.0-106.153
ProcVersionSignature: Ubuntu 3.13.0-106.153-generic 3.13.11-ckt39
Uname: Linux 3.13.0-106-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Dec 21 01:06 seq
 crw-rw---- 1 root audio 116, 33 Dec 21 01:06 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.23
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Wed Dec 21 01:57:13 2016
Ec2AMI: ami-ba3e14d9
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: ap-southeast-2c
Ec2InstanceType: c4.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Xen HVM domU
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-106-generic root=UUID=63e1d5b9-1fa6-4ff3-b8d6-7a63b4b18e59 ro console=tty1 console=ttyS0
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-106-generic N/A
 linux-backports-modules-3.13.0-106-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 11/11/2016
dmi.bios.vendor: Xen
dmi.bios.version: 4.2.amazon
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd11/11/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
dmi.product.name: HVM domU
dmi.product.version: 4.2.amazon
dmi.sys.vendor: Xen

[Garth Kidd (garth-deadlybloodyserious)]

What I think is going on: looking up the boot partition by label or bootable flag; finding its UUID, then trying to mount it by the UUID, at which point it's possible to get the wrong partition.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.10 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc2

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]