[udev] Ubuntu 15.10 Alpha-1 candidates do not boot in EC2 with Xen

[Ben Howard]

Ubuntu 15.10 Alpha-1 Candidates are not booting in EC2. Instances are dropping to
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/disk/by-uuid/e420d299-69ee-46eb-a1a4-893b54ab89a7 does not exist. Dropping to a shell!

This is happening on all instances and whether disks are mounted by label or UUID.

[Ben Howard (utlemming)]

Xen PV instance booted via EC2's PVGRUB:

$ ec2-get-console-output i-91507038
i-91507038
2015-06-23T19:38:54+0000
Xen Minimal OS!
  start_info: 0x112e000(VA)
    nr_pages: 0xf0000
  shared_inf: 0x7dd69000(MA)
     pt_base: 0x1131000(VA)
nr_pt_frames: 0xd
    mfn_list: 0x9ae000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack: 0x96d840-0x98d840
MM: Init
      _text: 0x0(VA)
     _etext: 0x7dc7d(VA)
   _erodata: 0x9a000(VA)
     _edata: 0x9fce0(VA)
stack start: 0x96d840(VA)
       _end: 0x9ade40(VA)
  start_pfn: 1141
    max_pfn: f0000
Mapping memory range 0x1400000 - 0xf0000000
setting 0x0-0x9a000 readonly
skipped 0x1000
MM: Initialise page allocator for 18ba000(18ba000)-f0000000(f0000000)
MM: done
Demand map pfns at f0001000-20f0001000.
Heap resides at 20f0002000-40f0002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0xf0001000.
Initialising scheduler
Thread "Idle": pointer: 0x20f0002050, stack: 0x2070000
Thread "xenstore": pointer: 0x20f0002800, stack: 0x2080000
xenbus initialised on irq 1 mfn 0x665ee7
Thread "shutdown": pointer: 0x20f0002fb0, stack: 0x2090000
Dummy main: start_info=0x98d940
Thread "main": pointer: 0x20f0003760, stack: 0x20a0000
"main" "root=/dev/sda1" "ro" "4"
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********

backend at /local/domain/0/backend/vbd/377/2049
16777216 sectors of 512 bytes
**************************
vbd 2064 is hd1
******************* BLKFRONT for device/vbd/2064 **********

backend at /local/domain/0/backend/vbd/377/2064
8377344 sectors of 512 bytes
**************************
[H[J Booting 'Ubuntu Wily Werewolf (development branch), kernel 3.19.0-22-generic'

root (hd0)
 Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-3.19.0-22-generic root=LABEL=cloudimg-rootfs ro console=h
vc0
initrd /boot/initrd.img-3.19.0-22-generic

============= Init TPM Front ================
Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT
Tpmfront:Info Shutting down tpmfront
close blk: backend=/local/domain/0/backend/vbd/377/2049 node=device/vbd/2049
close blk: backend=/local/domain/0/backend/vbd/377/2064 node=device/vbd/2064
[ 0.000000] PAT configuration [0-7]: WB WT UC- UC WC WP UC UC
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.19.0-22-generic (buildd@allspice) (gcc version 4.9.2 (Ubuntu 4.9.2-10ubuntu13) ) #22-Ubuntu SMP Tue Jun 16 17:15:15 UTC 2015 (Ubuntu 3.19.0-22.22-generic 3.19.8-ckt1)
[ 0.000000] Command line: root=LABEL=cloudimg-rootfs ro console=hvc0
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] ACPI in unprivileged domain disabled
[ 0.000000] Released 0 page(s)
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000...

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1468091/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Ben Howard (utlemming)]

Confirmed that this is not happening on OpenStack.

I suspect that this is cause is that /dev/disk/by-*/ targets are missing for Xen Block devices.

[Robert C Jennings (rcj)]

Last good test for EC2 occurred on June 9th with serial 20150608

https://jerff.canonical.com/job/NG_Test-EC2/376/

Ben is trying out:
wily server daily 20150608 ebs-ssd amd64 us-east-1 ami-6f9e7404 hvm

[Robert C Jennings (rcj)]

Here are the package changes between the last working and first failing builds:

20150608 good
20150610 bad

$ diff manifest-amd64-daily-20150608.txt manifest-amd64-daily-20150610.txt
101c101
< iso-codes 3.57-1
---
> iso-codes 3.58-1
248c248
< libpam-systemd:amd64 219-10ubuntu1
---
> libpam-systemd:amd64 220-5ubuntu2
296c296
< libsystemd0:amd64 219-10ubuntu1
---
> libsystemd0:amd64 220-5ubuntu2
306c306
< libudev1:amd64 219-10ubuntu1
---
> libudev1:amd64 220-5ubuntu2
458,459c458,459
< systemd 219-10ubuntu1
< systemd-sysv 219-10ubuntu1
---
> systemd 220-5ubuntu2
> systemd-sysv 220-5ubuntu2
476c476
< udev 219-10ubuntu1
---
> udev 220-5ubuntu2

[Robert C Jennings (rcj)]

So the it's just udev and systemd changes here. utlemming took the working 20150608 ami, booted, upgraded, and rebooted and the instance didn't survive the reboot.

This affects HVM only, PV is unaffected.

[Ben Howard (utlemming)]

Actually, PV and HVM instances are both affected. I narrowed it down to udev; installing all updates except for udev results in a bootable instance.

[Martin Pitt (pitti)]

Fixed upstream in https://github.com/systemd/systemd/commit/bb5c512de22, part of 221 release. I'll upload a backport if that's critical for alpha-1.

[Ben Howard (utlemming)]

I've confirmed the fix in -proposed in EC2. New builds will happen automagically once the promotion from -proposed happens.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 220-7ubuntu2

---------------
systemd (220-7ubuntu2) wily; urgency=medium

  * persistent block rules: Whitelist xvd* devices for Xen. (LP: #1468091)

 -- Martin Pitt <email address hidden> Wed, 24 Jun 2015 07:02:50 +0200