image won't boot after upgrading to yakkety's 4.8 kernel because efi

[Tycho Andersen]

If I use a relatively recent yakkety daily with uvt-kvm,

com.ubuntu.cloud.daily:server:16.10:amd64 20160918

and upgrade the kernel to the latest kernel (which happens to be 4.8), my VM won't boot any more. kern.log has:

http://paste.ubuntu.com/23211671/

and systemd complains that,

[FAILED] Failed to mount /boot/efi
See 'systemctl status boot-efi.mount' for details.

and dumps me to a recovery console. Of course, I don't have EFI since I'm booting this thing on KVM. The problem here (I think) is that /boot/efi is in /etc/fstab, and systemd fails to boot when anything in /etc/fstab doesn't work.

If I comment /boot/efi out of /etc/fstab, it boots just fine.

[Tycho Andersen (tycho-s)]

requested by smoser,

yakkety:~ cat /proc/partitions
major minor #blocks name

 254 0 31457280 vda
 254 1 31343599 vda1
 254 14 4096 vda14
 254 15 108544 vda15
 254 16 366 vdb
yakkety:~ blkid
/dev/vda1: LABEL="cloudimg-rootfs" UUID="072136f3-5666-4381-83bd-8d7175059be3" TYPE="ext4" PARTUUID="651ee987-3881-4315-91bd-7b4fa084d71a"
/dev/vda15: LABEL="UEFI" UUID="5EC7-12FA" TYPE="vfat" PARTUUID="035f3d9d-e9a5-47ef-9d4a-22911a283a10"
/dev/vdb: UUID="2016-09-21-08-57-02-00" LABEL="cidata" TYPE="iso9660"

[Tycho Andersen (tycho-s)]

yakkety:~ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
#LABEL=UEFI /boot/efi vfat defaults 0 0

[Scott Moser (smoser)]

Of interest here i think is:
Sep 21 14:15:48 ubuntu kernel: [ 358.567356] FAT-fs (vda14): invalid media value (0x7d)
Sep 21 14:15:48 ubuntu kernel: [ 358.567828] FAT-fs (vda14): Can't find a valid FAT filesystem
Sep 21 14:15:48 ubuntu kernel: [ 358.573850] ISOFS: Unable to identify CD-ROM format.
Sep 21 14:15:48 ubuntu kernel: [ 358.582031] XFS (vda14): Invalid superblock magic number
Sep 21 14:15:48 ubuntu kernel: [ 358.705645] FAT-fs (vda14): invalid media value (0x7d)
Sep 21 14:15:48 ubuntu kernel: [ 358.706621] FAT-fs (vda14): Can't find a valid FAT filesystem
Sep 21 14:15:48 ubuntu kernel: [ 358.712868] ufs: You didn't specify the type of your ufs filesystem

why is something looking at vda14 when blkid shows:
  /dev/vda15: LABEL="UEFI"

odd.

[Scott Moser (smoser)]

OK, i recreated with simple:
 uvt-kvm create sm-y1 release=yakkety
 uvt-kvm ssh --insecure sm-y1
 % echo "root:passw0rd" | sudo chpasswd
 % sudo apt-get update && sudo apt-get dist-upgrade && sudo reboot
 ...

then at the virsh console i see the fail, log in and:

root@sm-y1:~# mount /dev/disk/by-label/UEFI /mnt
[ 224.272443] FAT-fs (vda15): codepage cp437 not found
mount: wrong fs type, bad option, bad superblock on /dev/vda15,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@sm-y1:~# dmesg | tail
[ 4.468785] call_modprobe: nls_utf8 2
[ 4.473050] ISO 9660 Extensions: RRIP_1991A
[ 4.501051] audit: type=1400 audit(1474476002.188:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=543 comm="apparmor_parser"
[ 4.501270] audit: type=1400 audit(1474476002.188:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=543 comm="apparmor_parser"
[ 4.501471] audit: type=1400 audit(1474476002.188:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=543 comm="apparmor_parser"
[ 4.501656] audit: type=1400 audit(1474476002.188:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=543 comm="apparmor_parser"
[ 4.514047] audit: type=1400 audit(1474476002.200:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=566 comm="apparmor_parser"
[ 224.270568] FAT-fs (vda15): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
[ 224.271666] call_modprobe: nls_cp437 2
[ 224.272443] FAT-fs (vda15): codepage cp437 not found
root@sm-y1:~# find /lib/modules/ -name "*cp437*"
root@sm-y1:~# ls /lib/modules/
4.4.0-9136-generic 4.8.0-11-generic
root@sm-y1:~# find /lib/modules -name "nls_*"
/lib/modules/4.4.0-9136-generic/kernel/fs/nls/nls_iso8859-1.ko
/lib/modules/4.8.0-11-generic/kernel/fs/nls/nls_iso8859-1.ko

[Stefan Bader (smb)]

CONFIG_FAT_DEFAULT_IOCHARSET somehow got changed to utf8 instead of iso8859-1.

[Tim Gardner (timg-tpi)]

UBUNTU: [Config] CONFIG_VFAT_FS=y
UBUNTU: [Config] CONFIG_NLS_CODEPAGE_437=y
UBUNTU: [Config] CONFIG_FAT_DEFAULT_IOCHARSET=iso8859-1

Will be released with Ubuntu-4.8.0-15.16

[Martin Pitt (pitti)]

> CONFIG_FAT_DEFAULT_IOCHARSET somehow got changed to utf8 instead of iso8859-1.

Related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833238

TL;DR: Use this:
  CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
  FAT_DEFAULT_UTF8=y

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.8.0-15.16

---------------
linux (4.8.0-15.16) yakkety; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1626239

  * image won't boot after upgrading to yakkety's 4.8 kernel because efi
    (LP: #1626158)
    - [Config] CONFIG_FAT_DEFAULT_IOCHARSET=iso8859-1
    - [Config] CONFIG_NLS_CODEPAGE_437=y
    - [Config] CONFIG_VFAT_FS=y

  * Miscellaneous Ubuntu changes
    - SAUCE: seccomp: log actions even when audit is disabled

 -- Tim Gardner <email address hidden> Wed, 21 Sep 2016 06:41:03 -0600