[arm64] nova instances can't boot with 3.13.0-92

[Junien F]

Hi,

It would appear that arm64 nova instances can't boot on 3.13.0-92.

The serial console displays :

error: plain Image kernel not supported - rebuild with CONFIG_(U)EFI_STUB enabled.
unaligned pointer 0xb90002e058000c57
Aborted. Press any key to exit.

I suspect this is due to LP#1566221, more specifically :
   * [Config] CONFIG_EFI=n for arm64

This is pretty serious. I'm not 100% sure the above is the cause, but if so, it should be reverted asap.

Thank you.
---
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.21
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg:
 [ 27.314355] random: nonblocking pool is initialized
 [ 43.711734] init: jenkins-slave main process (1169) terminated with status 143
DistroRelease: Ubuntu 14.04
Ec2AMI: ami-000008ce
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.medium
Ec2Kernel: None
Ec2Ramdisk: None
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-85-generic root=UUID=88f61243-be05-46db-b772-d7ed13eb39de ro earlyprintk
ProcVersionSignature: Ubuntu 3.13.0-85.129-generic 3.13.11-ckt36
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-85-generic N/A
 linux-backports-modules-3.13.0-85-generic N/A
 linux-firmware 1.127.22
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty ec2-images
Uname: Linux 3.13.0-85-generic aarch64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True

[Junien F (axino)]

apport information

[Junien F (axino)]

apport information

[Junien F (axino)]

apport information

[Junien F (axino)]

apport information

[Junien F (axino)]

apport information

[Junien F (axino)]

apport information

[Junien F (axino)]

apport information

[Junien F (axino)]

apport information

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1608854

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Junien F (axino)]

I think all the files are here ?

[Tim Gardner (timg-tpi)]

Junien - can you use lts-xenial until I get this sorted out ? The arm64 backport to enable EFI as well as support secure boot signed module enforcement is going to require quite a bit of effort.

[Tim Gardner (timg-tpi)]

Now I remember why I thought it was OK to disable EFI in arm64 Trusty. v3.13 had no arm64 support for EFI to begin with. However, I backported the UEFI patchset from Utopic (v3.16) which _did_ have arm64 support. I thought it would be OK to simply disable EFI in Trusty, but clearly something went awry. I think I can get away with reverting arch/arm64 EFI commits since they appear to be well isolated.

Please try this test kernel at http://people.canonical.com/~rtg/arm64-efi-lp1608854/

[Tim Gardner (timg-tpi)]

Never mind the test kernel. Its bogus according to Dann Frazier. Back to the drawing board.

[Junien F (axino)]

OK. For now I'm using 3.13.0-91 or inferior, which works fine.

[Tim Gardner (timg-tpi)]

I've refreshed the test kernel at http://people.canonical.com/~rtg/arm64-efi-lp1608854. Please try arm64 which now has a backport from Utopic (v3.16).

[dann frazier (dannf)]

Kernel from comment #15 boots fine in EFI mode in QEMU:

ubuntu@ubuntu:~$ cat /proc/version
Linux version 3.13.0-94-generic (rtg@gloin) (gcc version 4.8.2 20140110 (prerelease) [ibm/gcc-4_8-branch merged from gcc-4_8-branch, revision 205847] (Ubuntu/Linaro 4.8.2-13ubuntu1) ) #141 SMP Mon Aug 8 15:06:28 UTC 2016

[Tim Gardner (timg-tpi)]

git://kernel.ubuntu.com/rtg/ubuntu-trusty.git arm64-efi-lp1608854

I backported EFI functionality for arm64 and amd64 from Utopic (v3.16) to Trusty (v3.13) because much of the arch dependent infrastructure in v3.16 is required in order to support signed module enforcement with a MOK managed key override (for externally built modules such as DKMS). See the "UBUNTU: SAUCE: UEFI:" series of patches. The approach that I took was to revert all EFI functionality in Trusty back to vanilla v3.13, then cherry-picked and backported all EFI patches from Utopic (including stable updates). Ultimately, there is little diff between the Trusty backport and Utopic in the directories and files that matter (arch/x86/platform/efi, drivers/firmware/efi, arch/arm64/kernel/efi*). This backport is possible because EFI is almost completely isolated from the rest of the kernel. However, the patchset is so large that it is essentially unreviewable. I think the only way to verify this patchset is through testing. Fortunately it is one of those features that either works or it doesn't. Thus far my test results and those of #16 are positive for both arches.

[Junien F (axino)]

So, just to be clear, you want us to test http://people.canonical.com/~rtg/arm64-efi-lp1608854/ ? This is the kernel built from git://kernel.ubuntu.com/rtg/ubuntu-trusty.git arm64-efi-lp1608854 ?

[Tim Gardner (timg-tpi)]

Junien - yes, that is the one.

[Junien F (axino)]

Looks good to me :

$ cat /proc/version
Linux version 3.13.0-94-generic (rtg@gloin) (gcc version 4.8.2 20140110 (prerelease) [ibm/gcc-4_8-branch merged from gcc-4_8-branch, revision 205847] (Ubuntu/Linaro 4.8.2-13ubuntu1) ) #141 SMP Tue Aug 9 14:58:41 UTC 2016

[Tim Gardner (timg-tpi)]

This backport patchset is not successful. There are multiple platform regressions that appear intransigent. Therefore, I've reverted all patches related to Secure Boot and signed module enforcement. This should restore arm64 EFI functionality. Platforms wishing to use secure boot and signed module enforcement must install lts-xenial.

git://kernel.ubuntu.com/rtg/ubuntu-trusty.git arm64-efi-lp1608854-revert

test kernel at http://people.canonical.com/~rtg/arm64-efi-lp1608854-revert/

[Tim Gardner (timg-tpi)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Tim Gardner (timg-tpi)]

Junien - please try the refreshed kernel with boot issue fix at http://people.canonical.com/~rtg/arm64-efi-lp1608854/ - You should hopefully not see any arm64 regressions.

[Junien F (axino)]

Hi Tim,

I can't see any arm64 kernel in http://people.canonical.com/~rtg/arm64-efi-lp1608854/, only amd64.

[Tim Gardner (timg-tpi)]

Junien - oops, there are arm64 binaries there now.

[Junien F (axino)]

It boots fine !

$ cat /proc/version
Linux version 3.13.0-97-generic (rtg@gloin) (gcc version 4.8.2 20140110 (prerelease) [ibm/gcc-4_8-branch merged from gcc-4_8-branch, revision 205847] (Ubuntu/Linaro 4.8.2-13ubuntu1) ) #144 SMP Mon Sep 12 16:38:12 UTC 2016

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-96.143

---------------
linux (3.13.0-96.143) trusty; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1618083

  * CVE-2015-8767
    - sctp: Prevent soft lockup when sctp_accept() is called during a timeout
      event

  * MacBookPro11,4 fails to poweroff or suspend (LP: #1587714)
    - SAUCE: PCI: Workaround to enable poweroff on Mac Pro 11

  * 3.13: libvirtd: page allocation failure: order:4, mode:0x1040d0
    (LP: #1616193)
    - vhost-net: extend device allocation to vmalloc
    - vhost-net: don't open-code kvfree

  * [arm64] nova instances can't boot with 3.13.0-92 (LP: #1608854)
    - Revert "UBUNTU: [Config] CONFIG_EFI=n for arm64"
    - Revert "UBUNTU: SAUCE: UEFI: Set EFI_SECURE_BOOT bit in x86_efi_facility"
    - Revert "UBUNTU: SAUCE: UEFI: Add secure boot and MOK SB State disabled
      sysctl"
    - Revert "UBUNTU: SAUCE: UEFI: Display MOKSBState when disabled"
    - Revert "UBUNTU: SAUCE: UEFI: efi: Disable secure boot if shim is in insecure
      mode"
    - Revert "UBUNTU: SAUCE: UEFI MODSIGN: Import certificates from UEFI Secure
      Boot"
    - Revert "UBUNTU: SAUCE: UEFI: efi: Make EFI_SECURE_BOOT_SIG_ENFORCE depend on
      EFI"
    - Revert "UBUNTU: SAUCE: UEFI: Add option to automatically enforce module
      signatures when in Secure Boot mode"
    - Revert "UBUNTU: [Config] CONFIG_EFI_SECURE_BOOT_SIG_ENFORCE=y"
    - Revert "UBUNTU: SAUCE: UEFI: x86: Restrict MSR access when module loading is
      restricted"
    - Revert "UBUNTU: SAUCE: UEFI: kexec: Disable at runtime if the kernel
      enforces module loading restrictions"
    - Revert "UBUNTU: SAUCE: UEFI: acpi: Ignore acpi_rsdp kernel parameter when
      module loading is restricted"
    - Revert "UBUNTU: SAUCE: UEFI: Restrict /dev/mem and /dev/kmem when module
      loading is restricted"
    - Revert "UBUNTU: SAUCE: UEFI: asus-wmi: Restrict debugfs interface when
      module loading is restricted"
    - Revert "UBUNTU: SAUCE: UEFI: ACPI: Limit access to custom_method"
    - Revert "UBUNTU: SAUCE: UEFI: x86: Lock down IO port access when module
      security is enabled"
    - Revert "UBUNTU: SAUCE: UEFI: PCI: Lock down BAR access when module security
      is enabled"
    - Revert "UBUNTU: SAUCE: UEFI: Add secure_modules() call"
    - Revert "x86/efi: Build our own EFI services pointer table"
    - Revert "efi: Add separate 32-bit/64-bit definitions"

  * [Hyper-V] storvsc messages for CD-ROM medium not present tray closed
    (LP: #1590655)
    - scsi: storvsc: Filter out storvsc messages CD-ROM medium not present

  * CVE-2016-3841
    - ipv6: add complete rcu protection around np->opt

 -- Kamal Mostafa <email address hidden> Tue, 16 Aug 2016 10:20:51 -0700

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Seth Forshee (sforshee)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[dann frazier (dannf)]

ubuntu@ubuntu:~$ archdetect
arm64/efi
ubuntu@ubuntu:~$ cat /proc/version
Linux version 3.13.0-99-generic (buildd@bos01-arm64-032) (gcc version 4.8.4 (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3) ) #146-Ubuntu SMP Wed Oct 12 20:56:37 UTC 2016

[dann frazier (dannf)]

The most recent d-i for trusty was built w/ 3.13.0-110, so I'll go ahead and mark fix released. I suspect, however, that this kernel still won't boot on any modern OpenStack deployment, since the EFI firmware will require ACPI support that was not available in 3.13.