initramfs in cloud-images does not contain crypt modules

[Scott Moser]

overlayroot with crypt seems busted at the moment in trusty cloud images.
To show this:
1.) boot an image
  I booted us-east-1/ebs/ubuntu-trusty-daily-amd64-server-20140108 (ami-ef665086) as an m1.small on amazon with:
  euca-run-instances -t m1.small ami-ef665086

 2.) configure overlayroot
  echo "overlayroot='crypt:dev=xvdb'" | sudo tee -a /etc/overlayroot.conf
 3.) reboot

You expect to be booted into crypt overlayroot, but you wont be.

Further investigation shows the following in /dev/.initramfs/overlayroot.log:
| /dev/disk/by-label/cloudimg-rootfs/etc/overlayroot.local.conf set cfgdisk='LABEL=OROOTCFG'
| get_cfg(LABEL=OROOTCFG): not present
| fstype=ext4 pass= mapname=secure
| mkfs=1 dev=/dev/xvdb timeout=0
| [warning]: setting up new luks device at /dev/xvdb
| [failure]: luksFormat /dev/xvdb failed
| [failure]: failed setup crypt for crypt:dev=xvdb (per /dev/disk/by-label/cloudimg-rootfs/etc/overlayroot.conf)

Then, console output shows:

| Warning: overlayroot: setting up new luks device at /dev/xvdb
| 6 bytes were erased at offset 0x0 (crypto_LUKS)
| they were: 4c 55 4b 53 ba be
| [57432116.935753] device-mapper: table: 252:0: crypt: Error allocating crypto tfm
| [57432116.935767] device-mapper: ioctl: error adding target to table
| device-mapper: reload ioctl on temporary-cryptsetup-171 failed: No such file or directory
| Failed to open temporary keystore device.
| device-mapper: remove ioctl on temporary-cryptsetup-171 failed: No such device or address
| device-mapper: reload ioctl on temporary-cryptsetup-171 failed: No such device or address
| device-mapper: remove ioctl on temporary-cryptsetup-171 failed: No such device or address
| device-mapper: remove ioctl on temporary-cryptsetup-171 failed: No such device or address
| device-mapper: remove ioctl on temporary-cryptsetup-171 failed: No such device or address
| device-mapper: remove ioctl on temporary-cryptsetup-171 failed: No such device or address
| Failure: overlayroot: luksFormat /dev/xvdb failed
| Failure: overlayroot: failed setup crypt for crypt:dev=/dev/xvdb (per /dev/disk/by-label/cloudimg-rootfs/etc/overlayroot.conf)
| done.

The root of the problem seems to be that initramfs is stale, or for some reason does not contain necessary crypt modules. Running 'update-initramfs -u' will fix this problem.

See the diff between the shipped version of initramfs and the newly updated one.

--- list.orig 2014-01-08 19:34:30.517630999 +0000
+++ list.new 2014-01-08 19:34:18.313630999 +0000
@@ -1,4 +1,4 @@
-$ lsinitramfs /boot/initrd.img-3.12.0-7-generic.orig | sort > list.orig
+$ lsinitramfs /boot/initrd.img-3.12.0-7-generic | sort > list.new
 .
 bin
 bin/busybox
@@ -29,7 +29,7 @@
 bin/sha512sum
 bin/sleep
 bin/udevadm
-/boot/initrd.img-3.12.0-7-generic.orig
+/boot/initrd.img-3.12.0-7-generic
 conf
 conf/arch.conf
 conf/conf.d
@@ -73,7 +73,17 @@
 lib/modules
 lib/modules/3.12.0-7-generic
 lib/modules/3.12.0-7-generic/kernel
+lib/modules/3.12.0-7-generic/kernel/arch
+lib/modules/3.12.0-7-generic/kernel/arch/x86
+lib/modules/3.12.0-7-generic/kernel/arch/x86/crypto
+lib/modules/3.12.0-7-generic/kernel/arch/x86/crypto/ablk_helper.ko
+lib/modules/3.12.0-7-generic/kernel/arch/x86/crypto/aesni-intel.ko
+lib/modules/3.12.0-7-generic/kernel/arch/x86/crypto/aes-x86_64.ko
+lib/modules/3.12.0-7-generic/kernel/arch/x86/crypto/glue_helper.ko
 lib/modules/3.12.0-7-generic/kernel/crypto
+lib/modules/3.12.0-7-generic/kernel/crypto/cryptd.ko
+lib/modules/3.12.0-7-generic/kernel/crypto/gf128mul.ko
+lib/modules/3.12.0-7-generic/kernel/crypto/lrw.ko
 lib/modules/3.12.0-7-generic/kernel/crypto/xor.ko
 lib/modules/3.12.0-7-generic/kernel/drivers
 lib/modules/3.12.0-7-generic/kernel/drivers/ata

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: overlayroot 0.21ubuntu2
ProcVersionSignature: User Name 3.12.0-7.15-generic 3.12.4
Uname: Linux 3.12.0-7-generic x86_64
ApportVersion: 2.12.7-0ubuntu6
Architecture: amd64
Date: Wed Jan 8 19:35:58 2014
Ec2AMI: ami-ef665086
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1e
Ec2InstanceType: m1.small
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-initramfs-tools
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.overlayroot.conf: 2014-01-08T18:53:17.189849

[Scott Moser (smoser)]

This looks like a build-time bug in both overlayroot and crypt-setup.
 overlayroot/hooks/overlayroot does:
    egrep -qswo "aes" /proc/cpuinfo && manual_add_modules aesni_intel || true

And cryptsetup's /usr/share/initramfs-tools/hooks/cryptroot does:
                # Load hardware aes module
                if cpu_has_aesni; then
                        echo aesni
                fi
and
cpu_has_aesni() {
        return $(grep -q "^flags\s*:\s*.*aes" /proc/cpuinfo)
}

Seems like both of these should not be dynamic based on host cpu support, but constant built in on module availability in the kernel.

[Scott Moser (smoser)]

Seems that sometime in saucy we got:

  * debian/initramfs/cryptroot-hook:
    - Do not unconditionally include cryptsetup utils in the initramfs.
    - Do not include any modules or utils in the initramfs, unless
      rootfs/resume devices are encrypted or CRYPTSETUP is set to 'y' in
      the initramfs.conf configuration file.

[Scott Moser (smoser)]

@xnox,
  Could you explain why you did this? Generally debian/ubuntu initramfs policy is to build a "large generic" initramfs instead of a customized one per configuration of the system. Ie, lsinitramfs shows me some net/ppp/ drivers that i'm a lot less likely to use in my initramfs than crypt.

[Steve Langasek (vorlon)]

On Wed, Jan 08, 2014 at 10:04:31PM -0000, Scott Moser wrote:
> Seems that sometime in saucy we got:

> * debian/initramfs/cryptroot-hook:
> - Do not unconditionally include cryptsetup utils in the initramfs.
> - Do not include any modules or utils in the initramfs, unless
> rootfs/resume devices are encrypted or CRYPTSETUP is set to 'y' in
> the initramfs.conf configuration file.

That is unrelated to the aes modules being conditional on detected aes
support.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.22ubuntu1

---------------
cloud-initramfs-tools (0.22ubuntu1) trusty; urgency=medium

  * correctly identify partition and block device for devices named
    like devnameXpN, such as mmcblk0p1 (LP: #1236380).
  * always include aesni_intel rather than doing it if there is
    cpu support (LP: #1267225).
 -- Scott Moser <email address hidden> Mon, 03 Feb 2014 22:09:00 +0200

[Robert C Jennings (rcj)]

Latest daily build for 14.04 has fix and resolves this issue. (Verified on 20120206 daily build)

[Rolf Leggewie (r0lf)]

saucy has seen the end of its life and is no longer receiving any updates. Marking the saucy task for this ticket as "Won't Fix".