boot fails on cryptsetup random swap partition with "invalid argument"

[arsen stasic]

Booting with systemd fails always at same stage.
Using upstart just works.

grep GRUB_CMDLINE_LINUX /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="systemd.log_target=kmsg systemd.log_level=debug"

grep debug /boot/grub/grub.cfg
linux /vmlinuz-3.19.0-13-generic root=/dev/mapper/vg-root ro systemd.log_target=kmsg systemd.log_level=debug

Sadly systemd-debug doesn't reveals more information what fails :-(

lsb_release -rd
Description: Ubuntu Vivid Vervet (development branch)
Release: 15.04

[arsen stasic (arsen)]

@Martin Pitti: As you requested I have filed this systemd bug. What would you suggest to debug this issue?

[Martin Pitt (pitti)]

This smells a lot like bug 953875. To confirm, can you please give me the output of

  cat /etc/crypttab
  sudo blkid

Does it boot again if you comment out the cryptswap1 device from /etc/crypttab?

[arsen stasic (arsen)]

Hi Martin,

Thanks, if I disable vg-swap it works!
Is there already a fix for this?
Can I try any updated software which fixes this?

cat /etc/crypttab
# <target name> <source device> <key file> <options>
vg-swap /dev/dm-6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

blkid
/dev/sda1: UUID="ojI1MN-U1rH-rsCw-HeS8-y1wj-fr0T-g0QCzi" TYPE="LVM2_member" PARTUUID="25600bb7-01"
/dev/mapper/vg-root: UUID="42926330-dfed-499a-89d3-ad4189c67924" TYPE="ext4"
/dev/mapper/vg-boot: UUID="32b2d107-6180-4a5e-bed3-9fb532964b40" TYPE="ext4"
/dev/mapper/vg-home: UUID="3fc9dd10-d6a2-494e-bab0-ca245ac4064f" TYPE="ext4"
/dev/mapper/vg-tmp: UUID="7bc1332b-4522-45f2-b899-e048cbe29663" TYPE="ext4"
/dev/mapper/vg-usr: UUID="930c046c-32ca-4501-a3da-65ed1e5cec02" TYPE="ext4"
/dev/mapper/vg-var: UUID="6330c02a-a448-4b97-9a97-2b4801a6f32d" TYPE="ext4"
/dev/mapper/vg-swap: UUID="fddf0272-8330-438a-b622-66853baf4c46" TYPE="swap"
/dev/mapper/vg-opt: UUID="597f4903-0555-43ab-9756-9417267543c5" TYPE="ext4"

[Martin Pitt (pitti)]

Ah, so not quite bug 953875, but a custom LVM setup. Adding /dev/dm-6 to crypttab does sound dangerous, as these are not very stable. Adding /dev/mapper/vg-swap or UUID=fdd.. to crypttab is usually better. What does

  ls -l /dev/mapper/vg-swap

say, does it actually point to dm-6? If not, that's the problem, can you try with the stable symlinks?

If it does point to dm-6, can you please boot with the debug shell as per /usr/share/doc/systemd/README.Debian, and when it hangs capture "journalctl -b > /home/you/journal.txt" and then reboot and attach journal.txt here? Thanks!

[Martin Pitt (pitti)]

Sorry, my previous reply was quite bogus, too early still. I suppose that output comes from under upstart, so /dev/mapper/vg-swap is the *result* of crypttab.

The blurred screenshot says "invalid argument". So please leave that line disabled, and once the system is running, try

  sudo SYSTEMD_LOG_LEVEL=debug strace -fvvs1024 -o /tmp/t /lib/systemd/systemd-cryptsetup attach vg-swap dev/dm-6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

This should give some "invalid argument" at some point, does it? For me this command works fine on current vivid. If it reproduces the "invalid argument" for you, please copy&paste the output and /tmp/t. Thanks!

[arsen stasic (arsen)]

SYSTEMD_LOG_LEVEL=debug strace -fvvs1024 -o /tmp/t /lib/systemd/systemd-cryptsetup attach vg-swap dev/dm-6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

crypt_init() failed: Block device required

if I put a / in front of /dev/dm-6 I'm getting the flowing:
SYSTEMD_LOG_LEVEL=debug strace -fvvs1024 -o /tmp/tt /lib/systemd/systemd-cryptsetup attach vg-swap /dev/dm-6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

Set cipher aes, mode cbc-essiv:sha256, key size 256 bits for device /dev/dm-6.
Failed to activate with key file '/dev/urandom': Invalid argument
Please enter passphrase for disk vg-swap on none!

<Just pressing Enter, because I can't remember the passphrase any more>

Loading of cryptographic parameters failed: Invalid argument
Failed to activate: Invalid argument

But this could it be, it waits for input and because no one is coming it waits for ever ;-)

do you still need the output of /tmp/tt?

[Martin Pitt (pitti)]

> if I put a / in front of /dev/dm-6

Oops, sorry :-)

> <Just pressing Enter, because I can't remember the passphrase any more>

No point, no human being will ever know it. It will get reinitialized with a random key on every boot.

> do you still need the output of /tmp/t

That would be helpful indeed. I still have no clue what causes the "invalid argument", the above command works just fine for me if I run it on a partition here.

[arsen stasic (arsen)]

I have now booted with upstart as init, because I've forgotten to commentout /etc/crypttab.
I hope that the attached /tmp/t still makes sense even if booted with upstart

[Martin Pitt (pitti)]

Hang on, I think there's something wedged here. You already have a /dev/mapper/vg-swap but this is *not* being created by crypttab as that obviously fails. This is also what confused me at first in comment 5. If you already have an LV which is called vg-swap, you can't additionally define a cryptswap target with the same name. I. e. you are trying to have /dev/mapper/vg-swap being the underlying encrypted device for the to-be-created /dev/mapper/vg-swap encrypted device.

Can you please rename vg-swap in /etc/crypttab and /etc/fstab to something else, like "cryptswap1"?

[arsen stasic (arsen)]

That did the trick!
I booted now with systemd without any issues!
Thanks

 cat /etc/crypttab
# <target name> <source device> <key file> <options>
#vg-swap /dev/dm-6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256
cryptswap1 /dev/dm-6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

grep cryptswap1 /etc/fstab
/dev/mapper/cryptswap1 none swap sw 0 0

[arsen stasic (arsen)]

This is quite interesting although
cryptswap1 should be on /dev/dm-6 it's mounted at dm-8

grep cryptswap1 /etc/crypttab
cryptswap1 /dev/dm-6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

ls -l /dev/mapper/cryptswap1
lrwxrwxrwx 1 root root 7 Apr 17 09:17 /dev/mapper/cryptswap1 -> ../dm-8

swapon -s
Filename Type Size Used Priority
/dev/dm-8 partition 4194300 12 -1

[Martin Pitt (pitti)]

> cryptswap1 should be on /dev/dm-6 it's mounted at dm-8

No, no. The *underlying* encrypted device /dev/mapper/vg-swap is dm-6. cryptsetup builds a new device /dev/mapper/cryptswap1 -> dm8 from that which is the unencrypted "view" of that, but an entirely new block device. Therefore it gets a new number. Closing then, as this is a local misconfiguration.