cryptsetup fails to decrypt root partion during boot

Bug #1986623 reported by Lorenzhuether@hotmail.de
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
cryptsetup (Ubuntu)
Confirmed
Critical
Unassigned
linux (Ubuntu)
Incomplete
Critical
Unassigned

Bug Description

During boot, cryptsetup fails to decrypt the root partition in a, seemingly, non-deterministic fashion. I know that the password is correct and that the keymap is not a fault either, because I have specifically chosen the very weak password "123456" for testing purposes. A hardware defect seems also rather unlikely as this behavior does not affect other Linux distributions or FreeBSD. Earlier Ubuntu versions do not seem to be affected either, as this bug appears to have been introduced during a kernel update in 20.04 and persists throughout 20.04-22.04. Unfortunately I cannot pinpoint the exact kernel update that introduced this bug. I have appended the output of cryptsetup when manually called from the initramfs shell. Here the second attempt succeeded in decrypting the root partition, however, it usually takes a lot more attempts to do so.

As for some additional information, I can decrypt the same luks partition from a live USB without any problems whatsoever.

echo 123456 | cryptsetup open --type luks --debug /dev/nvme0n1p3 nvme0n1p3_crypt
# cryptsetup 2.4.3 processing "cryptsetup open --type luks --debug /dev/nvme0n1p3 nvme0n1p3_crypt"
# Running command open.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/nvme0n1p3.
# Trying to open and read device /dev/nvme0n1p3 with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/nvme0n1p3.
# Crypto backend (OpenSSL 3.0.2 15 Mar 2022 [default]) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.15.0-46-generic x86_64.
# Loading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/nvme0n1p3.
# Opening lock resource file /run/cryptsetup/L_259:3
# Verifying lock handle for /dev/nvme0n1p3.
# Device /dev/nvme0n1p3 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/nvme0n1p3
# Verifying locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:99172356e66a2fec247b1e5c758af8bc1338a3fb8bd973aab5e1512a93b2dbdc (on-disk)
# Checksum:99172356e66a2fec247b1e5c758af8bc1338a3fb8bd973aab5e1512a93b2dbdc (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/nvme0n1p3
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:655a50aef64b6fd4e10b8863df72d7fc62a7020f74684cb3419d4e22adb6fd9c (on-disk)
# Checksum:655a50aef64b6fd4e10b8863df72d7fc62a7020f74684cb3419d4e22adb6fd9c (in-memory)
# Device size 497776852992, offset 16777216.
# Device /dev/nvme0n1p3 READ lock released.
# PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# Activating volume nvme0n1p3_crypt using token (any type) -1.
# dm version [ opencount flush ] [16384] (*1)
# dm versions [ opencount flush ] [16384] (*1)
# Detected dm-ioctl version 4.45.0.
# Detected dm-crypt version 1.23.0.
# Device-mapper backend running with UDEV support enabled.
# dm status nvme0n1p3_crypt [ opencount noflush ] [16384] (*1)
No usable token is available.
# STDIN descriptor passphrase entry requested.
# Activating volume nvme0n1p3_crypt [keyslot -1] using passphrase.
# dm versions [ opencount flush ] [16384] (*1)
# dm status nvme0n1p3_crypt [ opencount noflush ] [16384] (*1)
# Keyslot 0 priority 1 != 2 (required), skipped.
# Keyslot 1 priority 1 != 2 (required), skipped.
# Trying to open LUKS2 keyslot 0.
# Running keyslot key derivation.
# Reading keyslot area [0x8000].
# Acquiring read lock for device /dev/nvme0n1p3.
# Opening lock resource file /run/cryptsetup/L_259:3
# Verifying lock handle for /dev/nvme0n1p3.
# Device /dev/nvme0n1p3 READ lock taken.
# Reusing open ro fd on device /dev/nvme0n1p3
# Device /dev/nvme0n1p3 READ lock released.
# Verifying key from keyslot 0, digest 0.
# Digest 0 (pbkdf2) verify failed with -1.
# Trying to open LUKS2 keyslot 1.
# Running keyslot key derivation.
# Reading keyslot area [0x47000].
# Acquiring read lock for device /dev/nvme0n1p3.
# Opening lock resource file /run/cryptsetup/L_259:3
# Verifying lock handle for /dev/nvme0n1p3.
# Device /dev/nvme0n1p3 READ lock taken.
# Reusing open ro fd on device /dev/nvme0n1p3
# Device /dev/nvme0n1p3 READ lock released.
# Verifying key from keyslot 1, digest 0.
# Digest 0 (pbkdf2) verify failed with -1.
# Releasing crypt device /dev/nvme0n1p3 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/nvme0n1p3.
# Unlocking memory.
Command failed with code -2 (no permission or bad passphrase).

echo 123456 | cryptsetup open --type luks --debug /dev/nvme0n1p3 nvme0n1p3_crypt
# cryptsetup 2.4.3 processing "cryptsetup open --type luks --debug /dev/nvme0n1p3 nvme0n1p3_crypt"
# Running command open.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/nvme0n1p3.
# Trying to open and read device /dev/nvme0n1p3 with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/nvme0n1p3.
# Crypto backend (OpenSSL 3.0.2 15 Mar 2022 [default]) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.15.0-46-generic x86_64.
# Loading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/nvme0n1p3.
# Opening lock resource file /run/cryptsetup/L_259:3
# Verifying lock handle for /dev/nvme0n1p3.
# Device /dev/nvme0n1p3 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/nvme0n1p3
# Verifying locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:99172356e66a2fec247b1e5c758af8bc1338a3fb8bd973aab5e1512a93b2dbdc (on-disk)
# Checksum:99172356e66a2fec247b1e5c758af8bc1338a3fb8bd973aab5e1512a93b2dbdc (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/nvme0n1p3
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:655a50aef64b6fd4e10b8863df72d7fc62a7020f74684cb3419d4e22adb6fd9c (on-disk)
# Checksum:655a50aef64b6fd4e10b8863df72d7fc62a7020f74684cb3419d4e22adb6fd9c (in-memory)
# Device size 497776852992, offset 16777216.
# Device /dev/nvme0n1p3 READ lock released.
# PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# Activating volume nvme0n1p3_crypt using token (any type) -1.
# dm version [ opencount flush ] [16384] (*1)
# dm versions [ opencount flush ] [16384] (*1)
# Detected dm-ioctl version 4.45.0.
# Detected dm-crypt version 1.23.0.
# Device-mapper backend running with UDEV support enabled.
# dm status nvme0n1p3_crypt [ opencount noflush ] [16384] (*1)
No usable token is available.
# STDIN descriptor passphrase entry requested.
# Activating volume nvme0n1p3_crypt [keyslot -1] using passphrase.
# dm versions [ opencount flush ] [16384] (*1)
# dm status nvme0n1p3_crypt [ opencount noflush ] [16384] (*1)
# Keyslot 0 priority 1 != 2 (required), skipped.
# Keyslot 1 priority 1 != 2 (required), skipped.
# Trying to open LUKS2 keyslot 0.
# Running keyslot key derivation.
# Reading keyslot area [0x8000].
# Acquiring read lock for device /dev/nvme0n1p3.
# Opening lock resource file /run/cryptsetup/L_259:3
# Verifying lock handle for /dev/nvme0n1p3.
# Device /dev/nvme0n1p3 READ lock taken.
# Reusing open ro fd on device /dev/nvme0n1p3
# Device /dev/nvme0n1p3 READ lock released.
# Verifying key from keyslot 0, digest 0.
# Loading key (64 bytes, type logon) in thread keyring.
# dm versions [ opencount flush ] [16384] (*1)
# dm status nvme0n1p3_crypt [ opencount noflush ] [16384] (*1)
# Calculated device size is 972187648 sectors (RW), offset 32768.
# DM-UUID is CRYPT-LUKS2-a41dd71155cc4b6ca29d391c500c546d-nvme0n1p3_crypt
# Udev cookie 0xd4d65d7 (semid 0) created
# Udev cookie 0xd4d65d7 (semid 0) incremented to 1
# Udev cookie 0xd4d65d7 (semid 0) incremented to 2
# Udev cookie 0xd4d65d7 (semid 0) assigned to CREATE task(0) with flags DISABLE_LIBRARY_FALLBACK (0x20)
# dm create nvme0n1p3_crypt CRYPT-LUKS2-a41dd71155cc4b6ca29d391c500c546d-nvme0n1p3_crypt [ opencount flush ] [16384] (*1)
# dm reload (253:0) [ opencount flush securedata ] [16384] (*1)
# dm resume nvme0n1p3_crypt [ opencount flush securedata ] [16384] (*1)
# nvme0n1p3_crypt: Stacking NODE_ADD (253,0) 0:6 0660 [trust_udev]
# nvme0n1p3_crypt: Stacking NODE_READ_AHEAD 256 (flags=1)
# Udev cookie 0xd4d65d7 (semid 0) decremented to 1
# Udev cookie 0xd4d65d7 (semid 0) waiting for zero
# Udev cookie 0xd4d65d7 (semid 0) destroyed
# nvme0n1p3_crypt: Skipping NODE_ADD (253,0) 0:6 0660 [trust_udev]
# nvme0n1p3_crypt: Processing NODE_READ_AHEAD 256 (flags=1)
# nvme0n1p3_crypt (253:0): read ahead is 256
# nvme0n1p3_crypt: retaining kernel read ahead of 256 (requested 256)
Key slot 0 unlocked.
# Releasing crypt device /dev/nvme0n1p3 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/nvme0n1p3.
# Unlocking memory.
Command successful.

Release and package version below:

lsb_release -rd
Description: Ubuntu 22.04.1 LTS
Release: 22.04

apt-cache policy cryptsetup
cryptsetup:
  Installed: 2:2.4.3-1ubuntu1.1
  Candidate: 2:2.4.3-1ubuntu1.1
  Version table:
 *** 2:2.4.3-1ubuntu1.1 500
        500 http://de.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2:2.4.3-1ubuntu1 500
        500 http://de.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2022-08-15 (1 days ago)
InstallationMedia: Ubuntu 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809.1)
Package: linux
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 5.15.0-46.49-generic 5.15.39
Tags: jammy
Uname: Linux 5.15.0-46-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
cmdline: BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/vgubuntu-root ro quiet splash break=mountroot
crypttab: nvme0n1p3_crypt UUID=a41dd711-55cc-4b6c-a29d-391c500c546d none luks,discard

Steve Langasek (vorlon)
Changed in cryptsetup (Ubuntu):
importance: Undecided → Critical
Changed in linux (Ubuntu):
importance: Undecided → Critical
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1986623

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Lorenzhuether@hotmail.de (lorenzhuether) wrote : Dependencies.txt

apport information

tags: added: apport-collected jammy
description: updated
Revision history for this message
Lorenzhuether@hotmail.de (lorenzhuether) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Lorenzhuether@hotmail.de (lorenzhuether) wrote : ProcEnviron.txt

apport information

Revision history for this message
Lorenzhuether@hotmail.de (lorenzhuether) wrote : fstab.txt

apport information

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cryptsetup (Ubuntu):
status: New → Confirmed
Revision history for this message
fedechko_fedechko (mrflo777) wrote (last edit ):

having same issue, not clean ubuntu though

5.15.0-43 linux core boots fine, cryptsetup accepts correct password
5.15.0-46 core accepts no password, correct or not

upd: problem still persists all the way to 5.15.0-50 core

upd2: my cpu was AMD Ryzen 3200G at that time, now i have no access to it so no cpuinfo.
i was able to fix that behaviour through a live gparted, turns out that prior to 5.15.0-43 keyboard layout in cryptsetup prompt was cyrillic (with no indication), so it was during the initial encryption setup. so password was in cyrillic and after 5.15.0-46 default layout became english. thats the only explanation i can give but it worked. (sorry for bad wording, not native english speaker)

Revision history for this message
Yudi (yudi-v) wrote (last edit ):

Having same issue, just rebooted and now cannot login to the system, does not accept the password, was working perfectly fine prior to reboot.

This is on a box that was recently setup with 22.04LTS with ZFS + encryption via the install options.
Happy to contribute any additional info needed.

Revision history for this message
Patrick B (blue-noon) wrote (last edit ):

Having the same issue with an existing installation of kubuntu. Even when installing a fresh kubuntu on the same hardware (only difference on a brand new nvme drive) I have the issue after first boot even with that fresh Kubuntu 22.04.1 LTS. See https://askubuntu.com/q/1454265/1671849

I tried with Linux 5.15.0-43-generic x86_64. and it didn't work, so I don't know why that kernel worked for fedechko comment#7, maybe it is not the kernel.

Revision history for this message
Patrick B (blue-noon) wrote :

The original poster uses an AMD system. My affected system also uses an AMD (part of /proc/cpunifo attached). What are other affected systems using?

Revision history for this message
Gary Brainin (brainin) wrote :

I have an Intel Core i5-4590.

Also just tried a fresh install (Ubuntu 22.04) on a spare hard drive, and it boots fine using kernel 5.15.0-60-generic. Note that spare is a traditional hard drive and main drive is an SSD; next I'll try reinstalling on the SSD.

Revision history for this message
Gary Brainin (brainin) wrote :

Fresh install worked until I did today's recommended updates, then the problem recurred. Not sure exactly what was updated; I thought /var/log/dpkg.log was supposed to be human-readable, but mine isn't.

Revision history for this message
Gary Brainin (brainin) wrote :

I am able to reproduce the error when I install the new 5.19.0-32 kernel.

Also, when I install the package "Firmware for Linux kernel drivers," it modifies the files for the older kernels (5.15.0-60 and 5.15.0-43), making them have the same problem. The 5.15.0-60 file goes from 111,616,247 bytes to 111,639,335; 5.15.0-43 goes from 110,310,042 to 24,544,316 (not a typo, it's about 25% of the former size). So I can't even choose an earlier kernel from the grub menu.

The error seems to be associated with some change in the kernel or kernel related items.

Revision history for this message
Gary Brainin (brainin) wrote :

Correction: It is not the kernel itself I was talking about, above. It's the initrd.img file associated with the kernel.

Note also that replacing that file with the old version eliminates the problem (at least until some update changes it again).

Revision history for this message
Gary Brainin (brainin) wrote :

I'm getting a bit over my head, but it is seeming like there's a problem with creating a new initrd.

When I decrypt (using unmkinitramfs) the initrd.img file, I see that the "good" version has a /main/cryptroot/crypttab file which appears identical to /etc/crypttab. The "bad" version has an empty file there (0 bytes).

Attempting to manually repair (using sudo update-initramfs -u) gives the following error:
cryptsetup: WARNING: target 'tmpData' not found in /etc/crypttab
I: The initramfs will attempt to resume from /dev/dm-2
I: (/dev/mapper/vgubuntu-swap_1)
I: Set RESUME variable to override this.

The file thus generated has a zero-byte crypttab file as before.

Therefore, I'm thinking the problem is here, in the generation of the initramfs. Anyone have a clue from the above? Should I be posting a new bug report indicating that update-initramfs is failing?

Revision history for this message
Gary Brainin (brainin) wrote :

Success?

Based only on the error message, I tried defining tmpData in /etc/crypttab. Previously, the file consisted of one line:

sda6_crypt UUID={uuid} none luks,discard

I added a second line:

tmpData UUID={uuid} none luks,discard

and re-ran update-initramfs. It still produced the other messages above (lines starting with I:), but on reboot it did not drop me into a busybox shell.

Note that it did ask me for the password for tmpData, where before it was asking for the password for /dev/sda3 (the encrypted partition), but otherwise it appears to have booted normally.

So, not entirely sure if this is a solution or a kludge.

Revision history for this message
Gary Brainin (brainin) wrote :

(Correction: previously it asked to decrypt sda3_crypt, not /dev/sda3.)

Revision history for this message
Patrick B (blue-noon) wrote :
Revision history for this message
Gary Brainin (brainin) wrote :

It seems like my problem may have been a different underlying cause with similar symptoms.

In my case, the entry in /etc/crypttab didn't match the location of the encrypted partition. Partition is on sda3, but /etc/crypttab read:

sda6_crypt UUID={uuid} none luks,discard

It appears that when the initramfs image is generated (as when a new kernel is installed, but also on other updates), a mismatch between the naming in /etc/crypttab and the actual partition causes an error (which of course I didn't see) such that the file in the initramfs (located at /cryptroot/crypttab) was empty (zero bytes). This lack of a file is what causes the boot process to hang.

Correcting /etc/crypttab to:

sda3_crypt UUID={uuid} none luks,discard

solves the problem. Of course, if your initramfs is already honked it needs to be regenerated:

sudo update-initramfs -u

But future updates should proceed normally. Thanks to everyone who provided pointers/thoughts that helped me track this down.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.