Boot fails with encyprted BTRFS RAID1 array - open ctree failed

[Ryan S Kingsbury]

I have recently installed Linux Mint 17.1 into a system with three drives - 1 SSD containing the / partition (btrfs format), and 2 identical HDD's containing /home in an encrypted BTRFS RAID1 array. The system will not boot without manual intervention, using either the recommended 3.13.0-37 kernel or the most up to date 3.16.0-31 kernel.

At about the same time that I am prompted to enter my password to unlock the drive, I am prompted that a mount has failed. If I press M to enter manual recovery I see the emergency shell. Then all I have to do is exit (Ctrl+D) to continue booting and everything works normally.

It appears that a similar (maybe the same) bug was reported on ArchWiki:
https://wiki.archlinux.org/index.php/Btrfs#BTRFS:_open_ctree_failed

I also experienced this bug on OpenSUSE 13.2 and filed a bug report there:
https://bugzilla.opensuse.org/show_bug.cgi?id=912170

On a possibly related note, I am getting TWO password prompts even though both encrypted volumes use the same password. On OpenSUSE only one password was required. My understanding is that as soon as you suppy one password it is supposed to be tried on each volume in /etc/crypttab.

I will now try the fix that worked for me on OpenSUSE (see bug report)

copy /usr/lib/udev/rules.d/64-btrfs.rules into /etc/udev/rules.d/64-btrfs.rules and replace the following two lines

ENV{DM_NAME}=="", IMPORT{builtin}="btrfs ready $devnode"
ENV{DM_NAME}=="?*", IMPORT{builtin}="btrfs ready /dev/mapper/$env{DM_NAME}"

with single one

IMPORT{builtin}="btrfs ready $devnode"

and report back what happens.

Some system information:
=====================

relevant excerpts from /etc/fstab:

# SSD containing the main system
# / was on /dev/sda3 during installation
UUID=00a0b482-a861-400f-b29c-b003cbc697f9 / btrfs defaults,autodefrag,discard,space_cache 0 0
# /boot/efi was on /dev/sda2 during installation
UUID=26CC-DC02 /boot/efi vfat defaults 0 1

# encrypted RAID1 btrfs array containing /home
# the btrfs devid's are 2 and 3
UUID=e91f611f-524a-43f5-bde5-8ebb9672f146 /home btrfs defaults,discard,autodefrag,space_cache 0 0

/etc/crypttab:
encrypted-home-sdb UUID=9c8fb7d0-74e2-4e38-b7c7-6211bbb6d2b1 none luks, retry=1
encrypted-home-sdc UUID=1ca9d3ba-c409-4127-91f5-e3d9c21242bd none luks, retry=1

relevant excerpts from dmesg:

[ 9.487085] BTRFS: device fsid e91f611f-524a-43f5-bde5-8ebb9672f146 devid 3 transid 81360 /dev/dm-0
[ 9.497773] BTRFS info (device dm-0): turning on discard
[ 9.497776] BTRFS info (device dm-0): enabling auto defrag
[ 9.497777] BTRFS info (device dm-0): disk space caching is enabled
[ 9.498565] BTRFS: failed to read the system array on dm-0
[ 9.522859] BTRFS: open_ctree failed
[ 10.246821] init: mountall main process (322) terminated with status 2
[ 10.275253] init: plymouth-upstart-bridge main process ended, respawning
[ 11.110362] BTRFS info (device dm-0): turning on discard
[ 11.110365] BTRFS info (device dm-0): enabling auto defrag
[ 11.110366] BTRFS info (device dm-0): disk space caching is enabled
[ 11.113918] BTRFS: failed to read the system array on dm-0
[ 11.129428] BTRFS: open_ctree failed
[ 11.714810] BTRFS: device fsid e91f611f-524a-43f5-bde5-8ebb9672f146 devid 2 transid 81360 /dev/dm-1
[ 12.628797] init: mountall main process (1033) terminated with status 2
[ 13.088968] BTRFS info (device dm-1): turning on discard
[ 13.088971] BTRFS info (device dm-1): enabling auto defrag
[ 13.088973] BTRFS info (device dm-1): disk space caching is enabled
[ 13.145755] BTRFS: bdev /dev/mapper/encrypted-home-sdb errs: wr 0, rd 79, flush 0, corrupt 0, gen 0
[ 16.111916] init: failsafe main process (1436) killed by TERM signal

[Ryan S Kingsbury (ryanskingsbury)]

OK, the workaround from OpenSUSE did NOT work (with the 3.16 kernel). Same behavior as before. The steps I performed were:

copy /lib/udev.rules.d/64-btrfs.rules to /etc/udev/64-btrfs.rules

The line
IMPORT{builtin}="btrfs ready $devnode"

was already present in the original rules file, so I did not have to replace anything as I did in OpenSUSE.

[Ryan S Kingsbury (ryanskingsbury)]

Today's update to the 3.13.46-77 kernel did not change the behavior.

[Ryan S Kingsbury (ryanskingsbury)]

Here is the dmesg output at the time of the error message:

[ 8.515396] BTRFS: device fsid e91f611f-524a-43f5-bde5-8ebb9672f146 devid 2 transid 100593 /dev/dm-0
[ 8.525319] BTRFS info (device dm-0): enabling auto defrag
[ 8.525322] BTRFS info (device dm-0): disk space caching is enabled
[ 8.526281] BTRFS: failed to read the system array on dm-0
[ 8.547852] BTRFS: open_ctree failed
[ 10.106655] BTRFS: device fsid e91f611f-524a-43f5-bde5-8ebb9672f146 devid 3 transid 100593 /dev/dm-1
[ 10.943981] init: mountall main process (334) terminated with status 2
[ 10.972131] init: plymouth-upstart-bridge main process ended, respawning
[ 10.976545] init: plymouth-upstart-bridge main process (1253) terminated with status 1
[ 10.976550] init: plymouth-upstart-bridge main process ended, respawning
[ 19.846926] BTRFS info (device dm-1): enabling auto defrag
[ 19.846929] BTRFS info (device dm-1): disk space caching is enabled
[ 19.901994] BTRFS: bdev /dev/dm-0 errs: wr 0, rd 79, flush 0, corrupt 0, gen 0
[ 23.125160] init: failsafe main process (1465) killed by TERM signal

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.