does not mount decrypted secondary luks crypttab partitions (btrfs)

[Stefan Fleiter]

Binary package hint: mountall

I was told to open a bug in bug 502665 comment 2.

In this setup I have three partitions decrypted by luks cryptsetup.
The passprhase for / is entered manually, for /var and /home
key files located in /etc are used to avoid having to enter the passphrase
three times during the boot process.

mountall does mount / successfully, but does not mount /var and /home
even after they are decrypted by cryptsetup.

After opening a shell (ctrl-alt-del is mapped to open a shell on tty2 here)
I mount the missing partitions with "mount -a -v" and then restart mountall
by "sudo service mountall stop" and "sudo service mountall start".

Then mountall does its job.

I will attach debug log of mountall (first and second run) and "mount shortly.

ProblemType: Bug
Architecture: amd64
Date: Sun Jan 17 19:30:38 2010
DistroRelease: Ubuntu 10.04
NonfreeKernelModules: nvidia
Package: mountall 2.4
ProcEnviron:
 LANGUAGE=
 PATH=(custom, user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-10.14-generic
SourcePackage: mountall
Tags: lucid
Uname: Linux 2.6.32-10-generic x86_64

[Stefan Fleiter (stefan-fleiter)]

In comment 2 the output is by "mountall --debug" not "mountall --verbose".

[Johan Kiviniemi (ion)]

Please boot the system and without running mount -a or restarting mountall. The file /dev/.udev.log should exist at that point. Please attach it here. Also run ‘udevadm info --export-db >udev-dump’ and attach udev-dump here.

[Johan Kiviniemi (ion)]

Btw, sulogin.conf from http://upstart.ubuntu.com/wiki/OMGBroken is a good way to get a shell when the system fails to boot properly.

[Stefan Fleiter (stefan-fleiter)]

I attached the needed files.
Reading them I still think this is related to bug 502665.

Any further help on this is very much appreciated.

[Johan Kiviniemi (ion)]

There’s obviously a problem somewhere. I’m not sure where. Let’s see what Scott has to say when he returns from his vacation.

mountall never receives an udev event that would end up as ‘try_udev_device: block /dev/mapper/home ...’ in its output.

But then, /dev/mapper/home does exist at some point, even though mountall udev hasn’t told that to mountall.

[Scott James Remnant (Canonical) (canonical-scott)]

On Mon, 2010-01-18 at 13:57 +0000, Johan Kiviniemi wrote:

> There’s obviously a problem somewhere. I’m not sure where. Let’s see
> what Scott has to say when he returns from his vacation.
>
> mountall never receives an udev event that would end up as
> ‘try_udev_device: block /dev/mapper/home ...’ in its output.
>
> But then, /dev/mapper/home does exist at some point, even though
> mountall udev hasn’t told that to mountall.
>
Does udev ever send the event?

Scott
--
Scott James Remnant
<email address hidden>

[Stefan Fleiter (stefan-fleiter)]

How do I find that out?
Boot never continues without manually mounting the missing devices.

If you tell me what to try I will gladly debug this.

[Daniel Hahler (blueyed)]

This is happening for me, as of today, too.
Since it worked until booting today, I guess it's related rather to cryptsetup for me.
But the symptoms are the same: it's waiting for my passphrase file mounted partitions forever. I can work around this by killing init Alt-SysRq-I, logging in as root and running "mountall": it then mounts everything correctly.

[Michael Filippov (michael-idisys)]

I tested today and it mounts encrypted partitions after upgrade (mountall=2.10).
Configuration is almost the same - encrypted root (LVM) and keys in /etc/crypt for other partitions (also LVM):
-- from /etc/fstab: --
/dev/mapper/data /mnt/data ext4 defaults 0 2
-- from /etc/cryttab: --
data /dev/mapper/nzxt-data /etc/crypt/nzxt-data-key luks

[Stefan Fleiter (stefan-fleiter)]

This is not fixed for me with mountall 2.10 and plymouth 0.8.1-4.

[Scott James Remnant (Canonical) (canonical-scott)]

Your udev log doesn't show any events corresponding to these filesystems, which implies that they have not been correctly created by cryptsetup

[Daniel Hahler (blueyed)]

JFI: it works for me since a while already again. I'm not the reporter - therefore I'll leave the bug open.

[Steve Langasek (vorlon)]

Stefan,

Please post your /etc/crypttab as well.

[Stefan Fleiter (stefan-fleiter)]

Here my /etc/crypttab. The problem persists with all updates applied.
If I you have any patches to help pin this down I will happily apply them.

[Steve Langasek (vorlon)]

Thanks. Can you attach /etc/default/cryptdisks too?

Also, booting with --verbose on the kernel commandline, and sending /var/log/boot.log, may provide some insight.

[Stefan Fleiter (stefan-fleiter)]

I had to enable bootlogd in /etc/default/bootlogd first, but now here is the file.

[Steve Langasek (vorlon)]

Are you sure your system is up to date? You shouldn't have to enable bootlogd to get this log, it should be created for you by plymouth via /etc/init/plymouth-log.conf.

Is /etc/init/cryptdisks-udev.conf present on your system? There's no mention in the log of this job being started by upstart, but it should have been.

[Stefan Fleiter (stefan-fleiter)]

Enabling of bootlogd is probably needed because /var is on a separate partition
I have to mount with "mount -a". Therefore no filesystem event is triggered for /var.

# ls -al /etc/init/cryptdisks-udev.conf
-rw-r--r-- 1 root root 435 2010-02-20 07:28 /etc/init/cryptdisks-udev.conf

# ls -al /lib/cryptsetup/cryptdisks.functions
-rw-r--r-- 1 root root 16457 2010-02-24 21:48 /lib/cryptsetup/cryptdisks.functions

cryptdisks-udev starts on "block-device-added" event.
What triggers that event?
It's not mentioned in /var/log/boot.log.

[Steve Langasek (vorlon)]

> Therefore no filesystem event is triggered for /var.

Well, as long as mountall is allowed to finish running, it should notice the filesystems have been mounted and emit the 'filesystem' event anyway.

> cryptdisks-udev starts on "block-device-added" event.
> What triggers that event?

Those events are emitted by upstart-udev-bridge. Is /etc/init/upstart-udev-bridge.conf present? What does 'status upstart-udev-bridge' show?

[Stefan Fleiter (stefan-fleiter)]

You are right.
When I do not stop mountall /var/log/boot.log gets written without enabling bootlogd in /etc/default/bootlogd.
A minor nit is that the time of the file is wrong, it's two hours behind.
Her the shell output 5 minutes after boot:

# ls -al /var/log/boot.log
-rw-r--r-- 1 root root 19029 2010-04-16 16:50 /var/log/boot.log
# date
Fr 16. Apr 18:55:15 CEST 2010

upstart-udev-bridge is running:

# ls -al /etc/init/upstart-udev-bridge.conf
-rw-r--r-- 1 root root 313 2009-09-15 15:15 /etc/init/upstart-udev-bridge.conf

# status upstart-udev-bridge
upstart-udev-bridge start/running, process 493

[Stefan Fleiter (stefan-fleiter)]

Even after reinstalling every single package as described in bug 563260 comment 3 the problem persists.

[Stefan Fleiter (stefan-fleiter)]

It's clearly difficult to debug this when the log file contains only part of the truth.
Or how can it be that upstart-udev-bridge runs without /var/log/boot.log mentioning it?

[Steve Langasek (vorlon)]

Yes, that does make it hard to debug. I seem to have the same problem locally - the udev job is certainly running, but doesn't show up in /var/log/boot.log.

Sorry, I seem to be at a dead end for the moment, given that this seems not to be reproducible for most users (including me) and we're not getting useful debugging information.

[Stefan Fleiter (stefan-fleiter)]

Thanks a lot for your support until now.

But is there nothing I can do to pin this down?
No debug printfs or something like that?
Stracing a process?
Nothing?
I do not understand the plumbing layer enough to do this on my own.
So I would need some assistance...

And shouldn't a bug be opened regarding the missing lines in /var/log/boot.log?
I assume it is an upstart bug.
Is that correct?

[Steve Langasek (vorlon)]

Stefan,

Please check whether this problem goes away after installing dmsetup 2.02.54-1ubuntu4. There's a good chance this is a duplicate of bug #561390.

[Stefan Fleiter (stefan-fleiter)]

There is no new dmsetup version.
dmsetup has version 2:1.02.39-1ubuntu4, even if it was built from lvm2 2.02.54-1ubuntu4.

That's at least what "apt-cache policy dmsetup" and
https://launchpad.net/ubuntu/lucid/amd64/dmsetup
tell me.

[Steve Langasek (vorlon)]

sorry, you're right, the source and binary versions don't match.

[Steve Langasek (vorlon)]

So does installing dmsetup 2:1.02.39-1ubuntu4 fix the problem? This version wasn't yet available at the time of the earlier tests you reported.

[Stefan Fleiter (stefan-fleiter)]

Oh, sorry you are right, forgot to mention this.
The update did not fix the bug here.

I usually install the available updates every day
and will report here when the problem is gone.
But please do not hesitate to ask again if there
might be a fix.

I will attach new log files soon since many packages
where updated since last time.

[Steve Langasek (vorlon)]

Your latest mountall log shows that only /var fails to mount, and /home mounts successfully, so that seems like progress. I notice that we don't have a copy of your fstab in this bug log, can you attach that as well?

[Stefan Fleiter (stefan-fleiter)]

Here my /etc/fstab as used during the last tests.

[Stefan Fleiter (stefan-fleiter)]

I think I understand the problem better now.
When I found the log line
    try_udev_device: ignored /dev/mapper/var (not yet ready?)
I looked in mountall.c

»·······if ((! action)
»······· || (! strncmp (kernel, "dm-", 3))
»······· || (! strncmp (kernel, "md", 2))
»······· || (! strncmp (kernel, "loop", 4))
»······· || (! strncmp (kernel, "ram", 3))) {
»·······»·······if ((! usage) && (! type) && (! uuid) && (! label)) {
»·······»·······»·······if (action)
»·······»·······»·······»·······nih_debug ("ignored %s (not yet ready?)", devname);
»·······»·······»·······return;
»·······»·······}
»·······}

When I then looked in udev-dump.log the problem there became something clearer immediately.
The ID_FS_* attributes are missing for /dev/mapper/var.
The important question which remains is why they are missing. :-)

[Steve Langasek (vorlon)]

Aha - so this points to a bug either in udev, or in util-linux.

Can you please run 'sudo blkid /dev/mapper/var' and post the output?

[Scott James Remnant (Canonical) (canonical-scott)]

That should be:

 sudo blkid -p /dev/mapper/var

The -p is important

[Stefan Fleiter (stefan-fleiter)]

# blkid -p /dev/mapper/var
/dev/mapper/var: ambivalent result (probably more filesystems on the device, use wipefs(8) to see more details)

# wipefs -n /dev/mapper/var
offset type
----------------------------------------------------------------
0x10040 btrfs [filesystem]
                     UUID: a1124975-72fc-445c-a034-2c07ed4e89e6

0x438 ext2 [filesystem]
                     UUID: 260b7189-c063-42af-85a3-0a6bd9cb3d04

I suppose using wipefs to erase the ext2 filesystem signature will
destroy my /var partition, doesn't it?
What to do now?

[Scott James Remnant (Canonical) (canonical-scott)]

Thanks Stefan.

Ubuntu will never mount a filesystem with multiple signatures, since it cannot know which is correct and getting it wrong could badly corrupt the filesystem. You'll need to use wipefs to remove the signature that is incorrect.

I'd recommend backing up the device first, you probably only need the first megabyte at most, try:

  # dd if=/dev/mapper/var of=var.bak bs=1M count=1

Then

  # wipefs -o 0x438

After that, blkid -p /dev/mapper/var should only return the btrfs data and the drive should be mounted

[Stefan Fleiter (stefan-fleiter)]

I could "repair" my system and it boots now without interaction.

But IMHO there is still one bug left (probably in mountall):
If blkid does find more than filesystem signature, either an error should be printed or
the fs type from /etc/fstab should be used if it matches one finding of blkid.

Many linux users will not be able to fix such a problem themselves, even less
will be able to diagnose it, so the boot process should try the best...

[Scott James Remnant (Canonical) (canonical-scott)]

On Fri, 2010-04-30 at 17:34 +0000, Stefan Fleiter wrote:

> I could "repair" my system and it boots now without interaction.
>
> But IMHO there is still one bug left (probably in mountall):
> If blkid does find more than filesystem signature, either an error should be printed or
> the fs type from /etc/fstab should be used if it matches one finding of blkid.
>
> Many linux users will not be able to fix such a problem themselves, even less
> will be able to diagnose it, so the boot process should try the best...
>
Most Linux users will never encounter this kind of problem; I'm actually
surprised that there isn't an error in /var/log/boot.log - it seems like
the kind of thing that mount would output (which comes from the same
package as util-linux)

In terms of fstab not overriding the data, the problem is that that's
frequently wrong - all the time there are two signatures, we can't just
pick one, no matter how hard forced.

Scott
--
Scott James Remnant
<email address hidden>

[Stefan Fleiter (stefan-fleiter)]

As I understand the code mountall(8) did never spawn mount(8) because it did
not find the ID_FS_{TYPE,USAGE} attributes.
So mount only got called interactively by me, fetched the fs_vfstype from
fstab(5) and did what it was told to do, it called mount(2) which did not care
for the ext2 signature.
mount(8) does not search for fs signatures if the type is given.

Why can mountall not do the same?

If mountall would notice the ID_FS_AMBIVALENT attribute it could at least write
a warning message before spawning the recovery shell.

According to
https://bugzilla.redhat.com/show_bug.cgi?id=473514
cryptsetup and mkswap zap all existing filesystem signatures nowadays, but
existing filesystems could still have multiple signatures.

[Stefan Fleiter (stefan-fleiter)]

Before I forget this completely:
Many thanks for your wonderful support to all of you, especially Steve and Scott.
This is why and how Ubuntu makes a difference.