race condition between encrypted device creation and mountall probing with random-encrypted devices (swap, tmp)

Bug #475936 reported by Mike Buckley
188
This bug affects 48 people
Affects Status Importance Assigned to Milestone
cryptsetup (Ubuntu)
Fix Released
High
Unassigned
Karmic
Fix Released
High
Steve Langasek
Lucid
Fix Released
High
Unassigned

Bug Description

Binary package hint: mountall

I am using Ubuntu 9.10 with the latest packages applied (apt-get uprade)
Version of mountall: 1.0
Expected results: /dev/mapper/tmp should be mounted on /tmp
Unexpected results: /dev/mapper/tmp is not mounted on /tmp

In my /etc/crypttab I have:
tmp /dev/sda2 /dev/urandom tmp
swap /dev/sda3 /dev/urandom swap

In my /etc/fstab I have (among other lines), this:
/dev/mapper/tmp /tmp ext2 relatime 0 2

Most of the time the system boots OK, but /tmp never gets mounted.
This appears to be an issue with mountall because if I run mountall from the command line, then /tmp does get mounted.
When the /dev/mapper/tmp device is created (part of the cryptsetup process), the mkfs -t ext2 command is run on /dev/mapper/tmp. It appears mountall is not waiting for this to complete. Should there be a timeout for a dynamically created tmp to be available before it is mounted?

TEST CASE:
1. select a spare partition on your disk, referred to below as /dev/sdf12
2. install the cryptsetup package from karmic.
3. configure it for use with cryptsetup as a device with a random key by adding this line to /etc/crypttab:
    /dev/mapper/crypttmp /dev/sdf12 /dev/urandom tmp
4. configure the tmp partition to be auto-enabled by adding this line to /etc/fstab:
    /dev/mapper/crypttmp /tmp ext2 defaults 0 2
5. ensure the device contains no other filesystem signatures by blanking it with 'dd if=/dev/zero of=/dev/sdf12 bs=$((1024*1024))'
5. reboot
6. verify that the tmp partition is not successfully mounted: mountall should refuse to continue booting the system because the device is unavailable.
7. boot into the rescue shell to comment out the tmp line from /etc/fstab, then reboot
8. upgrade to the karmic-proposed version of cryptsetup
9. re-enable the crypttmp line in /etc/fstab
10. reboot
11. verify that the tmp partition has been successfully mount, by checking the output of 'mount | grep /tmp'

REGRESSION POTENTIAL:
In order to prevent mountall from seeing the block device before it's been formatted / mkswap'ed for use, cryptsetup must create the device under a different name initially, format, and then rename to the public device name. For sanity's sake, the proposed patch does this for /all/ cryptsetup devices, not just those configured for tmp or swap; and there is a small but finite risk that someone will already have a /dev/mapper/${name}_unformatted node on their system that conflicts with one of these devices, causing this device to fail to be set up at boot time due to the name collision.

The '${name}_unformatted' temp name was chosen to minimize this risk; I believe the risk is acceptably low for an SRU.

affects: mountall (Ubuntu) → cryptsetup (Ubuntu)
Revision history for this message
all-by-my-self (all-by-my-self) wrote :
Download full text (3.9 KiB)

Hi out there!

I had the same problem. On jaunty no problems but on karmic (complete updated on 4th Dec).

The new canonical-special init-system isn't perfect.

This routes to the unmountable (non-luks-) cryptsetup-devices:

The new init-system don't use the old /etc/inittab. That file is replaced by the *.conf-files in /etc/init/.

The *.conf-files uses a syntax for starting themselfs by events (see man startup, starting, started, stopping and stopped).

A normal init starts at /etc/init/mountall.conf (and maybe some others). If started the script starts /etc/init/udev.conf. These scripts start other scripts. And so on.

The cryptsetup-devices are starting after mountall.conf and udev.conf from /etc/init/cryptdisks-enable.conf (including /lib/cryptsetup/cryptdisks.functions). This script read the /etc/crypttab and /etc/default/cryptdisks to create or open the crypted devices.

The mountall.conf runs the new command "mountall" (binary!) that reads the /etc/fstab and tries to mount all devices described therein (exclude the ones with "noauto" option).

Mounting a closed or not created crypt-device like /dev/mapper/crypttmp isn't possible! This is the jumping point! It is necessary to open or create the crypt-devices BEFORE mounting them.

root-Luks-cryptsetup-devices with passphrase-input over keyboard seems to open automatically before running the *.conf-files.

This is my workaround:

--------

1) Basics (with and without an external key-file)

 - log in as root (on console with "sudo -s")
 - open /etc/init/cryptdisks-enable.conf in an editor
 - replace "start on stopped udevtrigger" with "start on startup"
 - save the script
 - open /etc/init/mountall.conf
 - replace "start on startup" with "start on stopping cryptdisks-enable"
 - save the script

--------

2) Additionals (just for using a key-file on an external device)

 - determine the external device (USB-stick) (e.g. /dev/sdb1)
 - for rookies only: replace sdXY in the following codes with the determined indication of your external device
 - open (still as root) /etc/fstab
 - add "/dev/sdXY /mnt ext2 noauto,ro,dev,nouser,async 0 0"
 - for rookies only: instead of /mnt you MUST choose the same directory you use for the key-file-path in /etc/crypttab
 - for rookies only: instead of ext2 you MUST choose the filesystem of your external device /dev/sdXY
 - for all: you can use other options, but MUST use "noauto", because mountall.conf will try to mount it if that option isn't set
 - save fstab
 - open /etc/default/cryptdisks in an editor
 - replace 'CRYPTDISKS_MOUNT=""' with 'CRYPTDISKS_MOUNT="/dev/sdXY"'
 - save the script

so far so good? It works, but...

... there is still a little problem left!

/etc/init/cryptdisks-enable.conf mounts /dev/sdXY but couldn't close it after using. I think, because udevd isn't running at the point of mounting /dev/sdXY. The script udev.conf is started after the cryptdisks-enable.conf stopped. So the mount isn't written in /etc/mtab or given to /proc/mounts.

fact: after booting the system, /dev/sdXY is still mounted on /mnt - but you cannot get any system-output for this! Nor "df" nor "cat /etc/mtab" nor "cat /proc/mounts" tells you something about it!

...

Read more...

description: updated
Revision history for this message
Jannis Menn (jannis-menn) wrote :

Thanks a lot for the workaround, I had the same problem here, now it works.
It also fixed a problem at bootup, where I sometimes got the message "/dev/mapper/crypttemp already mounted or /tmp busy", which resulted in mountall to fail, dropped into a root-shell and I needed to press CRTL-D to retry.
I think this bug may affect a lot of people who believe that /tmp is encrypted while it actually isn't.

Revision history for this message
Steve Langasek (vorlon) wrote :

The workaround proposed here will not work in the archive at large. I believe the correct fix is to signal mountall to rescan at the end of the do_swap() and do_tmp() functions in /lib/cryptsetup/cryptdisks.functions. Could someone test whether this fixes the issue?:

    PID=$(status mountall 2>/dev/null | sed -e '/,/{s/.*,[^0-9]*//;q};d')
    [ -n "$PID" ] && kill -USR1 $PID || true

If it doesn't, please also add the following command to the top of the "script" block in /etc/init/cryptdisks-enable.conf:

    initctl log-priority debug

and attach the resulting /var/log/syslog file to this bug.

Changed in cryptsetup (Ubuntu):
importance: Undecided → High
summary: - /dev/mapper/tmp encrypted with /dev/urandom not mounted on bootup
+ race condition between encrypted device creation and mountall probing
+ with random-encrypted devices (swap, tmp)
Revision history for this message
Steve Langasek (vorlon) wrote :

I've checked the mountall code; the SIGUSR1 handler only rescans for remote mount points. So no need for anyone to provide the debugging information requested in the previous comment.

I think I've sussed out how to do this correctly using 'dmsetup rename'; will have a go at the code and will hopefully have this uploaded to lucid soon.

Changed in cryptsetup (Ubuntu):
status: New → In Progress
Steve Langasek (vorlon)
Changed in cryptsetup (Ubuntu Karmic):
assignee: nobody → Steve Langasek (vorlon)
importance: Undecided → High
milestone: none → karmic-updates
status: New → Triaged
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cryptsetup - 2:1.1.0~rc2-1ubuntu4

---------------
cryptsetup (2:1.1.0~rc2-1ubuntu4) lucid; urgency=low

  [ Steve Langasek ]
  * debian/initramfs/cryptroot-script: if plymouth is present in the
    initramfs, use this directly, bypassing the cryptsetup askpass script;
    but keep support for these other frontends around on a transitional
    basis.
  * debian/cryptdisks.functions:
    - change interaction to use plymouth directly if present, and if not, to
      fall back to /lib/cryptsetup/askpass as before
    - wrap the call to /lib/cryptsetup/askpass with watershed, to make sure
      we only ever have one of these running at a time; otherwise multiple
      invocations could steal each other's input and/or write over each
      other's output
    - new function, crypttab_start_one_disk, to look for the named source
      device in /etc/crypttab (by device name, UUID, or label) and start it
      if configured to do so
  * debian/cryptdisks-enable.upstart: run the upstart job once for each block
    device, using the new crypttab_start_one_disk function, triggered by udev;
    this doesn't eliminate the possibility of a race with gdm when the
    decrypted volume isn't a 'bootwait' mount point (since gdm kills
    plymouth), but it does eliminate the race between udev and cryptsetup.
    LP: #454898.
  * debian/cryptdisks-enable.upstart: check that the package is installed
    and exit gracefully if it's not. LP: #435814
  * debian/cryptdisk.functions: initially create the device under a temporary
    name and rename it only at the end using 'dmsetup rename', to ensure that
    upstart/mountall doesn't see our device before it's ready to go.
    LP: #475936.

  [ Colin Watson ]
  * Add ext4 support to passdev.
 -- Steve Langasek <email address hidden> Tue, 15 Dec 2009 18:05:45 -0800

Changed in cryptsetup (Ubuntu):
status: In Progress → Fix Released
Steve Langasek (vorlon)
description: updated
Steve Langasek (vorlon)
description: updated
tags: added: regression-release
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Testcase warning : I believe the testcase in wrong. In 2 Karmic systems I have, encrypted swap usually works, but encrypted /tmp never works.

Bugfix testcase should be done against encrypted /tmp

Revision history for this message
Steve Langasek (vorlon) wrote :

True, a mkfs is generally going to take longer than an mkswap and is more likely to hit the race. Thanks, updating the testcase.

description: updated
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Other testcase comments : Steven you've left "swap" instead of "tmp" at some place ;-)

Also, the system _will_ boot successfully but with a short (almost invisible on a fast system) warning message that some fstab-defined partitions couldn't be mounted _if_ the fstab entry for /tmp has "0" for fsck pass.

Then, the system boots but has /tmp unmounted so actually /tmp data goes into the rootfs, unencrypted (and that makes a security issue).

According to my experience, the system will fail booting only if /tmp has a non-zero number for fsck pass in fstab ; then you'll get the failure (for fsck fails) and "Type root password or Ctrl-D to continue booting" message.

But it doesn't sound so logical to perform a fsck at boot on a filesystem that's just been created from scratch and will be everytime, so there are systems (mine) where the fsck pass for /tmp is set to zero, causing the system to boot with no encrypted /tmp at all, which is not very visible...

Steve Langasek (vorlon)
description: updated
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Steve, I'd like to test the fix, but can't currently get the fixed package from the "karmic-proposed" repositories. Could you please give me a download link, or would you mind emailing it to me at : s w a m i (at) p e t a r a m e s h (dot) o r g ?

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Hi again,

I've installed cryptsetup 2:1.1.0~rc2-1ubuntu5 (dated today, gotten from archive.ubuntu.com pool), which necessitated as dependancy that I also install libpopt0 from the same source.

After that, I rebooted my Karmic machine, but still, my encrypted /tmp is NOT mounted after reboot.

I can see that the encrypted tmp device has been properly prepared and formatted, yet it is not mounted.

So this "fix" doesn't fix my issue.

Revision history for this message
Steve Langasek (vorlon) wrote :

Swâmi,

Just confirmed that using the test case in lucid, the /tmp directory fails to be mounted. However, this is because mountall treats the /tmp directory specially. Varying the /etc/fstab line in either of two ways is enough to fix this:

 - set the 'pass' option to non-zero
 - mount it somewhere other than /tmp

After making either change, mountall successfully mounts the device for me.

I've updated the test case using the first of these options.

description: updated
Revision history for this message
Steve Langasek (vorlon) wrote :

(If you believe the mountall behavior is wrong, please file a separate bug report on the mountall package for this.)

description: updated
Steve Langasek (vorlon)
description: updated
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Hi Steve,

I'll give it a shot tonight with a non-zero "pass" entry for /tmp. However per your description I feel that the mountall behaviour is definitely wrong, and I'm not sure whether another bug report should be filed, or if this precise bugreport should be reaffected to "mountall", as this is the precise problem that this bug describes.

I have rethought about this, and as long as cryptsetup does its job (which is to create and format the temporary encrypted fs), the race condition cannot be considered a cryptsetup bug, the problem comes from the fact the things are'nt called in the right order or mountall doesn't wait for the things it triggers to be finished (filesystems to be ready) before going on.

I had initially filed a separate bug report (#493480) about this, againts upstart, bug this bug report was marked as a dup of the present one, so that's why I'm here, but I never considered this race condition to be something that would to possible be solve in cryptetup itself and alone - which I state in my initial bugreport.

I feel that, as the current bugreport has already some discussions and details about the issue, it would probably make more sense to reaffect this bugreport to mountall rather than have to create yet another one (that might end being marked again as a duplicate of this one).

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Hi,

Here, even with a "non-zero fsck pass for /tmp", it doesn't mount either.

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 475936] Re: race condition between encrypted device creation and mountall probing with random-encrypted devices (swap, tmp)

On Thu, Dec 17, 2009 at 06:43:40PM -0000, Swâmi Petaramesh wrote:
> Here, even with a "non-zero fsck pass for /tmp", it doesn't mount
> either.

Please edit /etc/init/mountall.conf, appending '-v > /dev/.mountall.log
2>&1' to the invocation of mountall, reboot, and attach the resulting file
to this bug report.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

I'm on travel and wont' be able to perform this test until next week.

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted cryptsetup into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in cryptsetup (Ubuntu Karmic):
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Testing karmic-proposed cryptsetup (2:1.0.6+20090405.svn49-1ubuntu7.1) : Still doesn't solve the issue with encrypted /tmp not mounting at boot. (Encrypted swap with random key and encrypted /home with fixed key-file do work, as previously).

So that's a "proposed fix doesn't change or improve anything here".

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

I ran mountall with logging (per Steve's request on comment #15) by calling mountall (in mountall.conf) with :

exec mountall --daemon -v $force_fsck $fsck_fix $tmptime > /dev/.mountall.log

The matter is that 1st time I rebooted with that I ended up with... encrypted /tmp properly mounted !

Then I rebooted and ended up with... /tmp not mounted (as usual) !

Then I rebooted a 3rd time and ended up with... /tmp mounted again !

As it never worked before and with logging turned on it works... or not... or yes... I assume that's because logging slows down mountall a bit, leaving more time for cryptsetup/mke2fs to hopefully finish their job in time...?

Anyway I attach 2 logs :
- .mountall-1.log : 1st time, ending with /tmp properly mounted
- .mountall-2.log : 2nd time, /tmp didn't mount

HTH...

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Re-opened my bug #493480, de-duplicated from the present one and affected to the "mountall" package per Steve suggestion in comment 12.

Revision history for this message
Steve Langasek (vorlon) wrote :

Marking this verification-failed based on further analysis in bug #493480. We now have a *different* race condition between mountall and cryptsetup, but we still have a race condition.

tags: added: verification-failed
removed: verification-needed
Steve Langasek (vorlon)
Changed in cryptsetup (Ubuntu Karmic):
status: Fix Committed → In Progress
Changed in cryptsetup (Ubuntu Lucid):
status: Fix Released → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cryptsetup - 2:1.1.0~rc2-1ubuntu8

---------------
cryptsetup (2:1.1.0~rc2-1ubuntu8) lucid; urgency=low

  * cryptdisks.functions: do_tmp should mount under /var/run/cryptsetup for
    changing the permissions of the filesystem root, not directly on /tmp,
    since mounting on /tmp a) is racy, b) confuses mountall something fierce.
    LP: #475936.
 -- Steve Langasek <email address hidden> Tue, 22 Dec 2009 20:24:28 +0000

Changed in cryptsetup (Ubuntu Lucid):
status: In Progress → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote :

cryptsetup 2:1.0.6+20090405.svn49-1ubuntu7.2 has been reuploaded to karmic-proposed, with a more complete fix for this issue (the one just uploaded to lucid). Awaiting SRU team processing.

tags: removed: verification-failed
Revision history for this message
Martin Pitt (pitti) wrote :

Accepted cryptsetup into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in cryptsetup (Ubuntu Karmic):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Hi,

I've tested "Version: 2:1.0.6+20090405.svn49-1ubuntu7.2" from karmic-proposed, and I'm not sure whether the issue is completely fixed or not :

At first reboot after having installed the package, I found myself in the following situation :
System displayed :

 * Starting init crypto disks...
* c_swap (starting)
* c_swap (started)...
* c_tmp (starting)
* c_tmp (started)...
* c_home (running)... [OK]
* Stopping NTP server ntpd
* Starting NTP server ntpd
* Stopping NTP server ntpd
* Starting NTP server ntpd

(Now the system waits here "forever", network is down, doesn't ping, keyboard unresponsive)

Pressing [Ctrl]-[Alt]-[Del], I got :

init: network-interface (lo) pres-start process (1022) killed by TERM signal
init: usplash main process (1696) killed by TERM signal
mountall: Job failed to start
/tmp: waiting for /dev/mapper/c_tmp
/home: waiting for /dev/mapper_c_home
swap: waiting for /dev/mapper/c_swap
/proc/bus/usb: waiting for none
 * Deconfiguring network interfaces

(Now the system waits here "forever", ignoring any keyboard action including [CTRL]-[Alt]-[Del]. Got scared of having an unbootable machine...)

=> I rebooted using MagicSysRq

=> Then the system booted fine, at the end all encrypted volumes are mounted, including /tmp

=> Then I rebooted again, and got again a good boot process ending with all encrypted devices properly mounted

So I wonder...

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 475936] Re: race condition between encrypted device creation and mountall probing with random-encrypted devices (swap, tmp)

On Wed, Dec 23, 2009 at 08:25:53PM -0000, Swâmi Petaramesh wrote:
> Pressing [Ctrl]-[Alt]-[Del], I got :

> init: network-interface (lo) pres-start process (1022) killed by TERM signal

I'm afraid that's probably bug #497299, unrelated to cryptsetup.

I would say cryptsetup is working for you, but of course this problem is
inherently racy, so we would appreciate feedback about whether this
continues to work for you across multiple reboots (discounting the upstart
regression mentioned above).

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Geeez, so many init bugs ! ;-)

Well, my system survived 2 more reboots with encrypted /tmp properly mounted each time, so as far as I'm concerned the cryptsetup fix is good :-)

Thanks for all your time and efforts Steve :-))

There's little chance I reboot it much more in the coming days : being a server, I avoid rebooting it unless necessary and I will be on travel in the coming week...

Well, I should do a couple other bug reports about other topics, but ain't got time tonight ;-))

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

Rebooted once more, still get correctly encrypted /tmp, so I confirm the fix "works for me".

Steve Langasek (vorlon)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

I've just noticed while doing something else ;-) with my system fully booted for more than 24 hrs, that "ls -l /dev/mapper" was showing me, among other things :

brw-rw---- 1 root disk 252, 16 2009-12-24 08:54 c_swap
brw-rw---- 1 root disk 252, 16 2009-12-24 08:54 c_swap_unformatted

I assume the 2nd one was not supposed to stay...?

Revision history for this message
Steve Langasek (vorlon) wrote :

Indeed, it was not supposed to stay. In my own testing, it did not. I don't think this should block publishing of the SRU, but could you file a new bug report about this (lower-priority) issue so we keep it on the radar?

Revision history for this message
Swâmi Petaramesh (swami-petaramesh) wrote :

I'll file a new bug report on this topic when I have a chance to be in front of the machine and reboot it again to confirm this. Probably next week. I feel this minor issue shouldn't prevent from committing the current fix anyway.

Revision history for this message
David Bartley (dtbartle-csclub) wrote :

I'm having a problem with this as well. With Steve's fix, mkdir -p /var/run/cryptsetup/... fails because / is mounted read-only at the time. This then prevents cryptsetup from setting the correct mode (1777). If I change /var/run/cryptsetup/... to /mnt in cryptdisk.functions everything works fine.

Revision history for this message
Steve Langasek (vorlon) wrote :

David,

This shouldn't be possible. the cryptdisks upstart jobs won't run until udev is started, and udev won't start until the "virtual-filesystems" are mounted, and /var/run is configured as a virtual filesystem (a tmpfs) standard in Ubuntu.

Do you have some other filesystem configured to mount as /var/run, via /etc/fstab?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cryptsetup - 2:1.0.6+20090405.svn49-1ubuntu7.2

---------------
cryptsetup (2:1.0.6+20090405.svn49-1ubuntu7.2) karmic-proposed; urgency=low

  * Depend on watershed.
  * cryptdisks.functions: do_tmp should mount under /var/run/cryptsetup for
    changing the permissions of the filesystem root, not directly on /tmp,
    since mounting on /tmp a) is racy, b) confuses mountall something fierce.
    LP: #475936.

cryptsetup (2:1.0.6+20090405.svn49-1ubuntu7.1) karmic-proposed; urgency=low

  * debian/cryptdisks.functions:
    - wrap the call to /lib/cryptsetup/askpass with watershed, to make sure
      we only ever have one of these running at a time; otherwise multiple
      invocations could steal each other's input and/or write over each
      other's output
    - new function, crypttab_start_one_disk, to look for the named source
      device in /etc/crypttab (by device name, UUID, or label) and start it
      if configured to do so
  * debian/cryptdisks-udev.upstart: new, additional upstart job run once for
    each block device, using the new crypttab_start_one_disk function,
    triggered by udev; this doesn't eliminate the possibility of a race with
    gdm when the decrypted volume isn't a 'bootwait' mount point (since gdm
    kills usplash), but it does eliminate the race between udev and
    cryptsetup. LP: #454898.
    The other cryptdisks-enable job is still needed as well, to give us the
    second pass needed to cover devices that are decrypted using keys stored
    on other encrypted disks. LP: #443980.
  * debian/cryptdisk.functions: initially create the device under a temporary
    name and rename it only at the end using 'dmsetup rename', to ensure that
    upstart/mountall doesn't see our device before it's ready to go.
    LP: #475936.
  * Make the 'start' action of the init script a no-op, this should be
    handled entirely by the upstart job now; ad remove any symlinks from
    /etc/rcS.d on upgrade. LP: #473615.
 -- Steve Langasek <email address hidden> Tue, 22 Dec 2009 23:29:32 +0000

Changed in cryptsetup (Ubuntu Karmic):
status: Fix Committed → Fix Released
Revision history for this message
Ken Miller (kenny-miller) wrote :

I'd like to comment on this fix (e.g. cryptsetup - 2:1.0.6+20090405.svn49-1ubuntu7.2 w/watershed 4 and mountall 1.0):

I've been running with encrypted root/swap for a month, but then had hardware problems forcing me to replace the system disk. All was well for nearly a week 5+ days and 6+ reboots, but now I'm seeing on boot usually (but I can't say every time) the recovery shell message:

One or more of the mounts listed in /etc/fstab cannot be mounted:
/: waiting for /dev/mapper/xtremepapa-root
/tmp: waiting for (null)
/boot: waiting for /dev/disk/by-uid/blahblahblah
Press ESC to enter into a recovery shell

The first curious thing above is /tmp is not a separate file system and thus not explicitly called out.

Secondly, when ESC to recovery, / is mounted (read-only), and a mount -av mounts /boot with no issues. If I ctrl-D I go back to the shell and see:

One or more of the mounts listed in /etc/fstab cannot be mounted:
/: waiting for /dev/mapper/xtremepapa-root
/tmp: waiting for (null)
Press ESC to enter into a recovery shell

Thirdly, when ESC to recovery, mount -o remount,rw / and ctrl-d again, I proceed to boot with no further errors.

This feels like a race-condition, and seems new to recent updates, though I guess it could be new simply because my new disk has gotten progressively slower in the last 6 days and am thus more susceptable to a race-condition, but from the bug responses to cryptsetup the last few days, it appears I'm not alone.

Not knowing the true cause, I'm not sure if I should submit this as a new bug, or if adding this information to this one which appears quite similar should suffice.

Thanks & Regards,

== k ==

Revision history for this message
Steve Langasek (vorlon) wrote :

Ken,

This error message probably indicates that the device name in /etc/fstab (and possibly in /etc/crypttab) does not match the device name that's being set up in the initramfs. Can you boot with init=/bin/sh, and compare the contents of /etc/fstab with /dev/mapper/ ?

Revision history for this message
Ken Miller (kenny-miller) wrote :

Yes, that was my intial assessment; however:

1. They do match up.
2. It sometimes works

This is why I suspect a race condition.

Regards,

== k ==

Revision history for this message
Steve Langasek (vorlon) wrote :

Well, this bug report was about a race condition specifically with random-crypted tmp and swap partitions. If you are encountering a race condition when mounting your root filesystem, it's an unrelated bug; please file a separate report.

Revision history for this message
Steve Brenneis (sbrenneis) wrote :

I am having the same issue after applying the last round of updates in synaptic. Mine seems related to the CDROM drive. If it is closed during boot, with or without a CD in it, I get the hang more often. If there is a CD in the drive, the system always hangs on startup.

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

I am having the /tmp issue with a Lucid fresh install. Is this really fixed?

Revision history for this message
Ken Miller (kenny-miller) wrote : Re: [Bug 475936] Re: race condition between encrypted device creation and mountall probing with random-encrypted devices (swap, tmp)

On Mon, 2010-05-03 at 13:55 +0000, Vreixo Formoso wrote:
> I am having the /tmp issue with a Lucid fresh install. Is this really
> fixed?
>

I've not tried Lucid yet, but it's managed to "go away" for quite some
time now on Karmic (for me, at least). This is a nasty race condition
that seems to come and go as it pleases.

Regards,

 == k ==

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

I've been reading a bit and I understand that the problem is that: a) at a given time during boot, the tmp encrypted partition gets "decrypted" by whatever process that handles /etc/crypttab. b) then, given the "tmp" option is specified in /etc/crypttab, mkfs.ext2 is executed on the given decrypted partition c) however, at "about the same time" udev detects the new decrypted partition, so ultimately mountall will see and mount it d) If at that time mkfs.ext2 has not finished yet, mountall cannot obviously mount the partition. Am I right? I mean, is this the race condition you mean?

In such case, an easy fix could be to pass the -U option to mkfs.ext2 when creating the new filesystem on the /tmp partition, and specifying in /etc/fstab a mount by UUID instead of a partition like /dev/mapper/crypttmp... Than way mountall should not attempt to mount the partition until filesystem is effectively created. Again: am I right? If so, is there any way to do that? In /etc/crypttab I can only set the filesystem type (with tmp=fs) according to man page... A workaround would be to create a fake filesystem name and custom mkfs.myfakefs to do the right thing, but that would be a very ugly hack, and I am not completely sure it would work...

Or is there any easier fix? Is the workaround in comment #2 (change init scripts dependencies) a better alternative? I do think it is, but I wonder if new problems might appear... I could even mount /tmp on login via libpam_mount, but I also wonder is there is any problem with that...

Thanks in advance,
Vreixo

Revision history for this message
papukaija (papukaija) wrote :

@Vreixo Formoso: This bug is fixed. Please open a new bug by running ubuntu-bug cryptsetup from terminal if you still have problems with your cryptsetup.

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

papukaija: it is not really fixed... I'm having this issue with a fresh lucid install and completely updated system. Should I really open a new (duplicate) bug? Or should I re-open the bug? Thanks

Revision history for this message
Vreixo Formoso (metalpain2002) wrote :

Ok, bug #571682 refers to the same problem and it is opened for lucid. I will comment there. Thanks

Revision history for this message
Mike Buckley (michael-buckley) wrote :

The following script seems to fix the tmp issue. It is a workaround though and not a fix. I've rebooted 5 times in a row and it is still mounting /dev/mapper/crypttmp on /tmp without hanging up the system.
I changed the stock /etc/init/mountall.conf script by commenting out the emits filesystem line. This is on Ubuntu 10.04 LTS.
My custom script called cryptdisks-tmp.conf, placed under /etc/init, has the following lines:
# cryptdisks-tmp - setup encrypted /tmp/device
#
# This is designed simply to setup and encrypted tmp device under
# /dev/mapper called crypttmp and then mount it on tmp.
# This requires a noauto entry in the fstab for the /tmp mount

# Also no crypttab entry is required to setup /dev/mapper/crypttmp
# as all that is taken care of in this script.
# Basically I commented out the "emit filesystem" section in
# /etc/init/mountall.conf, then created this script.
# This script will start once mountall is started and it will emit
# the filesystem emitter
# This will cause most other upstart jobs, which look for the filesystem
# emmitter not to start until this script is done.
# So I think it may fix the /tmp issue
# Also since the code is in a script, I think that the script must finish
# before the filesystem event is emitted. Someone please correct me if I
# am wrong about this.
#
description "script to setup encrypted /tmp to try to bypass race condition"

start on started mountall
emits filesystem
console output
task

script
  if [ -e /dev/mapper/crypttmp ]; then
    echo "For some reason /dev/mapper/crypttmp exists. Bailing out."
    exit 1
  fi
  if mount | grep /tmp > /dev/null 2>&1; then
    echo "/tmp directory is already mounted. Bailing out."
    exit 1
  fi
  ## Now we can proceed to create the /dev/mapper/crypttmp device and mount it.
  if ! cryptsetup create crypttmp /dev/sda3 --key-file=/dev/urandom > /dev/null 2>&1; then
    echo "Failed to create /dev/mapper/crypttmp on /dev/sda3. Bailing out."
    exit 1
  fi
  # Now we need to make the ext2 filesystem on /dev/mapper/crypttmp
  if ! mkfs.ext2 /dev/mapper/crypttmp > /dev/null 2>&1; then
    echo "Create ext2 filesystem on /dev/mapper/crypttmp failed. Bailing out."
    exit 1
  fi
  # We have an fstab entry with noauto in it, so mount /dev/mapper/crypttmp
  if ! mount /dev/mapper/crypttmp > /dev/null 2>&1; then
    echo "Failed to mount /dev/mapper/crypttmp on /tmp. Bailing out."
    exit 1
  else
    echo "/dev/mapper/crypttmp is mounted"
  fi
end script

Revision history for this message
papukaija (papukaija) wrote :

@Mike: This bug is fixed. Please open a new bug (after checking that your problem hasn't been reported) if you still have problems with swap or /tmp.

Revision history for this message
kapetr (kapetr) wrote :

Hello,

I have in Ubuntu 10.10 problem with encrypted swap - maybe it has something common with this bug.

At boot I get every time this info (even with splash on):

The disk drive for /dev/mapper/cryptswap1 is not ready yet or not present Continue to wait; or Press S to skip mounting or M for manual recovery

->see attached screenshot from video of monitor while booting.

But after boot all seems to be OK.
Swap is on - but I don't know if it is encrypted or not because <mount> lists swap not.

??

Is that normal ?

Thanks

--kapetr

Revision history for this message
kapetr (kapetr) wrote :
Revision history for this message
papukaija (papukaija) wrote :

@kapetr: Please open a new bug for your issue as this bug has been fixed a long time ago. Thanks in advance!

Revision history for this message
Dejan (dejan-rodiger) wrote :

Hi,

I think this bug has not been fixed. I have the same problem as kapetr. Encrypted swap is not mounted, but manual swapon activates swap.

kapetr, did you opened new bug? Which number is it?

Thanks

Revision history for this message
Mihai Capotă (mihaic) wrote :

I believe bug #874774 is the new version of this bug, which also affects me.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.