luks encrypted partition not detected

[b2m]

Some encrypted partitions which were created with older versions of cryptsetup (< 1.0.7) are not detected anymore in karmic. The problem is that older cryptsetup does not properly clean the superblock of the partition and this might leave bogus filesystem signatures.

Karmic is more precise on these signatures and gets confused when multiple filesystem signatures are found from single partition and refuses to automatically do anything.

This might manifest itself as simply as an encrypted external harddrive not mounting automatically any more (e.g. like in 9.04) or more severely like full encrypted root filesystem not found after standard distribution upgrade from jaunty to karmic.

Partitions also do not show under /dev/disk/by-uuid/.

[Hamish Downer (mishd)]

It might be related to bug #430605 - also automount not happening, though for different devices.

[Hamish Downer (mishd)]

Just tested this today and it worked fine for me. However bug #430605 was also fixed overnight for me. Can you retest this and report whether it is still an issue?

[b2m (b2m-deactivatedaccount)]

> It might be related to bug #430605 - also automount not happening, though for different devices.

I think not because it affects encrypted devices only.

 - CD: no problem
 - usb stick: no problem
 - same stick with encryped volume: not mounted
 - encrypted external hard drive: not mounted

[buzz (buzz2)]

I can confirm this situation with a new installed karmic version. Unencrypted USB Stick mounts fine (except nautilus is not opening the window), but encrypted external hard disc fails to prompt for a password and is only mountable via terminal.

[Steven Harms (sharms)]

I can also confirm on Karmic Beta 1 + updates.

[Martin Pitt (pitti)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Unfortunately I can't fix it without more information, since I cannot reproduce the problem (it works fine for my encrypted USB stick).

In Ubuntu 9.10 (Karmic) we have a new tool to debug storage related problems. Please run "ubuntu-bug", select "storage problem", and walk through the steps. This will create a new bug report, please send the new bug number here, so that I get to know it quickly.

Thank you!

[buzz (buzz2)]

As this bug also happens with my external harddrive i posted a new bug report via the ubuntu-bug tool: https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/448984

[Martin Pitt (pitti)]

Indeed the dk-disks data for /dev/sdb1 looks empty:

  usage:
  type:
  version:
  uuid:

For my encrypted USB stick it looks like

  usage: crypto
  type: crypto_LUKS
  version: 256
  uuid: 52785e43-6bb9-4f25-985f-7522f606e138

Please copy & paste the output of

  sudo blkid -c /dev/null /dev/sdb1

here. Can you confirm that you have the "cryptsetup" package installed?

[buzz (buzz2)]

buzz@buzz-laptop:~$ sudo blkid -c /dev/null /dev/sdb1
buzz@buzz-laptop:~$
buzz@buzz-laptop:~$ apt-cache policy cryptsetup
cryptsetup:
  Installiert: 2:1.0.6+20090405.svn49-1ubuntu4
  Kandidat: 2:1.0.6+20090405.svn49-1ubuntu4
  Versions-Tabelle:
 *** 2:1.0.6+20090405.svn49-1ubuntu4 0
        500 http://archive.ubuntu.com karmic/main Packages
        100 /var/lib/dpkg/status

[buzz (buzz2)]

Debugging the libblkid revealed that the probe functions detects two filesystem magics on the device.

luks and ext4

as the blkid is set to be not tolerant it aborts the probe and sends a null pointer. if i change blkid to be tolerant it works fine, it shows me the password dialog and mounts the harddrive without problems.

[Martin Pitt (pitti)]

buzz, thanks!

Would it be possible for you to get an image of the first 100 kB of data of this partition? This will contain your encrypted LUKS password and possibly some remainders of file system metadata from the old ext4, so please don't do it if you have potentially sensitive data there. I won't be able to decrypt it, of course, but I'm interested in putting the image on an USB stick and see if I can reproduce the problem, so that blkid can be fixed properly.

  sudo dd if=/dev/sdb1 of=/tmp/ext4-luks.img bs=1K count=100

(It might not actually be /dev/sdb1 if you have other USB drives attached, though).

Alternatively, if you have a proposed patch for blkid, that's appreciated as well, of course. :-)

[buzz (buzz2)]

i would do the next steps private via email if thats fine for you?

[Martin Pitt (pitti)]

buzz [2009-10-12 13:46 -0000]:
> i would do the next steps private via email if thats fine for you?

Sure, thanks!

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Steven Harms (sharms)]

Here is mine to help get this faster. Mine is a luks encrypted ext3 drive.

[buzz (buzz2)]

This bug has been discussed at the mailing list of util-linux: http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/2563/focus=2565

The problem here are devices which have been created with old versions of cryptsetup, new versions wipe out the whole superblock data. At the moment there is no official patch.

[Antti Kaijanmäki (kaijanmaki)]

How about adding a VERY big warning to the release notes if this doesn't get fixed before karmic is released? I too got bitten by this on standard upgrade from jaunty to karmic and now karmic initramfs doesn't find my encrypted LVM (root, swap) partition!

[Antti Kaijanmäki (kaijanmaki)]

Hmm.. odd thing is that if I boot using old jaunty menu entry then the partition is detected correctly...

[Timo Jyrinki (timo-jyrinki)]

Antti: Did you have this problem with a "standard" (automatic encryption from alternate install CD) encrypted HDD or something else? Just curious, since I'm running karmic without problems, and the installation has been originally done from alternate CD with the automatic option. I did however update to karmic already 1-2 months ago.

But apparently the question is more about with which version of Ubuntu the encryption was created, right? I think I started with 8.04 LTS when I got this laptop in May 2008, but I'm not 100% sure.

[Antti Kaijanmäki (kaijanmaki)]

Timo: I installed clean jaunty with "standard" alternate CD HDD encryption. Jaunty ships with cryptsetup 1.0.6 and the problem which causes this was fixed in 1.0.7

from the mailing list in comment #16:

     cryptsetup 1.0.7 changelog:
     - Wipe start of device (possible fs signature) before LUKS-formatting.

So this problem is not going to show up with everyone, but when it does it's nasty and very troublesome. I'll try some dd magic tomorrow to see if I can get the karmic kernel/initramfs to detect the partition correctly.

[Steven Harms (sharms)]

I dont believe we can fix this in an automatic way for the users. The best idea would be to document the dd command to clean the record and put it on the wiki and reference it in release notes.

There is not a reliable way to detect what type of filesystem is on it before cryptsetup 1.0.7, hence the reason for the breakage to begin with.

[Antti Kaijanmäki (kaijanmaki)]

I was wrong on #18. I do not actually have any old jaunty menu entries left, but I have 2.6.30-020630-generic which I installed for intel drivers on jaunty. I just forgot about it and didn't look hard enough last night.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.30/
Anyway this kernel/initrd is able to detect the crypto partition correctly. I don't know why..

[Antti Kaijanmäki (kaijanmaki)]

After some research this is what I've found out; udev is responsible of creating the entries under /dev/disks/by-uuid. Jaunty's udev (which is also in the old initrd) uses udev's vol_id library to detect the FS information. Karmic has new udev which has ditched vol_id and is using blkid instead. That's why the crypto partiton is "correctly" detected with the old initrd.

Now that odd behavior has an explanation and I can safely concentrate on solution.

Steven: How about creating a shell script that would handle the dd magic based on a couple of questions made to the user? Users could use live-cd to download and install the script manually to unencrypted /boot and we could do a safe backup of the superblock so that it can be restored if user has selected a wrong fs type? The scipt could also mount the modified partition as readonly to let the user verify that the files are there or something like that. Anyway it would be nice if there was a solution that would not require the user to do error prone DD magic by hand and is reversible if the user makes a mistake (wrong partition, wrong fs type, etc).

[Antti Kaijanmäki (kaijanmaki)]

OK. I now have a python program (attached) that I successfully used to make my karmic upgrade bootable again.
The idea is to remove all additional signatures from superblock. Currently only removing ext[234] signature is supported, but the program can be extended if needed.

The idea is that you can use live-CD to download the script and run it without needing to install any dependencies or what so ever.

I've tried very hard not to mess with people's hard drives:
 - superblock is not altered without --force option
 - a backup is made before superblock is altered so that changes can be reverted
 - if there's unsupported signatures detected the program refuses to alter the superblock
 - trying to restore a backup of a wrong partition is not allowed
 - I used vbindiff to check that only the right bytes get altered and that the restore works

But of course there might be bugs or it might not work for everyone so here's a quote from the license:
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.

If you use live-CD or something like that make sure to mount, copy and run the script from unencrypted /boot partition so that the backups get really written to disk and you won't loose them when you try to do a reboot.

And last: You have to know which signature is the right one to keep! If you choose a wrong signature and mount the partition as read/write or run wrong fsck on it you will most probably loose your data and a simple superblock backup won't save you.

$ sudo python fix_superblock.py help

Usage:

    # fix_superblock.py [command] [options]

    Available commands:
       - help
       - check
       - fix
       - restore

    check:
      check [partition]

      Print signatures from superblock of [partition].

    fix:
      fix [partition] [type] --force

      Fix the superblock of [partition] to contain only
      the the signature of filesystem [type].

      If --force is omitted the superblock is not altered

    restore:
      restore [backup] [partition] --force

      Restore a superblock backup.

      If --force is omitted the superblock is not altered

$ mkdir boot
$ sudo mount /dev/sda5 boot
$ sudo cp fix_superblock.py boot/
$ cd boot/

$ sudo python fix_superblock.py check /dev/sda1

Found 2 signatures on partition /dev/sda1:
    - crypto_LUKS
    - ext3

$ sudo python fix_superblock.py fix /dev/sda1 crypto_LUKS

Found the wanted signature from superblock.
adding ext3 remove handler
creating backup 'sda1_superblock_20091017130751_701226'
1+0 records in
1+0 records out
65536 bytes (66 kB) copied, 0.000174184 s, 376 MB/s
backup done.
EXT remove handler
EXT signature found
Not removing EXT signature (--force needed)

$ sudo python fix_superblock.py fix /dev/sda1 crypto_LUKS --force

Found the wanted signature from superblock.
adding ext3 remove handler
creating backup 'sda1_superblock_20091017130757_475293'
1+0 records in
1+0 records out
65536 bytes (66 kB) copied, 0.000177885 s, 368 MB/s
backup done.
EXT remove handler
...

[Sean Cox (4o66)]

Antti Kaijanmäki: You sir, are my hero.

The python script did the trick, and now my external firewire drive detects and asks for a password right away.

Thanks!

[Antti Kaijanmäki (kaijanmaki)]

NOTE!
if you see:
    Found 2 signatures on partition /dev/sdXN:
        - ext3
        - ext2

do not try to remove the signatures! Multiple EXT signatures are not a problem.

[jordilin (jordilin)]

I've got this problem as well. I have a usb partition with LUKS and in Jaunty cryptsetup 1.0.6 a window pops up asking for a password. In Karmik nothing happens and I have to mount it manually with mount.crypt (I seem to remember as I don't have my karmik machine in here now).

[Pinpin (spam-sylvain)]

Great Job.
I have got the same problem with my external crypted storage.
The Python script solve it.
Thanks a lot !

[andbelo (andbelo)]

The python script worked for me, too.
Thanks

[Ilkka Laukkanen (ilkka)]

Here's a version of the Python script that I slightly modified so that it will run (at least the "check" part) in Jaunty. The difference is that Jaunty's blkid doesn't recognize the -p parameter and the format of the partition ID lines is slightly different. Same caveats as the original.

[Ilkka Laukkanen (ilkka)]

Now that I thought to check it, Karmic's blkid reports its version so that the version check in fix_superblock_jaunty.py will not work correctly. So, DON'T USE fix_superblock_jaunty.py IN KARMIC. Probably you shouldn't use it at all.

[Antti Kaijanmäki (kaijanmaki)]

I updated my original script to handle removing of swap signatures. I created the test image on jaunty with 16MB USB memory stick by first using mkswap followed by luksformat.

Make sure to read comment #24 before using!

And no, this will not work in jaunty. It's easiest to download karmic live-CD from http://releases.ubuntu.com/releases/9.10/ .

$ sudo python fix_superblock.py check swap_crypto_sig.img

Found 2 signatures on partition swap_crypto_sig.img:
    - crypto_LUKS
    - swap

$ sudo python fix_superblock.py fix swap_crypto_sig.img crypto_LUKS --force

Found the wanted signature from superblock.
adding swap remove handler
creating backup 'swap_crypto_sig.img_superblock_20091019123445_977941'
1+0 tietuetta sisään
1+0 tietuetta ulos
65536 tavua (66 kB) kopioitu0,000731936 sekunnissa, 89,5 MB/s
backup done.
Swap remove handler
SWAP signature 'SWAPSPACE2' found at offset 0xff6
Removing the signature...
Signature removed.

$ sudo python fix_superblock.py check swap_crypto_sig.img

Found 1 signatures on partition swap_crypto_sig.img:
    - crypto_LUKS

[Gunni (fgunni)]

Does this fix the problem permanently?
I come from https://bugs.launchpad.net/ubuntu/+bug/446591 and my workaround there had to be applied after each kernel update (after each initrd creation to be exact if i am right).

[Antti Kaijanmäki (kaijanmaki)]

Gunni: Yes, after fix_superblock.py removes any bogus signatures the partition is once again correctly autodetected permanently. You can safely run

    $ sudo fix_superblock.py check /dev/sdXN

from karmic live-CD and you should see it reporting multiple signatures. If this is not the case then your problem is something else.

[Gunni (fgunni)]

As i can start my system (with my workaround changing cryptsetup to /dev/sda1 instead of the default /dev/disk/by-uuid/xyz1234-...), can i run the script from the booted system (real or recovery)?

[jordilin (jordilin)]

After reformatting my usb using cryptsetup in karmik, it works again with a window popping up requesting for a password. Of course, it was just a pendrive, but it proves that different version of cryptsetup in Karmik was the problem.

[Antti Kaijanmäki (kaijanmaki)]

Gunni:
I would not recommend touching the superblock of a partition which is in use just to be safe, but in theory it should work. I would give it a try, but don't blame me if something breaks, OK ;-)

jordlilin:
actually as stated in this report, the problem is cryptsetup versions prior to karmic.

[Gunni (fgunni)]

Thx, it worked. To be a bit more safe i started in recovery mode and ran the script, and now its fine again.

[Peter Matulis (petermatulis)]

So is someone working on fixing cryptsetup in karmic? I see the importance is Undecided and is not assigned to anyone.

[Antti Kaijanmäki (kaijanmaki)]

AFAIK there's nothing to fix for karmic. Karmic's cryptsetup is already fixed as it clears the superblock of newly created partitions. It's the old versions of cryptsetup that are broken and it might be a good idea to patch them up for every release which still is officially supported.

Anyway IMO this bug report must be kept open so that people will find it through bug search when karmic is released and will get bitten by this when they upgrade.

[Peter Matulis (petermatulis)]

Antti Kaijanmäki wrote:
> AFAIK there's nothing to fix for karmic. Karmic's cryptsetup is already
> fixed as it clears the superblock of newly created partitions. It's the
> old versions of cryptsetup that are broken and it might be a good idea
> to patch them up for every release which still is officially supported.

The truth is that this is going to cause a lot of grief for a great many
users. Think about all the 8.04 users who will eventually upgrade to 10.04.

Is it too dangerous to incorporate the intelligence of your Python
script into karmic's cryptsetup? Maybe invoke debconf for confirmation
of any action? I think updating all old versions is a lot more work
than making the new one more intelligent.

> Anyway IMO this bug report must be kept open so that people will find it
> through bug search when karmic is released and will get bitten by this
> when they upgrade.

Then it will need to stay open until EOL of 8.04 which, for the server,
is April 2013. It's just not the way to go IMO.

[David Brownlee (david-m-brownlee)]

I agree with Peter. The problem isn't that an old cryptsetup was used, that worked fine with the older release. That's just the cause of the problem. The real problem is that upgrading to Karmic will result in a system that isn't bootable and difficult for non-technical people to fix. I imagine that the full disk encryption is a popular option on laptops (since they are easier to steal) so I would think this a release-blocking bug. I'm not sure that big warnings in the release notes would help either given how easy it is to upgrade from update-manager. Better to try to detect this configuration and refuse to update with an error pointing to the bug if this cannot be fixed by release.

[Antti Kaijanmäki (kaijanmaki)]

If we want to concentrate on the "real problem", like David put it, then we should look at udev. Udev is responsible of creating the /dev/disks/by-uuid/ entries and it uses blkid to do so. Missing entries cause the boot to fail with fully encrypted hard drives and udev is the first failing step on automatic handling of other encrypted partitions, too.

Here's what I propose:

1. Create a help.ubuntu.com page
This page should document the problem in detail and give affected people instructions how to fix it. As many people as possible should also blog about it (planet.ubuntu.com) to get google ratings high enough so that affected users find the page.

This page can easily be up until and beyond April 2013.

2. patch udev
Udev should notify that there's multiple signatures on newly available partition. We can then notify the user in appropriate way for the current environment (see below) .

3. inform the user

There are basically three types of environments where this bug occurs:
   a) desktop users with encrypted removable media
   b) full disk encryption during boot
   c) others

a) is the easiest to fix. When user inserts an encrypted USB stick or such during normal desktop session udev notifies that there's multiple signatures and can't go on. We pop up a window to inform the user of the situation and give the link to the friendly help.ubuntu.com page.

For long term solution you can create a nice graphical helper program and launch it here automatically. For karmic's release I don't see there's enough time to do this.

b) is harder as the bug happens on initrd environment which does only contain the bare minimum to access the root partition. At this point there's nothing more we can do other than inform the user on splash screen output and give the link to the help.ubuntu.com page which has specific instructions how to fix the problem on upgraded machine with unaccessible root partition.

Long term solution is to have update manager to check the signature of root partition before upgrading.

c) contains all the other kind of installation, like console only and etc. Just make sure udev or what ever print outs a big informational warning to syslog which gives the link to the help page. Also make sure release notes for upcoming releases contain a warning about the situation. Administrators of critical systems probably read the release notes and thus know what to expect and maybe even fix the signatures before doing the upgrade. And if they don't then they at least should check the syslog when they notice stuff doesn't work how it's meant to.

[Martin Pitt (pitti)]

Peter Matulis [2009-10-21 0:30 -0000]:
> Is it too dangerous to incorporate the intelligence of your Python
> script into karmic's cryptsetup?

I'm not sure it is a good idea to attempt the conversion on the fly
while the file system is being mounted (i. e. during dist-upgrade).

I think what we should do here is:

 * Detect the situation in initramfs/during boot, when file systems
   fail to mount due to this. If so, print an error message with
   instructions.

 * Ship this Python script on the live CD.

 * The instructions should briefly explain what went wrong, and ask
   the user to run a live system and that script.

That would be a safe, but admittedly clumsy first step to get from
"WTH??" to "ugh, complicated, but I know what to do".

A more advanced solution would be to translate/reduce the Pythons
script into a very small shell script which only uses blkid and dd
(and other tools available in the initramfs), then the initramfs could
auto-repair the file systems before mounting them. Anyone happens to
be interested in writing such a thing?

Thanks,

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Martin Pitt (pitti)]

From the discussion with upsptream it became clear that blkid's current behaviour is considered "correct", and that trying to choose between several signatures is a dangerous trait that shouldn't be kept any more.

So from the recent discussion I think that we should modify cryptsetup and its initramfs scripts accordingly (which might include shipping additional udev rules).

[Antti Kaijanmäki (kaijanmaki)]

Martin: What do you mean by modifying cryptsetup? It's not causing anything anymore on karmic.

Patching cryptsetup on hardy, intrepid and jaunty however to clear the superblock of newly created partitions would be in order.

[Steve Langasek (vorlon)]

If we accept upstream's position that blkid shouldn't pick one fs id over the other (which I disagree with - this is a case where we clearly know which of the UUIDs is the correct one, which is why we're talking about autocleaning the superblock on upgrades, so why don't we just fix blkid to DTRT instead of adding another layer to work around blkid's behavior after the fact?), then AFAICS cryptsetup is still not the right package to fix this, because in an ideal world the cryptsetup initscripts and upstart jobs will all go away in favor of udev rules, and then there's nothing to trigger cryptsetup's cleaning in the first place /because/ of this bug. I think this bug belongs to udev.

[Steve Langasek (vorlon)]

Opening a release notes task, since this information was lost when 362315 was marked as a duplicate and we definitely want this added to the release notes.

[Antti Kaijanmäki (kaijanmaki)]

Steve: I must be missing something here.. how do we know which UUIDs is the correct one? Especially with removable media?

And yes, you could use the uuid parameter from root boot option and patch blkid to force read the uuid's from every type of signature it founds and then guess the right type by matching the uuid's and...

Blkid is doing the right thing. It clearly indicates that there are multiple signatures and refuses to do any guess work. The fact that we are in this situation because of buggy cryptosetup is unfortunate (same will happen with any mkfs which does not clear the superblock).

Now we just have to deal with the defected superblocks out there and in my opinion patching and changing working software that's doing what it suppose to do just to deal with buggy legacy is not the way to go. There are situations where that most probably goes wrong as we guess wrong and trash users data (how about activating a "new" swap partition which in fact is encrypted luks partition).

We can clearly detect the situation and we can inform the users about it and we can help them fix it for good. Users know which signature is the correct one. Let's leave the decision to the user, but give all the information and help we can to make it a correct one.

[Martin Pitt (pitti)]

We discussed that in #ubuntu-release. Given how close karmic's release is, we think this is a viable strategy:

 (1) For karmic, patch blkid to prefer luks over ext*

    Rationale: This has been the behaviour for years, and worked well. We also know that creating a luks volume did not properly clean up traces from a previous ext file system, while we know that creating an ext file system over a luks file system worked properly. (I'll verify that again, though)

  Reopening an util-linux task for that.

 (2) For Lucid, discuss a proper migration strategy to clean up superblocks on upgrades. This isn't release critical for Karmic any more then.

  Keeping non-karmic cryptsetup task for that.

[Antti Kaijanmäki (kaijanmaki)]

Martin: this happens also with swap and probably other FS' too. Simply prefer over ext is not sufficient.

[Martin Pitt (pitti)]

For the record, I'm not really insisting on putting the migration stuff into cryptsetup; if another package is more appropriate (like udev), I'm fine with that as well, of course. I just thought that we don't need the migration scripts on systems where cryptsetup isn't even installed.

[Antti Kaijanmäki (kaijanmaki)]

If you decide to patch blkid, make sure to test it darn well! This is not just like reverting a patch or two on blkid and you have to be very careful. blkid has not been used on previous releases to detect partitions so you can't make any assumptions how patched blkid will be performing compared to vol_id.

[Martin Pitt (pitti)]

Antti Kaijanmäki [2009-10-21 8:44 -0000]:
> Martin: this happens also with swap and probably other FS' too. Simply
> prefer over ext is not sufficient.

So for karmic, we should perhaps just prefer it over anything else? Or
at least over swap and ext*? I think these two are the only ones that
truly matter for booting. As you say, we shouldn't do too much
guesswork on broken USB sticks anyway (with cases like luks over VFAT,
or VFAT over luks, both of which are plausible, and both of which are
known not to clean up superblocks properly).

Thanks, Martin

[Martin Pitt (pitti)]

I declined the hardy/intrepid/jaunty nominations. I don't think that there's anything to do there:

 * In these releases, cryptsetup still was broken, but vol_id preferred luks over other FSes, so it didn't manifest as a practical problem.
 * Fixing cryptsetup in those releases can't be sufficient; we cannot assume that updating cryptsetup in stables magically fixes all the existing unclean superblocks out there.
 * Introducing complex migration scripts into stable releases is out of the question, since they have a high potential to screw everything up even further.

We need to introduce migration scripts into the release that stops preferring luks over other FSes. As discusssed above, it's too late to do that for karmic, so it should be done in lucid (which is also LTS, and thus a convenient place to do that).

[Antti Kaijanmäki (kaijanmaki)]

Only reason for nominations was to make sure existing releases do not create *more* broken superblocks, but if you plan to handle them automatically in future releases then it doesn't really matter.

[Martin Pitt (pitti)]

As a first step, I created a script which exercises all combinations of luks with ext2/ext3/ext4/swap/vfat.

This runs fine on current karmic, which shows that all the mkfs.* are fixed now and we don't create further breakage with the karmic versions. The script needs to run as root, and uses /dev/ram0 for testing, so please make sure that you don't use this! (or change it in the script, it's a variable at the top). Expected output:

$ sudo ./test-luks-blkid.sh
*** test pair: luks/ext2 ***
ext2 luks ext2
*** test pair: luks/ext3 ***
ext3 luks ext3
*** test pair: luks/ext4 ***
ext4 luks ext4
*** test pair: luks/swap ***
swap luks swap
*** test pair: luks/vfat ***
vfat luks vfat

On Jaunty, this demonstrates that cryptsetup doesn't properly clean traces of ext:

$ ./test-luks-blkid.sh

*** test pair: luks/ext2 ***
ext2 luks
FAIL: Expected fs type: crypto_LUKS; blkid result
/dev/ram0: LABEL="MyExt2" UUID="a359cbd4-d406-4aa8-a0a2-807697a50710" TYPE="ext2"

Since most things in Jaunty use vol_id, I also added a vol_id mode to the script, which gets a little further due to the different preference order of vol_id. It still stumbles over luks-after-swap:

[jaunty] # ./test-luks-blkid.sh vol_id
*** test pair: luks/ext2 ***
ext2 luks ext2
*** test pair: luks/ext3 ***
ext3 luks ext3
*** test pair: luks/ext4 ***
ext4 luks ext4
*** test pair: luks/swap ***
swap luks unknown or non-unique volume type (--probe-all lists possibly conflicting types)
[jaunty] # vol_id --probe-all /dev/ram0
swap
crypto_LUKS

On hardy, blkid behaves exactly the same. vol_id did not yet prefer luks over swap at that time (and the script can't test ext4, sinced that wasn't available yet):

[hardy] # ./test-luks-blkid.sh vol_id
*** test pair: luks/ext2 ***
ext2 luks ext2
*** test pair: luks/ext3 ***
ext3 luks ext3
*** test pair: luks/swap ***
swap luks
FAIL: Expected fs type: crypto_LUKS; vol_id result
ID_FS_USAGE=other
ID_FS_TYPE=swap
ID_FS_VERSION=2
ID_FS_UUID=fa48b160-08fb-44a5-9db5-1be7440912e5
ID_FS_UUID_ENC=fa48b160-08fb-44a5-9db5-1be7440912e5
ID_FS_LABEL=
ID_FS_LABEL_ENC=
ID_FS_LABEL_SAFE=

[hardy] # vol_id --probe-all /dev/ram0
swap
crypto_LUKS

Another real-world test case is the ext3-luks.img which is attached here, which can be used to verify a proposed karmic fix for blkid:

$ BLKID_DEBUG=0xffff blkid -p /tmp/ext3-luks.img
[...]
ERROR: ambivalent result detected (2 filesystems)!
/tmp/ext3-luks.img: ambivalent result (probably more filesystems on the device)

[Martin Pitt (pitti)]

Improved version which uses blkid -p if available (not yet on hardy/jaunty, but on karmic).

I installed hardy's cryptsetup (2:1.0.5-2ubuntu12) on karmic, and as expected the script fails now:

$ sudo ./test-luks-blkid.sh

*** test pair: luks/ext2 ***
ext2 luks /dev/ram0: ambivalent result (probably more filesystems on the device)

So the goal for Karmic is that the test succeeds under those conditions.

[Fabián Rodríguez (magicfab)]

I hit this a few weeks ago when upgrading from Jaunty to Karmic beta, here is the upstream I found:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=541823

At the time I asked a few colleagues about it and no one had hit it so I assumed something wrong with my own setup. My original setup was Alternate Install + LVM encrypted root (Jaunty).

My workaround was to add this to kernel options to test botting once:
cryptopts=target=sda1_crypt,source=/dev/sda1,lvm=bianca-root root=/dev/mapper/bianca-root ro

Then once I confirmed it worked, I made it permanent by adding it to kopts in /boot/grub/menu.lst (I don't use grub2):
# kopt=cryptopts=target=sda1_crypt,source=/dev/sda1,lvm=bianca-root root=/dev/mapper/bianca-root ro

Make sure cryptopts is before root= or else this won't work.

[Steve Langasek (vorlon)]

Documented in https://wiki.ubuntu.com/KarmicKoala/TechnicalOverview:

When creating LUKS encrypted partitions, some earlier versions of cryptsetup did not wipe out any pre-existing filesystem metadata on the partition. The current version of blkid used in the Ubuntu 9.10 RC will refuse to export a UUID for a partition containing more than one type of metadata signature. This means that encrypted disks may fail to be decrypted at boot time, possibly preventing the system from booting at all. Users of LUKS system-level disk encryption are advised to wait until the Ubuntu 9.10 final release before upgrading. (428435)

(obviously the text would need revising if this bug persists in final)

[Steve Langasek (vorlon)]

Copying from IRC, the rationale why it's reasonable to fix this in blkid:

Given that no LUKS partition *can* be activated without external configuration telling how to decrypt it (i.e.: /etc/crypttab), I think in all cases it's safe to prefer LUKS over * if the alternative is returning no UUID at all

[Steve Langasek (vorlon)]

i.e.: the rationale for not guessing which signature to prefer is that it's not *safe*. But LUKS is always safe, because this never goes any further than a symlink under /dev unless something else on the system is already configured to look for *that specific UUID* and treat it as a LUKS device.

[Martin Pitt (pitti)]

Scott uploaded the fix, see http://launchpadlibrarian.net/34100474/util-linux_2.16-1ubuntu5_source.changes (in unapproved now).

I confirm that this makes the test script succeed again, and also works with Steven Harms' image:

$ blkid -p /tmp/ext3-luks.img
/tmp/ext3-luks.img: UUID="fa583620-b1ef-448c-b4c3-7aaa794849f8" VERSION="256" TYPE="crypto_LUKS" USAGE="crypto"

[Launchpad Janitor (janitor)]

This bug was fixed in the package util-linux - 2.16-1ubuntu5

---------------
util-linux (2.16-1ubuntu5) karmic; urgency=low

  * Always return encrypted block devices as the first detected encryption
    system (ie. LUKS, since that's the only one) rather than probing for
    additional metadata and returning an ambivalent result. LP: #428435.

 -- Scott James Remnant <email address hidden> Wed, 21 Oct 2009 14:22:31 +0100

[Jonas H (jonash)]

I've got a manual cryptsetup setup, with /dev/sda1 as encrypted partition which has ext2. On the encrypted pseudo-partition /dev/mapper/_dev_sda1 I have an ext3 filesystem.

Can I safely upgrade to karmic? Seems like this case is somehow special.

[Michael Fritscher (michael-fritscher)]

Shouldn't be a problem because you don't use uuids.

[Jonas H (jonash)]

Er, what do you mean by "I do bot use uuids"? UUIDs for what?

[Michael Fritscher (michael-fritscher)]

you can access partitions using the automounter & UUIDs, e,g. /dev/disk/by-uuid/some_long_number. This way you'll stumble over this bug.
Using the tradiional way - /dev/sdx# you don't get this problem.

[Julien Plissonneau Duquene (julien-plissonneau-duquene)]

Nice to see that at last some sanity finally got in. I am a bit late here, but I have another suggestion below. Skip to the end to avoid the rant.

Context: FYI, yesterday I had to patch (live) the bootblock of a Debian "squeeze" system that stopped to boot a few updates ago. It was a pain diagnosing that one, because it was installed "cleanly" from scratch (I thought), and blkid on the live system reported both partitions: /dev/hda1 root ext3, and /dev/hda5 swap. In the initramfs, only swap was detected.

Side note: usually on a PC I like my MBR being a regular PC MBR (or debian's enhanced MBR that allows you to choose the partition), and my GRUB being on the boot block of the "boot" (if any) or "root" partition. Telling users that GRUB should be on the MBR is, IMHO, asking for trouble. But for the system above I decided to go by the book and just follow what the install CD told me to do next, next, next.

Context again: Finding what was wrong was a pain. First, I had to figure out that blkid used a cache, which explained why the root partition was shown on the live system but not at boot time. Using "-c /dev/null" the same problem was visible on the live system. Deleted the cache, hacked grub.conf, lazily waited a few weeks for the fix to come (or not) because my last experience of sending blkid bug reports left me with the impression that the only sane thing to do would be to fork the package and took it off the hands of its current maintainers.

Context continued: Somehow the ext3 partition managed to get back in blkid's cache over these weeks. Thought it was solved, tried "-c /dev/null", nope, deleted cache, nope. Then finally decided to take a few hours solving that "perfectly-legit-installed-by-the-book-no-longed-detected-ext3-partition" mystery. Played around with blkid options but could not get anything useful of it, excepted that "probably more filesystems" message (option -p) that did not tell which other signatures it found. Downloaded source, activated all debug flags in libblkid, tried again. Could finally see that libblkid detected a vfat and an ext3 signature. vfat?!? Then FINALLY I checked hda1's boot block, and found out that nothing erased the vfat boot block when I installed the Debian system on it, following the regular procedure. That old box used to have a W2K system on it... Of course, there are failures on multiple levels here, just like in this bug report.

Now the suggestion for blkid:

Implement an option (e.g. -a for "all") that will use blkid_do_probe and a loop instead of blkid_do_safeprobe, and report all detected signatures.

I guess that it should make everybody happy, because scripts can now try the "safe" (aka your system is so safe it won't boot anymore) version first, then -a, then handle special cases at script level or issue warning or error messages as needed. But the system will boot.

Alternative: a second run of blkid_do_probe in the error message of lowprobe_device so the poor user can get an idea of which signatures are confusing blkid.

[Martin-Éric Racine (q-funk)]

While I don't have an encrypted partition, I however encountered the issue that blkid reports an ambivalent filesystem signature on my root partition, which suddenly makes udev skip detection of this partition and, as a result, fail to reboot. Trying to run Antti's script, it reports having found two signatures (vfat and ext3). ext3 is the only one that should exist, however, asking the script to remove the vfat signature reports "Don't know what to do with signature 'vfat' Refusing to continue". Any solution?

[Steve Langasek (vorlon)]

resolved for karmic, doesn't need to be included in the final release notes.

[Ben Willmore (bdeb)]

Martin-Éric -- if you need a workaround, try adding 'GRUB_DISABLE_LINUX_UUID=true' to your boot parameters, and changing 'root=UUID=foo' to 'root=/dev/hda1' or whatever your boot partition actually is. This allowed me to boot my machine (which doesn't have an encrypted partition either) after upgrading to Karmic, and finding the same ambivalent signatures problem prevented me from booting.

[Martin Pitt (pitti)]

Ben Willmore [2009-10-29 16:42 -0000]:
> Martin-Éric -- if you need a workaround, try adding
> 'GRUB_DISABLE_LINUX_UUID=true' to your boot parameters, and changing
> 'root=UUID=foo' to 'root=/dev/hda1' or whatever your boot partition
> actually is. This allowed me to boot my machine (which doesn't have an
> encrypted partition either) after upgrading to Karmic, and finding the
> same ambivalent signatures problem prevented me from booting.

Hm, this problem should be fixed in Karmic for over a week. If you
still have it, please submit the output of

  sudo BLKID_DEBUG=0xffff blkid -p /dev/yourencryptedpartition

[Ben Willmore (bdeb)]

Martin-Éric and I are having a problem very like this one, but with non-encrypted partitions. Our bug is definitely not resolved in Karmic. It looks like we should be here: <https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/426027>. Sorry for the confusion.

[Martin-Éric Racine (q-funk)]

Martin, this issue appears on a non-encrypted partition.

The result of the blkid command has been submitted to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=552578

[kautilya (jedezhath)]

Hi Martin,
I am also struggling with a system that wouldn't boot because this bug affecting my "home' partition which is not encrypted. I have been using Karmic for a few weeks and suddenly the system wouldn't boot because of this. I tried the python, but that doesn't help: "Found the wanted signature from superblock. Don't know what to do with signature 'minix' Refusing to continue." Please see the output of BLKID_DEBUG as an attachment.

Thanks

[Tsu Jan (tsujan2000)]

I think, I've encountered this bug in Debian squeeze too, but in a safe way. I have 9 partitions on my desktop computer (P5QC) and only the UUID of sda9 isn't detected. Fortunately, it isn't the root partition.

I've never installed Windows on this computer but formatted its hard with etx3 from the start. My udev's version is 150-2. I also have an old Ubuntu Hardy, in which the UUID of sda9 is correctly detected. On Debian, this behavior happened since a few weeks ago, perhaps after udev was updated.

"sudo blkid -o udev -p /dev/sda9" gives:

/dev/sda9: ambivalent result (probably more filesystems on the device)

Is this problem related to partitioning with an older version of parted (GParted)? Or it's udev's fault?

[Tsu Jan (tsujan2000)]

I forgot to say that none of my partitions is encrypted. So, I think this bug isn't especially related to encryption.

[ceg (ceg)]

---------------
util-linux (2.16-1ubuntu5) karmic; urgency=low

  * Always return encrypted block devices as the first detected encryption
    system (ie. LUKS, since that's the only one) rather than probing for
    additional metadata and returning an ambivalent result. LP: #428435.

might that cause the precedence of reporting luks over raid_member for the usb disk and thus bug #531240 (blkid reports root raid_member (on usb) as luks, which is booted while raid remains "inactive")?

[asi (gmazyland)]

If blkid reports two signatures for one device (LUKS + ext3/swap) it is bug (introduced when old cryptsetup did not wipe old signature).

You cannot solve it updating to new cryptsetup version but you can use new cryptsetup version to fix it.

Read http://code.google.com/p/cryptsetup/wiki/FrequentlyAskedQuestions#3._Common_Problems
"blkid" sees a LUKS UUID and an ext2/swap UUID on the same device. What is wrong?
for fix description.

Alternatively you can use new "wipefs" command from util-linux-ng 2.17 (and select proper signature to wipe).
(wipefs is btw universal solution for these problems - it can selectively destroy old signatures - just by wiping the magic string
which blkid uses to detect device types)

[John Parrish (jp-dcresearch)]

Having used "Startup Disk Creator" with a 10.10BETA release acquired on 9/25/2010 i386 Desktop... I had this problem where the OS didn't recognize the disk that had the LUKS encrypted drive. I had initially installed 9.10 from the Alternate install CD.

Anyway - I'll try installing from the CD instead of a USB Stick.

FYI