Does not automount unpartitioned devices

[Martin Pitt]

Binary package hint: gvfs

Since a while ago (within lucid, sometime after alpha-1) my e-book stopped being automounted in GNOME. udisks --mount /dev/sdb works fine, but gvfs does not get to see the drive nor volume.

Normal partitioned USB sticks work fine, but the internal storage of the e-book is unpartitioned (raw /dev/sdb).

ProblemType: Bug
Architecture: amd64
Date: Wed Mar 10 15:10:23 2010
DistroRelease: Ubuntu 10.04
EcryptfsInUse: Yes
HotplugNewDevices: /dev/sdd /dev/sdb /dev/sdc
HotplugNewMounts:

Package: gvfs 1.5.5-0ubuntu1
ProcEnviron:
 PATH=(custom, user)
 LANG=de_DE.utf8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-16.24-generic
SourcePackage: gvfs
Symptom: storage
Title: Does not detect hotplugged storage device
Uname: Linux 2.6.32-16-generic x86_64

[Martin Pitt (pitti)]

I just checked that I can reproduce that in the test suite, which uses a clean and zeroed-out device, so it's not the usual "gets confused about two different signatures on the device".

http://cgit.freedesktop.org/udisks/commit/?id=47ac2676ca6d209705f9f814cded2daf19f24e76

Now the test suite properly fails:

$ sudo tests/run FS.test_vfat
Testing binaries from local build tree
daemon path: src/udisks-daemon
mdadm: array /dev/md125 started.
fs: FAT ... [cli]FAIL

======================================================================
FAIL: fs: FAT
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/run", line 412, in test_vfat
    self._do_fs_check('vfat')
  File "tests/run", line 443, in _do_fs_check
    self._do_mkfs_check(type)
  File "tests/run", line 473, in _do_mkfs_check
    self.failIf ('partition_scheme' in i)
AssertionError

[Martin Pitt (pitti)]

Interesting, it only seems to happen for VFAT, not for any other file system.

It seems likely that the bug is actually in libparted:

$ sudo /lib/udev/udisks-part-id /dev/sdb
device=0xe56e00' for devpath=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-8/1-8.2/1-8.2:1.0/host13/target13:0:0/13:0:0:0/block/sdb
Expected `linear' in UDISKS_DM_TARGETS_TYPE=`(null)'
using device_file=/dev/sdb syspath=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-8/1-8.2/1-8.2:1.0/host13/target13:0:0/13:0:0:0/block/sdb, offset=0 ao=0 and number=0 for /dev/sdb
Entering MS-DOS parser (offset=0, size=1028653056)
MSDOS_MAGIC found
looking at part 0 (offset 0, size 0, type 0x00)
new part entry
looking at part 1 (offset 0, size 0, type 0x00)
new part entry
looking at part 2 (offset 0, size 0, type 0x00)
new part entry
looking at part 3 (offset 0, size 0, type 0x00)
new part entry
Exiting MS-DOS parser
MSDOS partition table detected
UDISKS_PARTITION_TABLE=1
UDISKS_PARTITION_TABLE_SCHEME=mbr
UDISKS_PARTITION_TABLE_COUNT=0

[Zeuthen (zeuthen)]

(In reply to comment #2)
> Interesting, it only seems to happen for VFAT, not for any other file system.
>
> It seems likely that the bug is actually in libparted:

Nitpick: We don't use libparted code at all for detection - the code in question is in src/helpers/partutil.c

> $ sudo /lib/udev/udisks-part-id /dev/sdb
> device=0xe56e00' for
> devpath=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-8/1-8.2/1-8.2:1.0/host13/target13:0:0/13:0:0:0/block/sdb
> Expected `linear' in UDISKS_DM_TARGETS_TYPE=`(null)'
> using device_file=/dev/sdb
> syspath=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-8/1-8.2/1-8.2:1.0/host13/target13:0:0/13:0:0:0/block/sdb,
> offset=0 ao=0 and number=0 for /dev/sdb
> Entering MS-DOS parser (offset=0, size=1028653056)
> MSDOS_MAGIC found
> looking at part 0 (offset 0, size 0, type 0x00)
> new part entry
> looking at part 1 (offset 0, size 0, type 0x00)
> new part entry
> looking at part 2 (offset 0, size 0, type 0x00)
> new part entry
> looking at part 3 (offset 0, size 0, type 0x00)
> new part entry
> Exiting MS-DOS parser
> MSDOS partition table detected
> UDISKS_PARTITION_TABLE=1
> UDISKS_PARTITION_TABLE_SCHEME=mbr
> UDISKS_PARTITION_TABLE_COUNT=0

The (known) problem is that VFAT on the main block device looks a lot like a Master Boot Record partition table. We need to fix the MBR probing code.

[Martin Pitt (pitti)]

Personal notekeeping:

I checked out a very old udisks version, and the partition prober has the same problem. It just was hidden in the past by the different media detection logic.

Commit 589ec28a2c156676d4efe976e49bd87226e7dad2 works (wrt. the DeviceIsPartitionTable property), commit d1c41e95ffe11917dd945694be5ebad9a831d294 fails. I didn't bisect it down, since the particular patch which exposed that bug is most probably not at fault, we need to fix part-id itself.

[Martin Pitt (pitti)]

Got to stop for today. Ideas for tomorrow:

 - It's implausible to have a partition start at offset 0 (since that's where the MBR is)
 - TODO: check if partition type 0 is allowed
 - TODO: current blkid can tell them apart just fine, check its logic (since eventually udisks-part-id will go away, and we'll use blkid for partition probing)

[Zeuthen (zeuthen)]

(In reply to comment #5)
> Got to stop for today. Ideas for tomorrow:
>
> - It's implausible to have a partition start at offset 0 (since that's where
> the MBR is)
> - TODO: check if partition type 0 is allowed
> - TODO: current blkid can tell them apart just fine, check its logic (since
> eventually udisks-part-id will go away, and we'll use blkid for partition
> probing)
>

Right. Might be worth looking at the kernel partition table probing code too (it's in fs/partitions/msdos.c).

Note that partition type 0 and/or offset 0 and/or size 0 normally means "partition slot not used". For example, it's perfectly valid to have only two partitions in slots 1 and 3 - and the kernel represents such a device as /dev/sda for the whole disk and /dev/sda1 and /dev/sda3 for the two partitions.

[Martin Pitt (pitti)]

Binary package hint: gvfs

Since a while ago (within lucid, sometime after alpha-1) my e-book stopped being automounted in GNOME. udisks --mount /dev/sdb works fine, but gvfs does not get to see the drive nor volume.

Normal partitioned USB sticks work fine, but the internal storage of the e-book is unpartitioned (raw /dev/sdb).

ProblemType: Bug
Architecture: amd64
Date: Wed Mar 10 15:10:23 2010
DistroRelease: Ubuntu 10.04
EcryptfsInUse: Yes
HotplugNewDevices: /dev/sdd /dev/sdb /dev/sdc
HotplugNewMounts:

Package: gvfs 1.5.5-0ubuntu1
ProcEnviron:
 PATH=(custom, user)
 LANG=de_DE.utf8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-16.24-generic
SourcePackage: gvfs
Symptom: storage
Title: Does not detect hotplugged storage device
Uname: Linux 2.6.32-16-generic x86_64

[Martin Pitt (pitti)]

Confirmed by a duplicate bug. I also reformatted an USB stick (which previously automouted just fine) to have a VFAT partition on the raw device, and can reproduce the problem with that.

[Martin Pitt (pitti)]

So, both the kernel code and blkid look surprisingly similar (what a coincidence :-P). I also checked the MBR spec [1], and if we allow 0 offset/type partitions for empty partitions we can't indeed reliably detect an MSDOS partition table by itself, we have to check whether it is a valid FAT header (which has a much more rigid structure), and if we don't find one, assume that we have an MBR.

Now, I see two possibilities:

 1) We can copy the FAT detection logic from blkid (which is a tad more rigid than the kernel's)

 2) We can assume that blkid is reliable (I don't see much evidence that it isn't), and skip partition probing if we already have a fs:

  ENV{ID_FS_TYPE}=="", IMPORT{program}="udisks-part-id $tempnode"

(2) is essentially the same logic as (1) but avoids the overhead of calling part-id if we already know that there's a file system. It's also one step closer to getting rid of part-id in favor of a blkid based partition probing.

I verified that this rule brings back automounting, and the automatic tests are happy again as well.

So I'd favor (2), but if you see a reason to always call part-id, I can also work on (1).

[1] http://en.wikipedia.org/wiki/Master_boot_record

[Martin Pitt (pitti)]

Keeping notes: gdu volume monitor ignores /dev/sdb in should_drive_be_ignored() because gdu_pool_get_enclosed_presentables(/dev/sdb) returns an empty list.

[Martin Pitt (pitti)]

Debugging further, gdu gets confused because udisks claims that /dev/sdb is a partition table (DeviceIsPartitionTable property is true).

[Zeuthen (zeuthen)]

(In reply to comment #7)
> So, both the kernel code and blkid look surprisingly similar (what a
> coincidence :-P). I also checked the MBR spec [1], and if we allow 0
> offset/type partitions for empty partitions we can't indeed reliably detect an
> MSDOS partition table by itself, we have to check whether it is a valid FAT
> header (which has a much more rigid structure), and if we don't find one,
> assume that we have an MBR.
>
> Now, I see two possibilities:
>
> 1) We can copy the FAT detection logic from blkid (which is a tad more rigid
> than the kernel's)
>
> 2) We can assume that blkid is reliable (I don't see much evidence that it
> isn't), and skip partition probing if we already have a fs:
>
> ENV{ID_FS_TYPE}=="", IMPORT{program}="udisks-part-id $tempnode"
>
> (2) is essentially the same logic as (1) but avoids the overhead of calling
> part-id if we already know that there's a file system. It's also one step
> closer to getting rid of part-id in favor of a blkid based partition probing.
>
> I verified that this rule brings back automounting, and the automatic tests are
> happy again as well.
>
> So I'd favor (2), but if you see a reason to always call part-id, I can also
> work on (1).

Let's just go for (2) - seems simpler. Thanks.

>
> [1] http://en.wikipedia.org/wiki/Master_boot_record
>

[Christoph Reiter (lazka)]

I just wanted to say that I have the same problem, but with my (full of music) sansa clip.
It doesn't get automounted.

udisks --mount /dev/sdc works, but the label is missing and it doesn't show up in the media player (because of the partition table thing)

[Martin Pitt (pitti)]

http://cgit.freedesktop.org/udisks/commit/?id=cfc669846a53815517d32d0b7b96cbc4935cea7a

[Martin Pitt (pitti)]

I committed a fix upstream: http://cgit.freedesktop.org/udisks/commit/?id=cfc669846a53815517d32d0b7b96cbc4935cea7a

[Zeuthen (zeuthen)]

Actually I had to revert this:

http://cgit.freedesktop.org/udisks/commit/?id=b51dc7ac853ec3f11735a0e893dfe11365f14ee2

We should probably do the ID_FS_TYPE comparison inside part-id only when we are sure we are dealing with the main block device and not a partition. Have to be careful here, remember we support kpartx style partitions now...

[Martin Pitt (pitti)]

(In reply to comment #10)
> We should probably do the ID_FS_TYPE comparison inside part-id only when we are
> sure we are dealing with the main block device and not a partition. Have to be
> careful here, remember we support kpartx style partitions now...
>

> This commit actually broke partition table parsing for the partitions itself,
> e.g. this output [...] if the partition in question had a recognizable
> filesystem.

Hm, I'm afraid I don't follow -- If a partition has a recognizable file system, how can it have a partition table at the same time? The device which you quoted in the reversion git log, which ID_FS_* properties did it have, and which kind of partition was that?

I see that the previous change would break situations where a particular device (raw or partition, doesn't matter) is both a fs and a partition table, but I don't know how this is possible?

[Martin Pitt (pitti)]

Ergh, sorry for my density, ignore my previous comment. I'll work on a better fix.

[Martin Pitt (pitti)]

I made the test suite more complete (test partitions with and without file systems), to catch that regression:

http://cgit.freedesktop.org/udisks/commit/?id=b2b116296d9973203c20be1a1060015ebdd1ce39

[Martin Pitt (pitti)]

Ok, fixed for good now:

  http://cgit.freedesktop.org/udisks/commit/?id=65dd5515ada9f92ee69fe5709024eaefef01c4e2

[Martin Pitt (pitti)]

A new udisks package has been uploaded to lucid, but it's stuck in unapproved until the beta-1 freeze lifts.

[Launchpad Janitor (janitor)]

This bug was fixed in the package udisks - 1.0.0-0ubuntu1

---------------
udisks (1.0.0-0ubuntu1) lucid; urgency=low

  [ Martin Pitt ]
  * debian/control: Prefer building against current libparted0-dev. Add
    alternative libparted-dev build dependency as well.
  * debian/rules: Build a PO template during package build, to have an up to
    date one for translators. (LP: #538320)
  * Drop debian/local/ubuntu.pkla again: This will be provided by a new
    policykit-desktop-permissions package.

  [ Michael Biebl ]
  * New upstream release.
    - Clean up stale mounts from devices in /etc/fstab, too. (LP: #476654)
    - Fix recognition of unpartitioned VFAT drives, which were previously
      misdetected as MBR. (LP: #536670)
    - Use lseek64() instead of lseek() in job-drive-benchmark.c, to also work
      with large devices. (LP: #526524)
  * Refresh patches.
 -- Martin Pitt <email address hidden> Tue, 16 Mar 2010 14:57:23 +0100

[Christoph Reiter (lazka)]

Automount works now (and it's no longer a partition table), but it gets mounted to /media/%01 and the label is wrong.

[Jacob Peddicord (jpeddicord)]

The behavior I see now is the same as it was when it was working; perhaps a different bug should be filed for that?

[Lai Jiang (laijiang)]

I also have this problem, but for me even partitioned usb drive can not automount. I've tried several usb drives and have no luck. Manually mounting with udisks works fine. However when I report through ubuntu-bug it identifies gvfs as the package affected. Also I only had this problem since gvfs updated several days ago. I'm using Lucid Lynx.

[Sebastien Bacher (seb128)]

could you open a new bug using "ubuntu-bug storage"?

[Manjul Apratim (manzdagratiano)]

I experienced the same issue after a fresh install of Lucid - the Live USB would not automount when inserted after the system was running, and it was formatted as FAT. This is a regression.

[Manjul Apratim (manzdagratiano)]

Is the `fix-released' status appropriate? - since this bug was present in the final release as well. I would be happy to file a new bug report if that is what is warranted. Automount is a fairly important issue for me since I carry around with me four external HDDs.

[Martin Pitt (pitti)]

manzdagratiano [2010-05-04 5:21 -0000]:
> Is the `fix-released' status appropriate?

Yes, since the root cause of this bug was fixed.

If you still have a problem like this, please file a new bug with
"ubuntu-bug storage".

Thanks, Martin

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[alcarola (alcarola)]

I did a sudo apt-get remove halevt and that fixed it for me. I might have installed this package by accident.

Now my USB stick mounts, the camera mounts via USB cable and also its memory card via a cardbus adapter. All partitions on an external USB drive get mounted. (My old Ipod Shuffle gets recognised by dmesg and udisks but not mounted.) Before I removed halevt, udisks found nothing.

Hope this can be of help to someone.

[alcarola (alcarola)]

Slightly off-topic update: After I did a sudo apt-get remove libpod-common as suggested in http://ubuntuforums.org/showthread.php?t=1473082 the Ipod Shuffle (1st gen 512 MB) now mounts and opens automatically in Rhythmbox.

[narea92 (arena-g)]

I suffer also with the same problem!.
Removing usbmount, hal, installing libpmount0.0, changing the Bios settings did not help.
Gaetano Arena

[narea92 (arena-g)]

I am adding this note with the hope that it can be of help.
Actually, the Automount function operate properly and the icon is displayed on Places > Computer.
No icon, however, on the Desktop.
Gaetano Arena

[Manjul Apratim (manzdagratiano)]

I would like to point out that I experienced this problem only when I had tried to install an ext4 file system to root - I have used an ext4 system under Jaunty and Karmic before, without issues, but for some reason an ext4 on Lucid gave me automount issues on both my laptops - one is a Sony Vaio AR520E and the other is a Dell Vostro V13. I do not know if the issue is firmware related. However, formatting the root with ext3 the next time did away with this issue, and automount has since been working like a charm.

[dwpbike (dwpbike)]

my sandisk sansa clip won't mount, which it would do when this same computer was doing fc12. i received an error message that it couldn't mount sansa drive, so there was some intelligence displayed. now i get nothing. my ebay hong kong special mp3 player has no problem. guessing i need to do something to sansa, but what? all i want to do is trade my music out.

[jeremy-list (quick-dudley)]

I am also getting the same behaviour for normal partitioned disks

[jeremy-list (quick-dudley)]

same problem in Maverick (just upgraded)