When snapshots exists, mountall will not mount the parent partition

Bug #563902 reported by Alvin
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Release Notes for Ubuntu
Fix Released
High
Unassigned
lvm2 (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

Binary package hint: mountall

Steps to reproduce:

- Snapshot a volume (for example: /dev/vg0/home)
- reboot
- Mountall will not mount the volume
- Go into a recovery shell
- try # mount /volume
  mount: special device /dev/mapper/vg0-home does not exist
- try # lvremove /dev/mapper/vg0-home-snapshot
- mount /home (yes, now /dev/mapper/vg0-home will be seen)
- Exit the recovery shell to continue

When trying this with UUID entries in /etc/fstab, you could end up with the snapshot mounted instead of the parent partition. In that case, you can't remove the snapshot and have to reboot until the 'correct' volume can't be found anymore.

Revision history for this message
Thierry Carrez (ttx) wrote :

Might be linked to bug 561390 which shows more random LVM volume mounting failure cases.

Revision history for this message
Thierry Carrez (ttx) wrote :

Alvin: could you attach mountall --debug logs ?

This works for me:
Edit /etc/init/mountall.conf so that the mountall line reads:
exec mountall --debug --daemon $force_fsck $fsck_fix > /dev/mountall.log

Enter "M" mode:
cp /dev/mountall.log /dev/saved.log
fix it so that it boots... and attach saved.log here (should have Spawning... as the last line)

Revision history for this message
Steve Langasek (vorlon) wrote :

If you list your volumes by UUID in /etc/fstab instead of by volume name, mountall has no way to know which one is the right one to mount. LVs should always be listed in /etc/fstab by name which is unique; not by UUID, which may not be. All of the historical scripts in Ubuntu for transitioning /etc/fstab to UUID usage explicitly exclude LV names.

So I don't see that there's any bug here that can be fixed in mountall, though this seems like something we should document in the release notes.

Changed in mountall (Ubuntu):
status: New → Invalid
Changed in ubuntu-release-notes:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Alvin (alvind) wrote :

@Thierry Carrez
I'm sorry. I will visit the location of the Lucid machines in 4 days. Due to the nature of the bug, trying this with remote access is impossible for now.

@Steve Langasek
Maybe I wasn't clear in my description. This will also happen if you use /dev/mapper entries! The bug is in fact, if a snapshot exists, mount problems will occur (with message exactly like the problems in bug 527666, bug 561390 and bug 557909). UUID only makes this bug harder to reproduce, because bug 563117 will arise. The do-release-upgrade process to Lucid changes /dev/mapper entries to UUID.

Another issue that can arise when taking a snapshot is described in bug 563895 (/dev/mapper entries on fresh install)
Because I wasn't sure whether these were related, I filed them separately.

Revision history for this message
Steve Langasek (vorlon) wrote :

Alvin,

Sorry, I guess I missed that your example shows 'mount' failing to mount /dev/mapper/vg0-home.

When mount gives this error, *does* /dev/mapper/vg0-home exist on the filesystem?

What does the fstab entry for this look like?

Changed in mountall (Ubuntu):
status: Invalid → New
Revision history for this message
Alvin (alvind) wrote :

I can't tell you that. It's possible that /dev/mapper/vg0-home does not exist until after the snapshot is removed. The machine is on another locaten, so I can't see plymouth stuff during reboot) If I remember correctly, ls /dev/mapper does not show vg0-home, but I could be wrong there.

From /etc/fstab:
/dev/mapper/vg0-home /home /ext4 noatime 0 2

Revision history for this message
Scott James Remnant (Canonical) (canonical-scott) wrote :

I don't see any evidence of a mountall bug here - this sounds like an lvm2 issue

affects: mountall (Ubuntu) → lvm2 (Ubuntu)
Revision history for this message
Steve Langasek (vorlon) wrote :

I can't reproduce this problem. I've created an lvm snapshot of my /home partition, and /home is still mounted correctly on reboot. When you can get in front of this machine again, please verify whether the expected entries are present under /dev/mapper, and also grab the output of 'lvdisplay'.

Changed in lvm2 (Ubuntu):
status: New → Incomplete
Revision history for this message
Alvin (alvind) wrote :

I have 2 snapshots of /home (see bug 563117 about this being intentional)
Currently using UUID

On boot, plymouth says:
"Serious errors were found while checking the disk drive for /home"

# mount /home
mount: /dev/mapper/vg0-home-udsm1004200000 already mounted or /home busy
mount: according to mtab, /dev/mapper/vg0-home--udsm1004200000 is mounted on /srv/udsm/home-udsm1004200000

The last part is correct. That snapshot is mounted there, but should not be mounted on /home
- df does not show anything mounted on /home
- /home is not listed in /proc/mounts

At this point, /dev/mapper/vg0-home does exist (but with wrong permissions)
Also, the output of lvdisplay shows nothing wrong. /dev/vg0/home is there and is the source of 2 active snapshots.

I then proceed to removing all snapshots
# mount /home
Now, suddenly this works

Revision history for this message
Alvin (alvind) wrote :
Revision history for this message
Steve Langasek (vorlon) wrote :

> Currently using UUID

Why are you doing this? As previously stated, it's an error to try to use UUIDs in /etc/fstab for LVs, as UUID is not guaranteed to be unique in the face of snapshotting. The only reason this bug is not cloved as 'invalid' is because you said it occurs even when using LV names in /etc/fstab.

> At this point, /dev/mapper/vg0-home does exist (but with wrong permissions)

What's wrong about the permissions?

> Also, the output of lvdisplay shows nothing wrong. /dev/vg0/home is there
> and is the source of 2 active snapshots.

I asked for a copy of the lvdisplay output, please.

Revision history for this message
Alvin (alvind) wrote :

I was using UUID because Scott James Remnant wrote in bug 563117 that this was intentional.

When I change all LVM entries to /dev/mapper other bugs get in the way of testing. I just did that and also removed all snapshots. What happens then is that /opt can't be mounted (/dev/mapper/vg0-opt) # mount /opt in the recovery shell works.
Ownership permissions of /dev/mapper/vg0-opt are root:disk
Ownership of all other /dev/mapper entries are root:root

What is wrong about the permissions (UUID or otherwise) is described in bug 557909 (and a bit of bug 527666)

lvdisplay output of vg0/home is attached. Currently it does no longer shows the snapshots because I had to remove them in order to boot. If they are there, there's just a line extra that says 'source of <name-of-snapshot> [active]'

Revision history for this message
Steve Langasek (vorlon) wrote :

It may or may not have been /intentional/, but it's clearly a /bug/ to convert /dev/mapper entries to UUID; and as soon as we can identify the code responsible for such migration (which we haven't yet), that should be fixed.

> When I change all LVM entries to /dev/mapper other bugs get in the
> way of testing. I just did that and also removed all snapshots. What
> happens then is that /opt can't be mounted (/dev/mapper/vg0-opt)

If you convert all your fstab entries to /dev/mapper *except* for the vg0-opt entry that's currently giving you problems, and you create snapshots, does the boot work as expected?

> Ownership permissions of /dev/mapper/vg0-opt are root:disk

I don't see anything wrong with that; this is the default group ownership set by udev for block devices, and I have entries under /dev/mapper here that are owner root:disk. (I also have other devices that are owner root:root; the difference appears to be whether the devices were activated in the initramfs or in the root system. The devmapper device here that's owned by root:disk is an encrypted test partition that is only decrypted after chrooting to the rootfs.)

Do you know of any reason why vg0-opt would be activated after the end of your initramfs instead of at the same time ? Can you attach a /var/log/udev.log for this system?

Revision history for this message
Alvin (alvind) wrote : Re: [Bug 563902] Re: When snapshots exists, mountall will not mount the parent partition
  • udev Edit (184.0 KiB, text/plain; charset="ISO-8859-1"; name="udev")

On Tuesday 20 April 2010 11:07:59 Steve Langasek wrote:
> If you convert all your fstab entries to /dev/mapper *except* for the
> vg0-opt entry that's currently giving you problems, and you create
> snapshots, does the boot work as expected?

Weird as it may sound, but it does.
Also, the permissions are now root:root

crw-rw---- 1 root root 10, 59 Apr 20 12:46 control
brw-rw---- 1 root root 252, 1 Apr 20 12:46 vg0-home
brw-rw---- 1 root root 252, 4 Apr 20 12:46 vg0-images
brw-rw---- 1 root root 252, 3 Apr 20 12:46 vg0-opt
brw-rw---- 1 root root 252, 0 Apr 20 12:46 vg0-root
brw-rw---- 1 root root 252, 2 Apr 20 12:46 vg0-tmp

> Do you know of any reason why vg0-opt would be activated after the end
> of your initramfs instead of at the same time ? Can you attach a
> /var/log/udev.log for this system?

No idea. That part of /etc/fstab looks like:
/dev/mapper/vg0-root / ext4 noatime,errors=remount-ro 0 1
UUID=fd383da0-11be-43f0-86ce-81cfa996b085 /boot ext2 defaults 0 2
/dev/mapper/vg0-home /home ext4 noatime 0 2
UUID=2f0f6020-2dba-4205-92d2-f46e156a5a71 /opt ext4 noatime 0 2
/dev/mapper/vg0-images /srv/images ext4 noatime 0 2
none /tmp tmpfs defaults 0 0
UUID=349d41aa-825c-4941-96bd-7caca7d9ac49 none swap sw 0 0

Revision history for this message
Steve Langasek (vorlon) wrote :

That doesn't sound weird to me at all, it's exactly what I expected to hear based on my own testing here. :)

So if /dev/mapper devices get mounted correctly in the presence of snapshots, I think we can conclude that there's no bug in lvm2/mountall wrt snapshots. There seems to be a bug preventing /dev/mapper/vg0-opt from working for you, but that's unrelated to snapshotting and seems to be tracked as bug #557909; there's possibly a bug in some unidentified component that wrongly changes /dev/mapper entries to UUID in /etc/fstab on upgrade; and there's a need to document in the release notes that you shouldn't use UUID= for lvs in /etc/fstab.

Have I covered everything?

Changed in lvm2 (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Steve Langasek (vorlon) wrote :

btw, the udev log you attached shows no record of any of the LVs - which is consistent with them all having been set up during the initramfs, and consistent with them all having group root ownership (it seems).

Revision history for this message
Alvin (alvind) wrote :

On Wednesday 21 April 2010 03:15:32 Steve Langasek wrote:
> That doesn't sound weird to me at all, it's exactly what I expected to
> hear based on my own testing here. :)

No, it's still weird. I can only boot with a mix of UUID and /dev/mapper. For
the moment, let's call that reliable.

> So if /dev/mapper devices get mounted correctly in the presence of
> snapshots, I think we can conclude that there's no bug in lvm2/mountall
> wrt snapshots.

No, they don't. Without snapshots (and with the mix of UUID and /dev/mapper) I
can now boot, but I just tried this with snapshots.
- With a snapshot of vg0/home, /home can not be mounted
- With a snapshots of vg0/images, /srv/images can not be mounted

> There seems to be a bug preventing /dev/mapper/vg0-opt
> from working for you, but that's unrelated to snapshotting and seems to
> be tracked as bug #557909.

Yes. I must add that the ownership of the unmountable volumes 'root:disk' is
when a snapshot of that volume exists. So, maybe fixing bug 557909 (or bug
561390
) might also fix this one.

> there's possibly a bug in some unidentified
> component that wrongly changes /dev/mapper entries to UUID in /etc/fstab
> on upgrade; and there's a need to document in the release notes that you
> shouldn't use UUID= for lvs in /etc/fstab.

Exactly.

> Have I covered everything?

Almost. I'm experiencing bug 563895. Same cause as this bug, different
situation with mdadm. I'm only mentioning this because they might be related
somehow. (Could be wrong. I reported that bug against grub.)

Revision history for this message
Steve Langasek (vorlon) wrote :

Documented in the release notes at <https://wiki.ubuntu.com/LucidLynx/ReleaseNotes#LVM%20filesystems%20should%20be%20listed%20in%20/etc/fstab%20by%20name>:

In general, filesystems are listed in /etc/fstab by UUID rather than by device name, to ensure that the filesystem can always be found reliably. If you are mounting a filesystem located on LVM, however, it is recommended that you list them in /etc/fstab by device name, not by UUID, because UUIDs are not unique if LVM snapshots are used, which can result in wrong filesystems being mounted at boot. (563902)

Changed in ubuntu-release-notes:
status: Triaged → Fix Released
Revision history for this message
gigastress (a9711599) wrote :

Hey guys,

I just proceeded from 9.10 to 10.4 LTS lucid, after not upgrading for 2.5 years, I was about to proceed to the latest version.
I decided to keep the old grub, for I made a whole-disk-encryprtion and wanted to keep the settings. I dont know if this is the problem with lvm problem adressed in the lucid release notes. I think I installed the new kernel but Im not sure here.
Anyway the result of this installation is, that
When I boot, the display where Im supposed to put in the LVM/luks password is showing up, but when I put it in, it keeps saying - cryptsetup failed, bad password or options -
It is the correct pass, the release note said sth. about changing the UUID in the fstab with the actual name of the hd, but since the display where Im supposed to put in the LVM/luks password is showing up at boot, I dont know if this problem is related.
so it actually could be a kernel, fstab, or grub problem.
I got know idea how to handle this.
suggestions would be highly appreciated.
anyone, how can I proceed ???

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.