lvm2 - Logical Volume Manager

cannot boot root on lvm2 with (largish) snapshot

Reported by Seth on 2009-04-12
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
lvm2
Invalid
Undecided
Unassigned
lvm2 (Ubuntu)
Undecided
Unassigned

Bug Description

When running root-on-lvm2 with the root being part of a vg that contains snapshotted volume(s), booting may fail if the snapshot size (or fill rate) grows over a certain size.

The kernel will only wait a certain amount of time until dropping to an initramfs-shell reporting 'Gave up waiting for root device' as the reason. The harddisk drive activity indicator will still show lots of disk access for some more time (minutes in out case).
'ps | grep lvm' reveals that 'lvchange -a y' is taking a long time to complete.
Waiting for the disk activity to die down, and exiting the shell will allow the boot to resume normally.

Can anybody explain *why* it could take that long - each and every time?
==================== DETAILS
More specificly, my volume group contained an Intrepid root partition of 20Gb (15Gb filled). I created a snapshot of this (18Gb) in order to 'sandbox upgrade' to Jaunty beta. This I did. I was kind of surprised by the volume of changes on the snapshot: the snapshot was 53% full after upgrade (so about 9Gb of changed blocks vs. Intrepid). On reboot, I found out that the system would not reboot itself.

I have spent a long time around various seemingly related bugs (#290153, #332270) for a cause until I found out the culprit myself. I have not been able to find any (lvm) documentation warning to the fact that lvm operations might take several minutes (?!) to complete on snapshotted volumes.

At the very least this warrants a CAPITAL RED warning flag in the docs, IMHO: using large snapshots might render a system unbootable (remotely) with root on lvm. Manual intervention at console/serial is required!

1. [untested] it doesn't matter whether the root volume is actually a snapshot or origin, as long as the volume group contains said snapshot (in my case, intrepid was on the origin and jaunty on tje snapshot; both systems failed boot with the same symptoms).
2. [untested] if root volume is in a separate vg things might work ok (assuming activating several volume groups can be done in parallel)
3. [partially tested: freshly snapshotted system booted ok] I suspect nothing is the matter with a snapshot sized 18Gb, unless it fills up ('exception blocks' get used). However there is not much use in a snapshot of 18Gb if you are only allowed to use small parts of it.
4. [tested] as a side note, once booted, both systems were reasonably performant (at least in responsiveness)
5. [tested] other volume groups in the same server did not suffer noticeable performance penalties even when the problematic one performed badly
6. [tested] the performance was back to normal (acceptable), even though my 40Gb Home lv still featured a (largely unaltered) snapshot of 5Gb/6% (in the _same_ volume group). This indicates that nothing special to be wrong/corrupted in the vg metadata.
7. [tested] rebooting/shutting down seemed flawed when initiated from Jaunty (but it seems unrelated to me: Jaunty appears to use kexec for reboot, causing problems with my terminal display and harddisk spin-down as well)

-------------- executed from the initramfs shell:
/sbin/lvm vgchange -a n
/sbin/lvm vgchange -a y # takes several minutes to complete (disk light on continuously)
/sbin/lvm vgchange -a n
/sbin/lvm vgchange -a y # again, same agony (continuous period of solid activity)

/sbin/lvm lvremove vg/largish_snapshot

/sbin/lvm vgchange -a n
/sbin/lvm vgchange -a y # only takes seconds

Of course I made a backup of the data in the snapshot that I actually wanted to keep :)

Seth (bugs-sehe) wrote :

it seems I had missed the selector for 'project' first time round

affects: ubuntu-release-notes → lvm2
Colin Watson (cjwatson) wrote :

Ubuntu bugs should generally be filed on the appropriate package in Ubuntu rather than on the upstream project. Moving the bug to Ubuntu's lvm2 package.

Changed in lvm2:
status: New → Invalid
Peter Valdemar Mørch (pmorch) wrote :

I've seen this exact behavior on two different sets of hardware. Both running LVM on top of RAID-1. I've seen it happen in both intrepid and jaunty

Andrew Berry (andrewberry) wrote :

I've run into this still with Ubuntu 10.04. The issue is exactly as described, where lvchange -a y takes significant time to run, causing the boot to fail. In my case, I had a 200G allocated snapshot against a 2TB LV, with about 40% in use.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in lvm2 (Ubuntu):
status: New → Confirmed
Milko Krachounov (exabyte) wrote :

I'm experiencing the same thing. With a full snapshot and a disabled splash screen, I get an error message telling me that a command has timeouted (lvscan, IIRC), and it had been terminated. I removed the snapshot and it booted all right.

I think terminating boot commands using a predefined timeout is a double-edged sword - it does help if a program trying to access an unessential resource has hanged, but it does prevent boots on slow systems - examples would include lvm snapshots which slow the boot process quite a lot, and failing hardware. I've had the same issue when I was trying to boot from a hard drive that was failing. The boot never succeeded because initrd decided to kill everything.

Wouldn't it make more sense if a more intelligent approach is used? E.g. run the tools in parallel repeatedly until the root is found, or re-run the killed ones if the root wasn't found or something?

Ray Van Dolson (rayvd) wrote :

I believe this is normal behavior for LVM. LVM has to build an overlay layer from all of the blocks contained in the snapshot so they are available via access to the "parent" LV at any time.

The larger the snapshot, the longer it takes to read in all of the blocks and build the overlay layer.

On 01/07/2012 08:08 PM, Ray Van Dolson wrote:
> I believe this is normal behavior for LVM. LVM has to build an overlay
> layer from all of the blocks contained in the snapshot so they are
> available via access to the "parent" LV at any time.
>
> The larger the snapshot, the longer it takes to read in all of the
> blocks and build the overlay layer.
does that imply that the overlay / COW bitmap is in-memory only? I don't
get that fully, because without persistent information, it couldn't be
reconstructed on reboot. So, the real problem is the absense of lazy
initialization of the overlay layer

Also, even if this was by design, in my mind it wouldbe broken by
design. Read the issue, you'll notice that this will prohibit use of
LVM2 snapshots on many systems.

Just my $0.02

Seth

Persistent on disk, but cached completely in memory by reading the entire COW
device when activating it.

Alasdair G. Kergon (agk2) wrote :

Metadata (not data) is cached i.e. the block mappings

stefan (sschloesser) wrote :

I just reproduced this on a remote server, I had a "natty" snapshot (am now on 11.10). After deleting the snapshot system booted just fine. This effectively kills the snapshot function for the rootfs.
Took me a few hours to get a remote console and figure this one out.

T T (netti+ubuntu) wrote :

Forgot about the problem when I upgraded to Precise 12.04 from oneiric. I have root on lvm and I took a snapshot before upgrade. Everthing worked fine for couple of boots and after _one_ day of using the machine My system timedout on watersched vgchange -a y, but the lvm volume was availeble in initrd shell.

Removing the snapshot made my system bootable again. A workaround for this problem whould be appreciated.

pcworld (pcworld) wrote :

@Anderson Luiz Alves: Are you sure this bug is an exact duplicate of #995645?

Anderson Luiz Alves (alacn1) wrote :

A lvm root volume snapshot that has many changes takes too long to activate (lvchange -ay volume). At boot time it doesnt wait and kill process "watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'", then it won't boot because it didn't activate the boot volume.
Its happening here, with volume vg0/trusty and snapshot vg0/trusty_snap.

Seth (bugs-sehe) wrote :

@Anderson The fact that "it's happening here" doesn't make this a dupe. In fact, it makes /that/ a dupe of this one, ~3 years OLDER.

I'd like to have the record show that this bug is ancient so we can measure the level of neglect.

$0.02

Anderson Luiz Alves (alacn1) wrote :

It has the symptom of both bugs, 1. a lvm root volume with snapshot that doesn't boot the OS, and 2. at boot time it stalls, show "timeout: killing 'watershed... lvm vgscan..." and drop to shell. After I removed the root snapshot it didn't show "killing watershed" and OS booted up fine.

Alasdair G. Kergon (agk2) wrote :

On Fri, Apr 11, 2014 at 06:55:15PM -0000, Seth wrote:
> I'd like to have the record show that this bug is ancient so we can
> measure the level of neglect.

Snapshot loading got speeded up in upstream kernel 3.14.

Alasdair

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers