Silent wraparound on > 2 TB LVM snapshots in lucid and karmic

[Phillip Susi]

Ubuntu kernels silently wrap access to disk locations above 2 TB back around to zero. This can easily be reproduced using lvm to create a thin provisioned virtual disk:

lvcreate -s -n thin -L 1g --virtualsize 3t vg0

mke2fs -t ext4 -E lazy_itable_init /dev/vg0/thin
e2fsck -f /dev/vg0/thin

Fsck will find errors in the bitmap because it is actually reading the superblock instead of the allocation bitmap situated just after the 2 tb mark.

This affects Karmic and Lucid and will cause data loss and severe filesystem corruption that may go unnoticed for some time, if someone creates a snapshot of a 2TB + size logical volume. It was fixed upstream as of 2.6.33.

See comment #26 for details on the problem and what needs done to fix it.

[Phillip Susi (psusi)]

Seems CONFIG_LBDAF was barking up the wrong tree. It is set on the i386 build, and does not apply to amd64.

[Phillip Susi (psusi)]

Narrowed it down to lvm snapshots. Using dmsetup to create a linear mapping greater than 2tb has no problem reading and writing over the 2tb mark correctly, so it appears that the problem is in dm-snapshot.

[Jeremy Foshee (jeremyfoshee)]

Hi Phillip,

Please be sure to confirm this issue exists with the latest development release of Ubuntu. ISO CD images are available from http://cdimage.ubuntu.com/daily-live/current/ . If the issue remains, please run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 593086

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Jeremy Foshee (jeremyfoshee)]

This bug report was marked as Incomplete and has not had any updated comments for quite some time. As a result this bug is being closed. Please reopen if this is still an issue in the current Ubuntu release http://www.ubuntu.com/getubuntu/download . Also, please be sure to provide any requested information that may have been missing. To reopen the bug, click on the current status under the Status column and change the status back to "New". Thanks.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

apport information

[Phillip Susi (psusi)]

Quite some time? It's been 13 hours man... testing upstream kernel now...

[Phillip Susi (psusi)]

linux-image-2.6.35-999-generic_2.6.35-999.201006111153_amd64.deb does not seem to have this problem.

[Phillip Susi (psusi)]

2.6.35-2 currently in use on maverick also does not appear to have the problem.

[Phillip Susi (psusi)]

Also tested linux-image-2.6.32-020632-generic_2.6.32-020632_amd64.deb, which was the closest mainline build I could see to the 2.6.32-22 build lucid is using, and it is also affected, so it looks like this was an upstream bug that got fixed somewhere between 2.6.32 and 2.6.35.

[Phillip Susi (psusi)]

2.6.33 also is unaffected.

[Phillip Susi (psusi)]

It looks like it was fixed in mainline between 2.6.32 and 2.6.33. I can find no commits that sound intended to fix this, but there were a number of changes to the snapshot code, so I guess one of them inadvertently fixed this.

[Phillip Susi (psusi)]

After discussing it on <email address hidden>, I believe the issue has been found. There was a patch in 2.6.32 that caused this breakage, and was later fixed. The broken patch ended up in Ubuntu Lucid and Karmic kernels. This probably should be corrected with an SRU. From the mailing list discussion Mikulas Patocka wrote:

The bug existed even in upstream, but only in 2.6.32 kernel. The reason
was this function:
static inline chunk_t sector_to_chunk(struct dm_exception_store *store,
                                      sector_t sector)
{
        return (sector & ~store->chunk_mask) >> store->chunk_shift;
}

"store->chunk_mask" was changed to be unsigned in 2.6.32, so it was
masking the sector with 32-bit value. In 2.6.33 that masking was removed.
Ubuntu picked that 2.6.32 patch but didn't pick further patches.

[Jeremy Foshee (jeremyfoshee)]

Thanks for your testing Phillip. My apologies for the script bug that hit you. We are fixing that now.

~JFo

[Toby Hunt (tjeh)]

Just to clarify: Does this bug affect just snapshots of LVM logical volumes, or all logical volumes managed by LVM?

[Phillip Susi (psusi)]

Now that I look at it again, it looks like this only affects snapshots.

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".