device mappings for partitions not removed after build using --raw, leading to filesystem corruption
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| vm-builder (Ubuntu) |
Undecided
|
Unassigned | ||
| Lucid |
Undecided
|
Unassigned |
Bug Description
SRU Request for Lucid
[Impact] This bug occurs systematically when the --raw option is used to create a VM. This results in file system data corruption messages in the VM created.
[Development/Stable Fix] Fixed in 0.12.4+
[ Louis Bouchard ]
* Remove dev maps created by parted. (LP: #531599)
[Test Case] section with detailed instructions how to reproduce the bug. These should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem.
On an unmodified system with an existging 8Gb LVM volume named /dev/sysvg/
ubuntu-vm-builder kvm natty --lang=en_US --hostname=testvm --rootsize=8000 --mem=10240 --cpus=2 --rootpass=X --arch=amd64 --libvirt=
On an unmodified system, once completed, the dmsetup list command will show the following :
$ dmsetup ls
sysvg-testvmvol1p1 (252, 2)
sysvg-testvmvol1 (252, 1)
sysvg-testvmvol (252, 0)
The sysvg-testvmvol1p1 table should not be there
[Regression Potential] This fix needs to be backported to the LTS version to avoid potential file system corruption.
---
I've tried to install a vm using an LVM logical volume with --raw. The problem was that I used a too small logical volume.
vmbuilder told me:
2010-03-03 21:33:59,923 INFO : Adding partition table to disk image: /dev/mapper/
2010-03-03 21:34:00,007 INFO : Adding type 1 partition to disk image: /dev/mapper/
2010-03-03 21:34:00,879 INFO : Adding type 3 partition to disk image: /dev/mapper/
2010-03-03 21:34:00,890 INFO : [0] ../../libparted
2010-03-03 21:34:00,892 INFO : Cleaning up
Traceback (most recent call last):
File "/usr/bin/
VMBuilder.run()
File "/usr/lib/
frontend.run()
File "/usr/lib/
vm.create()
File "/usr/lib/
disk.
File "/usr/lib/
disk.
File "/usr/lib/
part.
File "/usr/lib/
run_
File "/usr/lib/
raise VMBuilderException, "Process (%s) returned %d. stdout: %s, stderr: %s" % (args.__repr__(), status, stdout, stderr)
VMBuilder.
, stderr: [0] ../../libparted
While this is a lot of noise, telling me 'location 1032 is outside of the device...' made me recognize my mistake.
The problem is that I wasn't able to just remove the logical volume, create a bigger one and start again:
sudo lvremove /dev/mapper/
Can't remove open logical volume "test-disk"
The reason is that there's another device map set up by vmbuilder: /dev/mapper/
This is because the first (root) partition did fit on the logical volume, but only the second (swap) one not.
I had to drop the device mapping using
sudo dmsetup remove /dev/mapper/
This would allow me to lvremove the logical volume now.
I would expect that vmbuilder cleans up on exceptions like this and remove the device mappings for the inner partitions.
Related branches
- Serge Hallyn: Needs Resubmitting on 2011-12-22
- Scott Moser: Pending requested 2011-12-21
-
Diff: 6519 lines (+6029/-0) (has conflicts)91 files modified.bzrignore (+11/-0)
AUTHORS (+29/-0)
MANIFEST.in (+7/-0)
Makefile (+19/-0)
NEWS (+58/-0)
VMBuilder/__init__.py (+135/-0)
VMBuilder/contrib/cli.py (+450/-0)
VMBuilder/disk.py (+549/-0)
VMBuilder/distro.py (+99/-0)
VMBuilder/exception.py (+27/-0)
VMBuilder/frontend.py (+53/-0)
VMBuilder/hypervisor.py (+121/-0)
VMBuilder/log.py (+36/-0)
VMBuilder/plugins/__init__.py (+329/-0)
VMBuilder/plugins/ec2/__init__.py (+150/-0)
VMBuilder/plugins/ec2/templates/ec2_version.tmpl (+1/-0)
VMBuilder/plugins/ec2/templates/landscape_client.tmpl (+1/-0)
VMBuilder/plugins/ec2/templates/sshd_config.tmpl (+77/-0)
VMBuilder/plugins/ec2/templates/sudoers.tmpl (+29/-0)
VMBuilder/plugins/firstscripts/__init__.py (+65/-0)
VMBuilder/plugins/firstscripts/templates/firstbootrc.tmpl (+10/-0)
VMBuilder/plugins/firstscripts/templates/firstloginrc.tmpl (+10/-0)
VMBuilder/plugins/kvm/__init__.py (+19/-0)
VMBuilder/plugins/kvm/vm.py (+71/-0)
VMBuilder/plugins/libvirt/__init__.py (+87/-0)
VMBuilder/plugins/libvirt/templates/libvirtxml.tmpl (+46/-0)
VMBuilder/plugins/libvirt/templates/libvirtxml_fsimage.tmpl (+33/-0)
VMBuilder/plugins/network/__init__.py (+158/-0)
VMBuilder/plugins/postinst/__init__.py (+77/-0)
VMBuilder/plugins/ubuntu/__init__.py (+19/-0)
VMBuilder/plugins/ubuntu/dapper.py (+375/-0)
VMBuilder/plugins/ubuntu/distro.py (+294/-0)
VMBuilder/plugins/ubuntu/edgy.py (+84/-0)
VMBuilder/plugins/ubuntu/feisty.py (+26/-0)
VMBuilder/plugins/ubuntu/gutsy.py (+26/-0)
VMBuilder/plugins/ubuntu/hardy.py (+58/-0)
VMBuilder/plugins/ubuntu/intrepid.py (+45/-0)
VMBuilder/plugins/ubuntu/jaunty.py (+40/-0)
VMBuilder/plugins/ubuntu/karmic.py (+33/-0)
VMBuilder/plugins/ubuntu/lucid.py (+23/-0)
VMBuilder/plugins/ubuntu/maverick.py (+23/-0)
VMBuilder/plugins/ubuntu/natty.py (+22/-0)
VMBuilder/plugins/ubuntu/oneiric.py (+28/-0)
VMBuilder/plugins/ubuntu/precise.py (+23/-0)
VMBuilder/plugins/ubuntu/suite.py (+28/-0)
VMBuilder/plugins/ubuntu/templates/51_update-motd-hardy.tmpl (+10/-0)
VMBuilder/plugins/ubuntu/templates/51_update-motd.tmpl (+10/-0)
VMBuilder/plugins/ubuntu/templates/dapper_fstab.tmpl (+10/-0)
VMBuilder/plugins/ubuntu/templates/dapper_fstab_fsimage.tmpl (+10/-0)
VMBuilder/plugins/ubuntu/templates/devicemap.tmpl (+3/-0)
VMBuilder/plugins/ubuntu/templates/etc_hosts.tmpl (+10/-0)
VMBuilder/plugins/ubuntu/templates/initctl-stub.tmpl (+3/-0)
VMBuilder/plugins/ubuntu/templates/interfaces.tmpl (+22/-0)
VMBuilder/plugins/ubuntu/templates/is-compat-env.tmpl (+3/-0)
VMBuilder/plugins/ubuntu/templates/kernelimg.tmpl (+8/-0)
VMBuilder/plugins/ubuntu/templates/locale.tmpl (+1/-0)
VMBuilder/plugins/ubuntu/templates/nostart-policy-rc.d.tmpl (+18/-0)
VMBuilder/plugins/ubuntu/templates/sources.list.tmpl (+15/-0)
VMBuilder/plugins/ubuntu/templates/sudoers.tmpl (+23/-0)
VMBuilder/plugins/ubuntu/templates/timezone.tmpl (+1/-0)
VMBuilder/plugins/ubuntu/templates/upstart.tmpl (+16/-0)
VMBuilder/plugins/ubuntu/templates/xen-ld-so-conf.tmpl (+1/-0)
VMBuilder/plugins/ubuntu/tests/test_distro.py (+14/-0)
VMBuilder/plugins/virtualbox/__init__.py (+19/-0)
VMBuilder/plugins/virtualbox/templates/vm_deploy_script.tmpl (+65/-0)
VMBuilder/plugins/virtualbox/vm.py (+59/-0)
VMBuilder/plugins/vmware/__init__.py (+19/-0)
VMBuilder/plugins/vmware/templates/esxi.vmx.tmpl (+57/-0)
VMBuilder/plugins/vmware/templates/flat.vmdk.tmpl (+19/-0)
VMBuilder/plugins/vmware/templates/vmware.tmpl (+30/-0)
VMBuilder/plugins/vmware/vm.py (+128/-0)
VMBuilder/plugins/xen/__init__.py (+20/-0)
VMBuilder/plugins/xen/vm.py (+89/-0)
VMBuilder/tests/__init__.py (+17/-0)
VMBuilder/tests/base_test.py (+67/-0)
VMBuilder/tests/disk_tests.py (+289/-0)
VMBuilder/tests/network_tests.py (+95/-0)
VMBuilder/tests/plugin_tests.py (+119/-0)
VMBuilder/tests/ubuntu_tests.py (+39/-0)
VMBuilder/tests/util_tests.py (+10/-0)
VMBuilder/util.py (+224/-0)
VMBuilder/vm.py (+257/-0)
examples/ec2-amd64-part-file.txt (+2/-0)
examples/ec2-firstboot.sh (+20/-0)
examples/ec2-firstlogin.sh (+49/-0)
examples/ec2-i386-part-file.txt (+3/-0)
setup.py (+26/-0)
ubuntu-vm-builder (+24/-0)
ubuntu-vm-builder.1 (+7/-0)
vmbuilder (+24/-0)
vmbuilder.1 (+170/-0)
Soren Hansen (soren) wrote : Re: [Bug 531599] [NEW] tricky leftovers when using a too small --raw device | #1 |
Changed in vm-builder (Ubuntu): | |
status: | New → Incomplete |
We'd like to figure out what's causing this bug for you, but we haven't heard back from you in a while. Could you please provide the requested information? Thanks!
Simon Huerlimann (huerlisi) wrote : | #3 |
I tried with the current version in Lucid (0.12.3.r435), error still present with similar output.
Changed in vm-builder (Ubuntu): | |
status: | Incomplete → Confirmed |
Simon Huerlimann (huerlisi) wrote : | #4 |
Doing this on a Lucid with vmbuilder 0.12.3.r435.
After some testing I found out that this is not restricted to failed/interrupted vmbuilder runs. Additionally leaving behind those device mappings seem to interfere with the filesystem in the VM.
If you use a LVM logical volume as --raw device, a partition table is created on the device and partitions are setup. For those partitions device mapper is configured. Example for --raw /dev/leo02/
brw-rw---- 1 shuerlimann shuerlimann 251, 8 2010-05-01 07:17 leo02-kvm-
brw-rw---- 1 root disk 251, 9 2010-04-22 16:44 leo02-kvm-
brw-rw---- 1 root disk 251, 10 2010-04-22 16:44 leo02-kvm-
As far as I can tell, no such leftovers exist if qcow images are used...
If you start a vm created using --raw and having those device mappings still configured, filesystem corruption in the vm happens. This is quite reliably reproducable. Here's a snipplet of the /var/log/kernel.log right after booting:
Apr 30 12:47:11 web01 kernel: [ 0.357236] Freeing unused kernel memory: 796k freed
Apr 30 12:47:11 web01 kernel: [ 0.357881] Write protecting the kernel read-only data: 7788k
Apr 30 12:47:11 web01 kernel: [ 0.380691] udev: starting version 151
Apr 30 12:47:11 web01 kernel: [ 0.540618] FDC 0 is a S82078B
Apr 30 12:47:11 web01 kernel: [ 0.586564] EXT4-fs (sda1): INFO: recovery required on readonly filesystem
Apr 30 12:47:11 web01 kernel: [ 0.586573] EXT4-fs (sda1): write access will be enabled during recovery
Apr 30 12:47:11 web01 kernel: [ 0.603151] EXT4-fs (sda1): recovery complete
Apr 30 12:47:11 web01 kernel: [ 0.651570] EXT4-fs (sda1): mounted filesystem with ordered data mode
Apr 30 12:47:11 web01 kernel: [ 1.476513] EXT4-fs error (device sda1): ext4_mb_
Apr 30 12:47:11 web01 kernel: [ 1.477954] JBD: Spotted dirty metadata buffer (dev = sda1, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
Apr 30 12:47:11 web01 kernel: [ 1.532262] udev: starting version 151
Apr 30 12:47:11 web01 kernel: [ 1.721612] vga16fb: initializing
Apr 30 12:47:11 web01 kernel: [ 1.721620] vga16fb: mapped to 0xffff8800000a0000
Apr 30 12:47:11 web01 kernel: [ 1.721680] fb0: VGA16 VGA frame buffer device
Apr 30 12:47:11 web01 kernel: [ 1.904983] Adding 999416k swap on /dev/sda2. Priority:-1 extents:1 across:999416k
Apr 30 12:47:11 web01 kernel: [ 2.241484] input: ImExPS/2 Generic Explorer Mouse as /devices/
Apr 30 12:47:11 web01 kernel: [ 3.210207] Console: switching to colour frame buffer device 80x30
Apr 30 12:47:11 web01 kernel: [ 4.219962] EXT4-fs error (device sda1): ext4_mb_
Apr 30 12:47:11 web01 kernel: [ 4.247451] EXT4-fs error (device sda1): ext4_mb_
Apr 30 12:47:23 web01 kernel: [ 13.720135] eth0: no IPv6 routers present
Apr 30 12:47:58 web01 kernel: [ 49.482760] EXT4-fs error (device sda1): ext4_mb_
summary: |
- tricky leftovers when using a too small --raw device + device mappings for partition not removed after build using --raw, + leading to filesystem corruption |
summary: |
- device mappings for partition not removed after build using --raw, + device mappings for partitions not removed after build using --raw, leading to filesystem corruption |
Simon Huerlimann (huerlisi) wrote : | #5 |
After some investigation it looks like vmbuilder is nicely registering a callback hook to unmap the devices it sets up using kpartx. The problem is udev rules adding device mappings using kpartx as soon as the raw device gets partitioned.
AFAIK the relevant udev rule is /lib/udev/
The kpartx call in vmbuilder does use the scheme /dev/mapper/
Hurl (hurl) wrote : | #6 |
I can confirm this bug too, using vmbuilder 0.12.4 on an up to date lucid.
Vmbuilder complete guest creation without an error. But launching the guest with the mapping cause immediate fs corruption.
This has nothing to do with the lv size.
Simon have done all the investigation correctly but I don't how to fix that.
The only solution is to not forget to remove this mapping before launchping the guest.
I would appreciate if this could be fixed.
Regards
Louis Bouchard (louis) wrote : | #7 |
After doing some test to identify what is creating this duplicate set of dm maps, it seems to be the following 'parted mkpart' commands that triggers the udev rule :
From /usr/share/
def create(self, disk):
"""Adds partition to the disk image (does not mkfs or anything like that)"""
running the equivalent command while watching the dm tables does show a new table entry when the 'parted --script {dev} mkpart primary ext2 {beginning} {size}' command is invoked.
Currently, it is unclear to me if the removal of those unnecessary dm entries should be done by a callback from create() or included in the existing unmap() callback for map_partitions.
Louis Bouchard (louis) wrote : | #8 |
I have been able to test a potential fix for this problem. The unmap() function is responsible for tearing apart the device maps created by the kpartx commands. Those are created by the map_partitions() function.
While tearing down the device maps created by the parted command, which is invoked in the partition() function might be better done by a callback in the partition() function, the patch is simpler by adding to the existing unmap() which is a callback from map_partition, which is invoked sequentially after partition() in hypervisor.py.
Please let me know if the patch in the proposed branch is unacceptable.
Launchpad Janitor (janitor) wrote : | #9 |
This bug was fixed in the package vm-builder - 0.12.4+
---------------
vm-builder (0.12.4+
[ Louis Bouchard ]
* Remove dev maps created by parted. (LP: #531599)
[ Serge Hallyn ]
* pre-hook to create vcsversion.py is failing, work around it.
-- Serge Hallyn <email address hidden> Thu, 22 Dec 2011 09:18:12 -0600
Changed in vm-builder (Ubuntu): | |
status: | Confirmed → Fix Released |
Simon Chan (simonchan) wrote : | #10 |
The new fix 0.12.4+
Repoduce the Bug : Use vmbuilder to create VM, execution would terminate right at "Unmounting target filesystem".
An error occurs when the system tries to unmount /dev/mapper/
It seems to be a typo, it should be /dev/mapper/loop0p1 (Only one 'p').
Rollbacked to 0.12.4+
Louis Bouchard (louis) wrote : | #11 |
@Simon
Could you please provide an example of the vmbuilder command that you used to generate the problem, so I can test with it ?
Kind regards
Louis Bouchard (louis) wrote : | #12 |
@Simon
Nevermind the previous request, I was able to reproduce the bug myself. As the issue fixed by this patch only happens when --raw is used, the modification is not restrictive enough and try to remove nonexistent maps. I'll work on refining the patch.
no longer affects: | vmbuilder |
description: | updated |
Micah Gersten (micahg) wrote : | #13 |
Unsubscribing sponsors as I don't see a patch for lucid, please resubscribe when a patch or merge proposal is ready
Louis Bouchard (louis) wrote : | #14 |
@micah
I'm sorry, I thought I had done so. Here is the debdiff against the ubuntu version. Let me know if this is sufficient.
Stefano Rivera (stefanor) wrote : | #15 |
Corrected the bug number in the changelog entry, retarrgeted the upload at lucid-proposed, and uploaded.
It's now waiting for SRU team review.
Changed in vm-builder (Ubuntu Lucid): | |
status: | New → Fix Committed |
Hello Simon, or anyone else affected,
Accepted vm-builder into lucid-proposed. The package will build now and be available at http://
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-
Further information regarding the verification process can be found at https:/
tags: | added: verification-needed |
Louis Bouchard (louis) wrote : | #17 |
Changed tag to verification-done. tested in a Lucid environment with initial reproducer and it no longer leaves a lingering dm map. Behavior is as expected.
tags: |
added: verification-done removed: verification-needed |
Launchpad Janitor (janitor) wrote : | #18 |
This bug was fixed in the package vm-builder - 0.12.4-0ubuntu0.3
---------------
vm-builder (0.12.4-0ubuntu0.3) lucid-proposed; urgency=low
* debian/
LP: #531599 which removes dev maps created by parted.
-- Louis Bouchard <email address hidden> Mon, 04 Jun 2012 12:33:21 +0100
Changed in vm-builder (Ubuntu Lucid): | |
status: | Fix Committed → Fix Released |
On Wed, Mar 03, 2010 at 09:06:21PM -0000, Simon Huerlimann wrote:
> Public bug reported:
>
> I've tried to install a vm using an LVM logical volume with --raw. The
> problem was that I used a too small logical volume.
Which version of VMBuilder are you using?
If you use "ubuntu-bug python-vm-builder" to report the bug, all this
information gets included automatically.
status incomplete
-- www.ubuntu. com/
Soren Hansen
Ubuntu Developer
http://