Kernel oops while localedef runs via debootstrap

[Marcus Grieger]

I'm not sure if this is the correct package, however, it's related to xen in general and linux-2.6.24-22-xen.

Ubuntu version is Ubuntu Hardy beta as of today.

Circumstances are a bit complicated:

The base system is an amd64 dual-core opteron running 2.6.24-12-xen on top of xen-3.2.

I'm working on a script to automatically install different guests via template configurations. The domUs are running 2.6.22-14-xen from gutsy, because networking with the hardy xen kernel in domU is currently not working (https://bugs.launchpad.net/bugs/204010).

That script basically sets up one or more LVM partitions, mounts them, debootstraps hardy with some predefined additional packages, runs a custom chrooted postinst-script, unmounts and then starts up the VM.

Right after a reboot, this works without problems. However, when I shutdown the guest, remove the LVM volumes and run the script again, I'll get the following kernel Oops when localedef is started from debootstrap on configuration of the locales package.

I can also install two almost identical guests, one after another, after a reboot without a problem.

At last, I tried rebooting the system, starting up a guest VM, shutting it down again and then reinstalling it. No oops.

Plan for tomorrow: Test the script again with dom0 running the gutsy xen kernel, too.

------------------------ snip
[ 646.675038] Unable to handle kernel paging request at ffff8800741d5008 RIP:
[ 646.675127] [<ffffffff80271e36>] iov_iter_advance+0x66/0x80
[ 646.675201] PGD 1fe5067 PUD 21e7067 PMD 2388067 PTE 0
[ 646.675251] Oops: 0000 [1] SMP
[ 646.675289] CPU 0
[ 646.675321] Modules linked in: xt_physdev iptable_filter ip_tables x_tables bridge netloop blktap nbd video output sbs sbshc dock battery container ac blkbk netbk xenbus_be psmouse parport_pc lp parport ipv6 sr_mod cdrom button i2c_piix4 evdev k8temp i2c_core shpchp pci_hotplug pcspkr ext3 jbd mbcache pata_serverworks sg sd_mod ohci_hcd pata_acpi ehci_hcd tg3 sata_promise ata_generic usbcore ssb libata scsi_mod raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod dm_mirror dm_snapshot dm_mod thermal processor fan xennet xenblk
[ 646.675844] Pid: 631, comm: localedef Not tainted 2.6.24-12-xen #1
[ 646.675882] RIP: e030:[<ffffffff80271e36>] [<ffffffff80271e36>] iov_iter_advance+0x66/0x80
[ 646.675946] RSP: e02b:ffff880059d83b10 EFLAGS: 00010246
[ 646.677898] RAX: 0000000000000000 RBX: 000000000000007f RCX: 0000000000000000
[ 646.677938] RDX: 0000000000000000 RSI: 000000000000007f RDI: ffff880059d83ba8
[ 646.677977] RBP: 000000000000007f R08: 0000000000000000 R09: 0000000000000000
[ 646.678016] R10: ffff8800741d5000 R11: 0000000000000000 R12: 0000000000000000
[ 646.678056] R13: 0000000000001000 R14: ffff88006cdedb48 R15: 0000000000000000
[ 646.678097] FS: 00007fd6f62096e0(0000) GS:ffffffff805b2000(0000) knlGS:0000000000000000
[ 646.678156] CS: e033 DS: 0000 ES: 0000
[ 646.678188] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 646.678228] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 646.678268] Process localedef (pid: 631, threadinfo ffff880059d82000, task ffff880064919800)
[ 646.678328] Stack: ffffffff80273e0e 0000000000001000 ffff8800677db970 0000000000000000
[ 646.678404] ffff880059d83db8 0000000000000000 ffff880059d83d38 ffff880072585440
[ 646.678477] ffff88006cdedb48 ffffffff881cbac0 ffff88006cdeda38 ffffffff881cbac0
[ 646.678527] Call Trace:
[ 646.678581] [<ffffffff80273e0e>] generic_file_buffered_write+0x1de/0x6e0
[ 646.678638] [<ffffffff881c852c>] :ext3:__ext3_journal_dirty_metadata+0x2c/0x70
[ 646.678699] [<ffffffff8023a97e>] current_fs_time+0x1e/0x30
[ 646.678741] [<ffffffff8027455f>] __generic_file_aio_write_nolock+0x24f/0x400
[ 646.678786] [<ffffffff802c38b6>] __getblk+0x26/0x250
[ 646.678826] [<ffffffff80274774>] generic_file_aio_write+0x64/0xd0
[ 646.678872] [<ffffffff881b8663>] :ext3:ext3_file_write+0x23/0xc0
[ 646.678915] [<ffffffff881b8640>] :ext3:ext3_file_write+0x0/0xc0
[ 646.678954] [<ffffffff8029d4ab>] do_sync_readv_writev+0xcb/0x110
[ 646.678998] [<ffffffff8024ca90>] autoremove_wake_function+0x0/0x30
[ 646.679038] [<ffffffff802a5b20>] permission+0xb0/0x160
[ 646.679076] [<ffffffff802a756f>] may_open+0xcf/0x290
[ 646.679112] [<ffffffff8029a0fc>] __kmalloc+0x13c/0x160
[ 646.679152] [<ffffffff8029dc4d>] do_readv_writev+0xfd/0x230
[ 646.679192] [<ffffffff8029bbca>] do_filp_open+0x3a/0x50
[ 646.679236] [<ffffffff8029e2c3>] sys_writev+0x53/0xc0
[ 646.679274] [<ffffffff8020c658>] system_call+0x68/0x6d
[ 646.679312] [<ffffffff8020c5f0>] system_call+0x0/0x6d
[ 646.679350]
[ 646.679376]
[ 646.679377] Code: 49 8b 52 08 49 89 d3 eb c4 4c 89 17 4c 89 4f 10 eb 99 0f 1f
[ 646.679544] RIP [<ffffffff80271e36>] iov_iter_advance+0x66/0x80
[ 646.679584] RSP <ffff880059d83b10>
[ 646.679615] CR2: ffff8800741d5008
[ 646.680368] ---[ end trace 39e94bb31c5ecce3 ]---
-------------------- snip

[Todd Deshane (deshantm)]

Do you have or can you make the scripts and templates publicly available so that they can be tested directly?

To me it looks like an LVM or ext3 problem... hard to say at this point.

One thing that I noticed in my testing of LVM during hardy alphas was that the LVM volume names in /dev/<vg name>/* are not available after a reboot. Maybe there is a bug with the hardy kernel or LVM packages?

LVM isn't well tested on the desktop, probably only on servers, so some subtle bug like that might linger longer since it is not a common path

[Marcus Grieger (mgrieger)]

xenomatic.tgz attached. Extract to /etc (archive contains subdirectory xenomatic) and generate VM with

$ /etc/xenomatic/xenomatic ldap-master

This will create LVM volumes /dev/vg/ldap-master-root and /dev/vg/ldap-master-swap. If your volumegroup has another name, check xenomatic.conf. A small README is also included. The script then debootstraps and fires up the VM with the console attached. To reproduce the bug shown above, shutdown or xm destroy the resulting VM, run the included cleanup script and run the xenomatic script again.

> One thing that I noticed in my testing of LVM during hardy alphas was that the LVM volume names in /dev/<vg name>/* are not available after a reboot. Maybe there is
> a bug with the hardy kernel or LVM packages?

I had that problem after dist-upgrading dapper to hardy alpha. Adding dm_mod to /etc/modules and setting the volumes available manually a couple of times fixed that problem.

[Marcus Grieger (mgrieger)]

Oh, a thing I forgot while changing IP-Addresses in the config files: xenomatic.conf still lists my apt-proxy as the ubuntu mirror, this wouldn't work outside our network and has to be changed.

[Marcus Grieger (mgrieger)]

I tested the system using the gutsy kernel (2.6.22-14-xen) on dom0, no problems there.

After booting back into the hardy kernel, running
the cleanup script and restarting installation, the Oops was back on the first run of debootstrap aber system boot.

[deti (deti)]

I hit the same bug in another context and found a solution. See Bug #231746
You can find there a precompiled xen kernel which includes the fix.