create image - kernel panic

[Andreas Pedersen]

Binary package hint: xen-tools

I'm trying to run xen-create-image with the options dist=gutsy.
After(1 - 10 min) debootstrap it gives me a kernel panic.

     /var/log/messages
Nov 5 09:52:10 strand kernel: [ 635.056207] kjournald starting. Commit interval 5 seconds
Nov 5 09:52:10 strand kernel: [ 635.056614] EXT3 FS on dm-4, internal journal
Nov 5 09:52:10 strand kernel: [ 635.056619] EXT3-fs: mounted filesystem with ordered data mode.
Nov 5 09:52:26 strand kernel: [ 651.233609] ee21c8c9
Nov 5 09:52:26 strand kernel: [ 651.233730] Modules linked in: bridge video dock sbs container battery button ac ipv6 lp loop pcspkr parport_pc serial_
core parport psmouse iTCO_wdt iTCO_vendor_support serio_raw shpchp i82875p_edac edac_mc intel_agp agpgart pci_hotplug af_packet evdev ext3 jbd mbcache sg
 sd_mod sr_mod cdrom ata_piix floppy ata_generic libata scsi_mod e1000 ehci_hcd uhci_hcd usbcore dm_mirror dm_snapshot dm_mod thermal processor fan fuse
apparmor commoncap
Nov 5 09:52:48 strand kernel: [ 673.340666] [bad_page+99/160] bad_page+0x63/0xa0
Nov 5 09:52:48 strand kernel: [ 673.340980] [get_page_from_freelist+980/1072] get_page_from_freelist+0x3d4/0x430
Nov 5 09:52:48 strand kernel: [ 673.341293] [__alloc_pages+79/832] __alloc_pages+0x4f/0x340
Nov 5 09:52:48 strand kernel: [ 673.341590] [anon_vma_prepare+29/224] anon_vma_prepare+0x1d/0xe0
Nov 5 09:52:48 strand kernel: [ 673.341878] [__pagevec_lru_add_active+218/256] __pagevec_lru_add_active+0xda/0x100
Nov 5 09:52:48 strand kernel: [ 673.342168] [__handle_mm_fault+2909/6784] __handle_mm_fault+0xb5d/0x1a80
Nov 5 09:52:48 strand kernel: [ 673.342455] [evtchn_do_upcall+182/240] evtchn_do_upcall+0xb6/0xf0
Nov 5 09:52:48 strand kernel: [ 673.342767] [schedule+854/2304] schedule+0x356/0x900
Nov 5 09:52:48 strand kernel: [ 673.343089] [follow_page+473/1056] follow_page+0x1d9/0x420
Nov 5 09:52:48 strand kernel: [ 673.343408] [get_user_pages+366/1328] get_user_pages+0x16e/0x530
Nov 5 09:52:48 strand kernel: [ 673.343719] [make_pages_present+115/160] make_pages_present+0x73/0xa0
Nov 5 09:52:48 strand kernel: [ 673.344062] [do_brk+379/528] do_brk+0x17b/0x210
Nov 5 09:52:48 strand kernel: [ 673.344375] [sys_brk+164/240] sys_brk+0xa4/0xf0
Nov 5 09:52:48 strand kernel: [ 673.344664] [syscall_call+7/11] syscall_call+0x7/0xb
Nov 5 09:52:48 strand kernel: [ 673.344958] [wait_for_completion_interruptible+304/336] wait_for_completion_interruptible+0x130/0x150
Nov 5 09:52:48 strand kernel: [ 673.345259] =======================

[Todd Deshane (deshantm)]

Is this a repeatable problem (every time you run the xen-create-imag)e?

Have you tried with other dist=options?

Can you also provide your kernel version, xen package versions, and your hardware configuration?

You could try running the following commands manually:

mkdir -p /tmp/guest
debootstrap gutsy /tmp/guest

then you can remove that directory with
rm -rf /tmp/guest

[Andreas Pedersen (alofflambas)]

I have also tried with different dist=options and it seems to die
after debootstrap.

Xen Packages:
$ dpkg --get-selections|grep xen
2.6.22-14-xenlibc6-xen
libxen3.1
linux-image-2.6.22-14-xen
linux-image-xen
linux-restricted-modules-2.6.22-14-xen
linux-restricted-modules-xen
linux-ubuntu-modules-2.6.22-14-xen
linux-xen
python-xen-3.1
ubuntu-xen-server
xen-docs-3.1
xen-hypervisor-3.1
xen-ioemu-3.1
xen-tools
xen-utils-3.1

I can do 'debootstrap distname /tmp/dist' with out any problems at all.

[Anders Wallenquist (aw)]

I have the same experience on a mac mini (gutsy) with ubuntu-xen-desktop and lvm2 installed.

The machine hangs during debootstrap in xm-create-image-command with a kernel panic.

aw@macken:~$ dpkg --get-selections|grep xen
libc6-xen install
libxen3.1 install
linux-image-2.6.22-14-xen install
linux-image-xen install
linux-restricted-modules-2.6.22-14-xen install
linux-restricted-modules-xen install
linux-ubuntu-modules-2.6.22-14-xen install
linux-xen install
python-xen-3.1 install
ubuntu-xen-desktop install
xen-docs-3.1 install
xen-hypervisor-3.1 install
xen-ioemu-3.1 install
xen-tools install
xen-utils-3.1 install
xenman install
aw@macken:~$

[Todd Deshane (deshantm)]

Can you guys specify the exact xen-create-image commands that you are running so that it is easier to reproduce?

Also, see the links below and see if you can provide any more useful logs and debug information:

https://wiki.ubuntu.com/Strace

https://wiki.ubuntu.com/KernelTeamBugPolicies

https://help.ubuntu.com/community/DebuggingSystemCrash

[Andreas Pedersen (alofflambas)]

Here is the logfiles i hope it helps.

 $ strace -Ff -tt xend-create-image --hostname test --dhcp 2>&1 | tee strace

[Andreas Pedersen (alofflambas)]

I have update to the pre-released xen kernel (2.6.22-14-38) update, and its seems as i can better point out where I'm getting my kernel panic now.

 $ strace -Ff -tt xend-create-image --hostname test --dhcp

[pid 7994] 10:58:29.074749 fcntl64(14, F_GETFD) = 0
[pid 7994] 10:58:29.074988 fcntl64(14, F_SETFD, FD_CLOEXEC) = 0
[pid 7994] 10:58:29.075220 _llseek(14, 0, [0], SEEK_CUR) = 0
[pid 7994] 10:58:29.075464 fstat64(14, {st_mode=S_IFREG|0644, st_size=716, ...}) = 0
[pid 7994] 10:58:29.075836 mmap2(NULL, 716, PROT_READ, MAP_SHARED, 14, 0) = 0xb7e33000
[pid 7994] 10:58:29.076094 _llseek(14, 716, [716], SEEK_SET) = 0
[pid 7994] 10:58:29.076371 munmap(0xb7e33000, 716) = 0
[pid 7994] 10:58:29.076635 close(14) = 0
[pid 7994] 10:58:29.076885 open("/etc/group", O_RDONLY) = 14
[pid 7994] 10:58:29.077156 fcntl64(14, F_GETFD) = 0
[pid 7994] 10:58:29.077385 fcntl64(14, F_SETFD, FD_CLOEXEC) = 0
[pid 7994] 10:58:29.077620 _llseek(14, 0, [0], SEEK_CUR) = 0
[pid 7994] 10:58:29.077866 fstat64(14, {st_mode=S_IFREG|0644, st_size=446, ...}) = 0
[pid 7994] 10:58:29.078208 mmap2(NULL, 446, PROT_READ, MAP_SHARED, 14, 0) = 0xb7e33000
[pid 7994] 10:58:29.078463 _llseek(14, 446, [446], SEEK_SET) = 0
[pid 7994] 10:58:29.078741 munmap(0xb7e33000, 446) = 0
[pid 7994] 10:58:29.079085 close(14) = 0
[pid 7994] 10:58:29.079353 lstat64("/usr/share/zoneinfo/posix/Asia/Yerevan", {st_mode=S_IFREG|0644, st_size=760, .
..}) = 0
[pid 7994] 10:58:29.079768 rmdir("/usr/share/zoneinfo/posix/Asia/Yerevan.dpkg-new") = -1 ENOENT (No such file or d
irectory)
[pid 7994] 10:58:29.080099 rmdir("/usr/share/zoneinfo/posix/Asia/Yerevan.dpkg-tmp") = -1 ENOENT (No such file or d
irectory)
[pid 7994] 10:58:29.080397 open("/usr/share/zoneinfo/posix/Asia/Yerevan.dpkg-new", O_WRONLY|O_CREAT|O_EXCL|O_LARGE
FILE, 0) = 14
[pid 7994] 10:58:29.080771 read(12, "TZif\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\16\0\0\0\16"..., 760) = 760
[pid 7994] 10:58:29.081089 write(14, "TZif\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\16\0\0\0\16"..., 760) = 760
[pid 7994] 10:58:29.081434 read(12, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 264) = 264
[pid 7994] 10:58:29.081725 fchown32(14, 0, 0) = 0
[pid 7994] 10:58:29.081971 fchmod(14, 0644) = 0
[pid 7994] 10:58:29.082197 close(14) = 0
[pid 7994] 10:58:29.082491 utime("/usr/share/zoneinfo/posix/Asia/Yerevan.dpkg-new", [2007/12/13-10:57:17, 2007/05/
22-02:58:21]) = 0
[pid 7994] 10:58:29.082849 link("/usr/share/zoneinfo/posix/Asia/Yerevan", "/usr/share/zoneinfo/posix/Asia/Yerevan.
dpkg-tmp") = 0
[pid 7994] 10:58:29.083247 rename("/usr/share/zoneinfo/posix/Asia/Yerevan.dpkg-new", "/usr/share/zoneinfo/posix/As
ia/Yerevan") = 0
[pid 7994] 10:58:29.083642 read(12, "./usr/share/zoneinfo/posix/Asia/"..., 512) = 512
[pid 7994] 10:58:29.083940 open("/etc/passwd", O_RDONLY) = 14
[pid 7994] 10:58:29.084199 fcntl64(14, F_GETFD) = 0
[pid 7994] 10:58:29.084417 fcntl64(14, F_SETFD, FD_CLOEXEC) = 0
[pid 7994] 10:58:29.084637 _llseek(14, 0, [0], SEEK_CUR) = 0
[pid 7994] 10:58:29.084874 fstat64(14, {st_mode=S_IFREG|0644, st_size=716, ...}) = 0
[pid 7994] 10:58:29.085202 mmap2(NULL, 716, PROT_READ, MAP_SHARED, 14, 0) = 0xb7e33000
[pid 799...

[Adam Ashley (adamashley)]

I am getting the same problem, same software setup running on a Athlon X2 4400. I don't think it is related to xen-tools themselves I believe it is actually a kernel bug.

Running xen-create-image under the normal kernel linux-image-2.6.22-14-server it will work perfectly. Running it under linux-image-2.6.22-14-xen will get a kernel panic every time without fail.

It will also panic on an heavy disk i/o ie copying large files, installing large packages, running multiple xen virtual machines at the same time, swapping in and out to the disk.

[Adam Ashley (adamashley)]

as a work around i've disabled SMP and the system is now behaving itself

[Adam Ashley (adamashley)]

update to that disabling SMP doesn't completely stop the problem but it does reduce it enough that the system is almost usable. Only needs to be rebooted once a day not every time you start a virtual machine.

[EQuake (equake)]

Hello.

Having the same problem, with a SMP intel xeon.
I'm attaching the output of everything that were open when the crash happened. Hope that helps.

Marcos

[Todd Deshane (deshantm)]

I have not used the xen-tools scripts all that much yet and have always preferred to make a guest manually.

One possible work around for this is to do it (more) manually with a custom script.

I have a config file that I have been using, you could make one automatically with a script based on it

This was built from source install, so the kernel and ramdisk should be updated for your needs.

kernel='/boot/vmlinuz-2.6.18.8-xen'
ramdisk='/boot/initrd-2.6.18.8-xen.img'
memory=256
disk=['tap:aio:/xen/images/ubuntu1.img,xvda1,w']
root='/dev/xvda1 ro'
vif=['']
extra='xencons=tty'

I have a simple script that I have been using. It is by no means robust yet. but you should be able to customize for your needs in most cases I bet.

The script also handles some post install stuff like ssh keys etc. you should customize based on your needs.

#!/bin/bash

if [ -z $1 ]
 then
 echo "usage $0 name"
 exit 1
fi

if [ -f /xen/images/${1}.img ]
 then
 echo "/xen/images/${1}.img file exists"
 exit 1
fi

echo 'making disk'
echo ''
sudo dd if=/dev/zero of=/xen/images/${1}.img seek=10000 bs=1024k count=1
echo 'formatting disk'
echo ''
sudo mkfs.ext3 -F /xen/images/${1}.img
echo 'mounting image'
echo ''
sudo mkdir -p /mnt/$1
sudo mount -o loop /xen/images/${1}.img /mnt/$1
echo 'running debootstrap'
echo ''
sudo debootstrap --components=main,universe,multiverse --include=openssh-server,language-pack-en,libc6-xen,denyhosts,mailx,logwatch gutsy /mnt/$1
echo 'copying fstab'
echo ''
sudo cp /xen/templates/fstab /mnt/$1/etc/
echo 'copying sources list'
echo ''
sudo cp /etc/apt/sources.list /mnt/$1/etc/apt/
echo 'setting hostname to:'
sudo echo $1 | sudo tee /mnt/$1/etc/hostname
sed "s/`hostname`/$1/" /etc/hosts | sudo tee /mnt/$1/etc/hosts
echo 'copying interfaces file'
echo ''
sudo cp /xen/templates/interfaces /mnt/$1/etc/network/
echo 'copying modules...'
echo ''
sudo cp -r /lib/modules/`uname -r` /mnt/$1/lib/modules/
echo 'copying ssh keys'
echo ' '
sudo mkdir /mnt/$1/root/.ssh/
sudo cp /xen/templates/ssh_key /mnt/$1/root/.ssh/authorized_keys
echo 'updating denyhosts config'
echo ''
sudo cp /etc/denyhosts.conf /mnt/$1/etc/
echo 'copying locale file'
echo ''
sudo cp /etc/default/locale /mnt/$1/etc/default/

[Todd Deshane (deshantm)]

Another possible work around, I just noticed this package that is basically a really nice shell script similar in functionality to xen-create-image

https://launchpad.net/ubuntu/hardy/+source/ubuntu-vm-builder/0.2

[Shimi Chen (shimi-chen)]

Is this bug still relevant to a current version of Ubuntu?

[Launchpad Janitor (janitor)]

[Expired for xen-tools (Ubuntu) because there has been no activity for 60 days.]