LVM Based KVM VM taking 100% CPU on first start and hangs.

Bug #1029889 reported by Lewdmole
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Incomplete
High
Unassigned
lvm2 (Ubuntu)
New
Undecided
Unassigned
qemu-kvm (Ubuntu)
Incomplete
High
Unassigned

Bug Description

KVM VM hangs taking 100% CPU on Ubuntu 12.04 Precise Pangolin:

OS: Ubuntu 12.04 LTS
ARCH: x86_64

VM created with vmbuilder no errors on install.
After defining vm in virsh:
virsh --connect qemu:///system
virsh # define /etc/libvirt/qemu/vm4.xml
virsh # start vm4
virsh # autostart vm4

VM starts OK taking 100% CPU then hangs, unable to shutdown only destroy can be used.
cant connect via ssh. Nothing in logs. Before last KVM update ALL VM installs worked correctly:

INSTALL DETAILS:

 Creating LVM based VM using vmbuilder:

DIR: /var/lib/libvirt/images/vm4/

First Boot Script:
/var/lib/libvirt/images/vm4/boot.sh
------------------- boot.sh -------------------
# This script will run the first time the virtual machine boots
# It is ran as root.

# Install openssh-server
apt-get update
apt-get install -qqy --force-yes openssh-server
------------------- /boot.sh -------------------

Partition Descriptor:
/var/lib/libvirt/images/vm4/vmbuilder.partition
------------------- vmbuilder.partition -------------------
root 12000
swap 1000
/var 3000
/var/log 4000
------------------- /vmbuilder.partition -------------------

Create LV:

root@host:~# lvcreate -L20G -n vm4 virtualvg

root@host:~# cd /var/lib/libvirt/images/vm4/

vmbuilder kvm ubuntu \
--suite=precise \
--flavour=virtual \
--arch=amd64 \
--mem=256 \
--hostname=vm4 \
--bridge=br0 \
--mirror=http://cz.archive.ubuntu.com/ubuntu \
-o --libvirt=qemu:///system \
--ip 82.208.49.156 \
--mask 255.255.255.224 \
--net 82.208.49.128 \
--bcast 82.208.49.159 \
--gw 82.208.49.158 \
--dns 8.8.8.8 \
--part=/var/lib/libvirt/images/vm4/vmbuilder.partition \
--raw=/dev/mapper/virtualvg-vm4 \
--templates=/var/lib/libvirt/images/kvm-custom-temp \
--user=vm-user \
--name=VM User \
--pass=12345 \
--addpkg=vim-nox \
--addpkg=acpid \
--firstboot=/var/lib/libvirt/images/vm4/boot.sh

LAST UPDATE LOG:

Start-Date: 2012-07-18 21:20:31
Commandline: apt-get upgrade
Upgrade: apport:amd64 (2.0.1-0ubuntu8, 2.0.1-0ubuntu11), libvirt0:amd64 (0.9.8-2ubuntu17.1, 0.9.8-2ubuntu17.2), python-problem-report:amd64 (2.0.1-0ubuntu8, 2.0.1-0ubuntu11), devscripts:amd64 (2.11.6ubuntu1.2, 2.11.6ubuntu1.3), python-apport:amd64 (2.0.1-0ubuntu8, 2.0.1-0ubuntu11), python-libvirt:amd64 (0.9.8-2ubuntu17.1, 0.9.8-2ubuntu17.2), libvirt-bin:amd64 (0.9.8-2ubuntu17.1, 0.9.8-2ubuntu17.2)
End-Date: 2012-07-18 21:20:54

Start-Date: 2012-07-25 02:04:54
Commandline: apt-get dist-upgrade
Install: linux-headers-3.2.0-27:amd64 (3.2.0-27.43, automatic), linux-image-3.2.0-27-generic:amd64 (3.2.0-27.43), linux-headers-3.2.0-27-generic:amd64 (3.2.0-27.43, automatic)
Upgrade: linux-server:amd64 (3.2.0.23.25, 3.2.0.27.29), linux-headers-server:amd64 (3.2.0.23.25, 3.2.0.27.29), xkb-data:amd64 (2.5-1ubuntu1, 2.5-1ubuntu1.3), qemu-kvm:amd64 (1.0+noroms-0ubuntu13, 1.0+noroms-0ubuntu14), kvm:amd64 (84+dfsg-0ubuntu16+1.0+noroms+0ubuntu13, 84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14), cgroup-lite:amd64 (1.1, 1.1.2), qemu-utils:amd64 (1.0+noroms-0ubuntu13, 1.0+noroms-0ubuntu14), linux-image-server:amd64 (3.2.0.23.25, 3.2.0.27.29), linux-libc-dev:amd64 (3.2.0-26.41, 3.2.0-27.43), qemu-common:amd64 (1.0+noroms-0ubuntu13, 1.0+noroms-0ubuntu14)
End-Date: 2012-07-25 02:06:02

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1029889/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for submitting this bug. Could you please

1. while the guest is running, do 'ps -ef' and paste the result.
2. attach the guest xml file ('virsh dumpxml vm4 > vm4.xml' should give you a file you can attach)
3. run 'apport-collect 1029889' so apport can send us relevant qemu-kvm and libvirt debugging information.

Changed in qemu-kvm (Ubuntu):
importance: Undecided → High
Changed in libvirt (Ubuntu):
importance: Undecided → High
status: New → Incomplete
Changed in qemu-kvm (Ubuntu):
status: New → Incomplete
Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Download full text (6.9 KiB)

VM BUILDER INSTALL LOG:

2012-07-27 14:36:20,094 INFO : Calling hook: preflight_check
2012-07-27 14:36:20,096 INFO : Calling hook: set_defaults
2012-07-27 14:36:20,097 INFO : Calling hook: bootstrap
2012-07-27 14:39:35,613 INFO : Calling hook: configure_os
2012-07-27 14:39:59,199 INFO : invoke-rc.d: policy-rc.d denied execution of start.
2012-07-27 14:40:05,423 INFO :
2012-07-27 14:40:05,423 INFO : Current default time zone: 'Etc/UTC'
2012-07-27 14:40:05,426 INFO : Local time is now: Fri Jul 27 12:40:05 UTC 2012.
2012-07-27 14:40:05,426 INFO : Universal Time is now: Fri Jul 27 12:40:05 UTC 2012.
2012-07-27 14:40:05,426 INFO :
2012-07-27 14:40:14,118 INFO : gpg: key 437D05B5: "Ubuntu Archive Automatic Signing Key <email address hidden>" not changed
2012-07-27 14:40:14,121 INFO : gpg: key FBB75451: "Ubuntu CD Image Automatic Signing Key <email address hidden>" not changed
2012-07-27 14:40:14,121 INFO : gpg: Total number processed: 2
2012-07-27 14:40:14,121 INFO : gpg: unchanged: 2
2012-07-27 14:40:17,726 INFO : invoke-rc.d: policy-rc.d denied execution of stop.
2012-07-27 14:40:25,580 INFO : invoke-rc.d: policy-rc.d denied execution of stop.
2012-07-27 14:40:29,498 INFO : invoke-rc.d: policy-rc.d denied execution of start.
2012-07-27 14:40:30,933 INFO : invoke-rc.d: policy-rc.d denied execution of start.
2012-07-27 14:40:33,683 INFO : invoke-rc.d: policy-rc.d denied execution of start.
2012-07-27 14:40:41,626 INFO : Cleaning up
2012-07-27 14:40:41,628 INFO : Calling hook: preflight_check
2012-07-27 14:40:42,395 INFO : Calling hook: configure_networking
2012-07-27 14:40:42,411 INFO : Calling hook: configure_mounting
2012-07-27 14:40:42,421 INFO : Calling hook: mount_partitions
2012-07-27 14:40:42,422 INFO : Mounting target filesystems
2012-07-27 14:40:42,422 INFO : Adding partition table to disk image: /dev/mapper/virtualvg-vm4
2012-07-27 14:40:43,007 INFO : Adding type 4 partition to disk image: /dev/mapper/virtualvg-vm4
2012-07-27 14:40:43,007 INFO : Partition at beginning of disk - reserving first cylinder
2012-07-27 14:40:43,411 INFO : Adding type 3 partition to disk image: /dev/mapper/virtualvg-vm4
2012-07-27 14:40:43,425 INFO : [0] ../../libparted/filesys.c:148 (ped_file_system_type_get): File system alias linux-swap(new) is deprecated
2012-07-27 14:40:44,111 INFO : Adding type 4 partition to disk image: /dev/mapper/virtualvg-vm4
2012-07-27 14:40:44,782 INFO : Adding type 4 partition to disk image: /dev/mapper/virtualvg-vm4
2012-07-27 14:40:45,655 INFO : Creating loop devices corresponding to the created partitions
2012-07-27 14:40:45,690 INFO : Creating file systems
2012-07-27 14:40:45,695 INFO : mke2fs 1.42 (29-Nov-2011)
2012-07-27 14:40:53,230 INFO : mkswap: /dev/mapper/virtualvg-vm4p2: warning: don't erase bootbits sectors
2012-07-27 14:40:53,230 INFO : on whole disk. Use -f to force.
2012-07-27 14:40:53,551 INFO : mke2fs 1.42 (29-Nov-2011)
2012-07-27 14:40:56,257 INFO : mke2fs 1.42 (29-Nov-2011)
2012-07-27 14:41:01,660 INFO : Calling hook: install_bootloader
2012-07-27 14:41:29,068 INFO ...

Read more...

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

I have mounted the LV of the VIRTUAL MACHINE and extracted some logs:
It seems that the machine has not booted as the boot.log is empty:

I have attached all thelogs and report for HOST and also for GUEST:

HOST:
libvirtd.log
shutdownlog.log
vm4.log
ps-while-guest-running.txt
top.txt
vmbuilder-install-log.txt

GUEST:
entire /var/log
vm4.xml

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

When I simply grab a working root image (i.e. https://cloud-images.ubuntu.com/releases/precise/release-20120616/ubuntu-12.04-server-cloudimg-amd64-disk1.img), convert it to raw, and dd it to an lvm partition, it comes up fine. So it appears vmbuilder may be doing something either wrong, or just something unorthodox which is triggering a bug in libvirt/qemu. Still trying.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I can't reproduce this.

To be clear, you did not have this problem with 1.0+noroms-0ubuntu13, but you did as of 1.0+noroms-0ubuntu14 ? Can you confirm that by re-installing 1.0+noroms-0ubuntu13 (using 'dpkg -i /var/cache/apt/archives/qemu*ubuntu13*.deb') ?

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

Will try to test it with the cloud image tmrw and also re-installing 1.0+noroms-0ubuntu13
Will do second test on other machine running the same setup.

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

1.0+noroms-0ubuntu13 worked fine with all VM com up fine with the same vmbuilder setup
after update to 1.0+noroms-0ubuntu14 VM doesnt come up
NEEDS CONFIRMATION

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

second hardware machine running exactly the same setup Ubuntu 12.04 and KVM
creates and runs this VM without problem.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

This makes me wonder whether one of the packages installed on the first machine is corrupt. Does sha1sum /var/cache/apt/archives/qemu*ubuntu14* show the same results on both systems?

The ps -ef output confirms that kvm is not being called by libvirt with -pidfile, so the change from ubuntu13 to ubuntu14 (which only deals with handling of the pid file) all the more doesn't seem to make sense.

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

1. Posted the SHA1SUMS
2. Server1 (problematic server) - I have tried again to install different VMs via vmbuilder all finished installation successfully but hanged on startup (via virsh) taking 100% cpu - I am pretty sure the VMs are not booting as there is nothing in boot log on LV of particular VM when mounted.
3. Server2 with exact setup runs other VMs with no problem. THE ONLY DIFFERENCE is in update procedure as this server have been through kernel updates 3.2.0-26 and 3.2.0-27.
Server1 has been updated directly to 3.2.0-27. (both via apt-get dist-upgrade)
4. The question is how can i trace what is happening when i start the problematic VM via virsh (e.g. start vm4) This could help to find out what causes the boot problem.
5. The problematic server running other (previously installed before kvm and kernel updates) VMs without problem just can not create another VMs which sucks...

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

I have downloaded the cloud image precise-server-cloudimg-amd64-disk1.img

1. convert to raw

$ qemu-img convert -O raw precise-server-cloudimg-amd64-disk1.img precise-server-cloudimg-amd64-disk1.raw

2. dd it onto the hard drive.

# dd if=precise-server-cloudimg-amd64-disk1.raw of=/dev/virtualvg/vm4

the VM does not boot again hangs with 100% cpu. (screenshot)

I have found similar issues:
https://www.redhat.com/archives/libvirt-users/2012-February/msg00106.html (resolved but trivial error)
http://serverfault.com/questions/366085/kvm-virtual-machine-hangs-at-boot-using-vmbuilder (not resolved)

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

Conclusions:
- The VM DOES NOT BOOT the image is not corrupt as the cloud image neither boots.
- Other vms on that problematic machine running fine so the kvm-qemu and libvirt should be fine as well.
- There is still enough memory on that machine. (screenshot in prev post)
- By comparing sha1sums of update packages all seems to be equal on both machines.
- I have tried VM with 256MB, 512MB RAM which should be enough to boot the kernel
- there is nothing strange in libvirt logs (posted above)

I guess It could be related to kernel, processor architecture or LVM as it hangs at boot
Not sure how to get some stacktrace what is happening at boot of the VM.

ARCHITECTURE DIFFERENCE:
Server 1 (problematic): Processor Intel Xeon
Server2 : Processor AMD Opteron

There was a bug for select models of Intel cpus before:
http://<email address hidden>/msg39396.html

Changed in libvirt (Ubuntu):
status: Incomplete → New
Changed in qemu-kvm (Ubuntu):
status: Incomplete → New
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Hi,

thanks for the info and links. The patch at
http://<email address hidden>/msg39396.html should certainly be present in precise's kernel, but just to test, on the machine where it fails, could you try running without apic (remove the

    <apic/>

line from the xml, or turn it off with virtual machine manager under 'machine details'?

Revision history for this message
Luke Randall (luke-randall) wrote :

I have experienced the same issue with 1 of 3 identical machines. All Intel Xeon E31230 @ 3.20GHz. Let me know if there is anything I can do to help diagnose it.

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

Hi,

I still suffer from the issue described above even after various kernel and libvirt updates
which were issued by ubuntu.
Serge Hallyn tried to repeat this issue but without success.

Please try the procedures I was trying and post the logs and debug information here along with
your VM setup etc.

You will find all the details in this post.

It would be great if we could resolve that issue as it affects my only machine with Xeon E processor
others with the same seutup and AMD are working fine.

Best

Lewdmole

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Please follow the instructions in the section "Debug Symbol Packages" on https://wiki.ubuntu.com/DebuggingProgramCrash to install qemu-kvm-dbgsym. Restart one of the machine which hangs taking 100% cpu, then from another terminal do

gdb -p $(pidof kvm) # or insert the pid of the hanging kvm instance

then type 'where' to get information about what kvm is doing.

Revision history for this message
Luke Randall (luke-randall) wrote :

#0 0x00007fc1e6d77823 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fc1ea661da0 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:456
#2 0x00007fc1ea5cb5ff in main_loop () at /build/buildd/qemu-kvm-1.0+noroms/vl.c:1482
#3 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/buildd/qemu-kvm-1.0+noroms/vl.c:3527

Revision history for this message
Luke Randall (luke-randall) wrote :

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit lb1
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>lb1</name>
  <uuid>8ee4c960-184a-fa11-f9e8-292cd904371a</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/kvm/lb1/image.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:25:32:6f'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Revision history for this message
Luke Randall (luke-randall) wrote :

I've attached the VM instance's definition and the debugging output from gdb (which seemed quite minimal). Let me know if you need me to install debugging symbols for libc as well, or if you need any other data.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Luke,

thanks for the information. Just to be sure, your symptoms are exactly the same as in the Description - kvm hangs (no response to network or the vnc console) and takes 100% cpu?

Unfortunately, based on your .xml file it looks like you're not using an LVM backed VM. I wonder if we should pursue this in a separate bug. Would you mind opening a new bug?

If you could install the libc debugging symptoms and reproduce that could definately help. It would be good to figure out why select is hanging. If possible, getting the list of arguments passed to select from gdb and comparing the list of fds to /proc/<pid>/fd/<n> entries could also help.

If you change your machine type to pc-1.1 or pc-0.12, or change the network device type and/or switch to using the default virbr0 network, does that help?

Is there anything in your host's syslog which looks helpful?

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Lewdmole,

have you seen this happen with non-lvm backed VMs?

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

@Serge

I have not tried that with non-LVM backend as we generalu use faster LVM backend.
Will do a test next week after my vacations.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Lewdmole,

have you had a chance to try this out without lvm?

Changed in libvirt (Ubuntu):
status: New → Incomplete
Changed in qemu-kvm (Ubuntu):
status: New → Incomplete
Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

Sorry for the delay I have tried File based VM works without problems.
I can add another File based VM - I have installed via vmbuilder:

vmbuilder kvm ubuntu \
--suite=precise \
--flavour=virtual \
--arch=amd64 \
--mem=512 \
--hostname=vm4 \
--bridge=br0 \
--mirror=http://de.archive.ubuntu.com/ubuntu \
-o --libvirt=qemu:///system \
--ip x.x.x.x \
--mask x.x.x.x \
--net x.x.x.x \
--bcast x.x.x.x \
--gw x.x.x.x \
--dns 8.8.8.8 \
--part=/var/lib/libvirt/images/vm4/vmbuilder.partition \
--templates=/var/lib/libvirt/images/kvm-custom-temp \
--user=tuser \
--name=Tem User \
--pass=xxx \
--addpkg=vim-nox \
--addpkg=acpid \
--firstboot=/var/lib/libvirt/images/vm4/boot.sh \
2>&1 | tee vmbuilder-install-log.txt

LVM based is not working! It is a mystery because all my other working VMs are LVM ...

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

vgdisplay and lvdisplay

Is there any minimum space required on volume group to create LVM based VM apart of the LV size of the particular VM ?

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Luke Randall (luke-randall) wrote :

Sorry for not responding for so long.

> Just to be sure, your symptoms are exactly the same as in the Description - kvm hangs (no response to network or the vnc console) and takes 100% cpu?

Yes, the symptoms are exactly the same.

> Unfortunately, based on your .xml file it looks like you're not using an LVM backed VM. I wonder if we should pursue this in a separate bug. Would you mind opening a new bug?

I've experienced the exact same bug (or at least the exact same symptoms) both with LVM and qcow2 images. If you feel that LVM is a sufficiently differentiating factor then I'm happy to file a new bug.

I've found that two out of three servers with 12.04 are affected by this bug. That means that brand new images built on these machines exhibit this bug. However, I am still able to build images on the third server which work as expected on any of the three servers. I don't know what to make of that, since the machines should be close to identical in setup (and are certainly identical from a hardware perspective).

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

@Serge,

Let me know what else we could try to diagnose this problem.

This is the same VM machine created via vmbuilder with LVM volumes:

lvcreate -L26G -n vm5 virtualvg

vmbuilder.partition

root 20000
swap 6000

boot.sh

# Install openssh-server
apt-get update
apt-get install -qqy --force-yes openssh-server

vmbuilder kvm ubuntu \
--suite=precise \
--flavour=virtual \
--arch=amd64 \
--mem=512 \
--hostname=vm5 \
--bridge=br0 \
--mirror=http://de.archive.ubuntu.com/ubuntu \
-o --libvirt=qemu:///system \
--ip x.x.x.x \
--mask x.x.x.x \
--net x.x.x.x \
--bcast x.x.x.x \
--gw x.x.x.x \
--dns 8.8.8.8 \
--part=/var/lib/libvirt/images/vm5/vmbuilder.partition \
--raw=/dev/mapper/virtualvg-vm5 \
--templates=/var/lib/libvirt/images/kvm-custom-temp \
--user=tuser \
--name=Temp User \
--pass=xxx \
--addpkg=vim-nox \
--addpkg=acpid \
--firstboot=/var/lib/libvirt/images/vm5/boot.sh \
2>&1 | tee vmbuilder-install-lvm-based-log.txt

The VM hangs taking 100% CPU!

There is screenshot:

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

VM Builder Logs for VM4 - File Based and VM5 - LVM Based

Partition setup and boot.sh are identical:

vmbuilder.partition

root 20000
swap 6000

boot.sh

# Install openssh-server
apt-get update
apt-get install -qqy --force-yes openssh-server

RESULT:
VM4 (File Based) - Works No Problem
VM5 (LVM Based) - HANGS 100% CPU

Tomas

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Lewdmole,

Looking at your vg and lvdisplay output, it looks like you should have plenty of space for vm5.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Isn't this duplicate of bug 1012946 ?
Can you please try parted from precise-proposed as per instructions in:
https://bugs.launchpad.net/ubuntu/+source/parted/+bug/1012946/comments/11

and comment here or on the bug 1012946.

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

Ok will give it a try and post on bug bug 1012946.

Revision history for this message
Lewdmole (493pocbrcycmdw7yksonho9o2qzzq06652mtv6nw767dautd9im529ah6j-admin-d18ecat4t1b76tkfi3vttrkfngli4hci2jxl2sxy9j3h0x9j9k8q3kmsgv) wrote :

Still not fixed... I have found a workaround which was working (bellow) but now its not working anymore...
Same stuff like above in previous posts...

Someone spotted:
Please note that if you want to create a qunatal or precise guest, you need to either add the option --addpkg=linux-image-generic or apply the patch from
http://markmail.org/threadhg747mkmnkpy4icc#query:+page:1+mid:7rxyi4wjo6vkssh7+state:results
because otherwise the build process will fail!!!

This KVM installation with VMBuilder is rotten somewhere deep in my opinion, it has never worked properly.
This is a different machine (X86_64) a lots of memory and space...

PROCEDURE TO INSTALL AS LVM WHICH WAS WORKING BEFORE...
1.) Instal VM as file based with --addpkg=linux-image-generic
2.) Wait for the install to finish...
3.)
Afterwards, you can find an XML configuration file for the VM in /etc/libvirt/qemu/
(=> /etc/libvirt/qemu/vm4.xml):

root@host:~# ls -l /etc/libvirt/qemu/

This process will create VM image (qcow2) in /var/lib/libvirt/images/vm4/ubuntu-kvm

root@host:~# cd /var/lib/libvirt/images/vm4/ubuntu-kvm

Now that we know the name of our image (tmp09XnBf.qcow2), we can convert it as follows:

root@host:~# qemu-img convert tmp09XnBf.qcow2 -O raw /dev/virtualvg2/vm4

Afterwards you can delete the disk image:

root@host:~# rm -f tmp09XnBf.qcow2

Now we must modify the VM's configuration...

root@host:~# virsh edit vm4

... and change the following section...

-------------------------------------------------------------------------------------
[...]
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm4/ubuntu-kvm/tmp09XnBf.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
[...]
-------------------------------------------------------------------------------------

... so that it looks as follows:

-------------------------------------------------------------------------------------
[...]
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/dev/virtualvg2/vm4'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
[...]
-------------------------------------------------------------------------------------

Now we activate the VM:

root@host:~# virsh --connect qemu:///system
virsh # define /etc/libvirt/qemu/vm4.xml
virsh # start vm4
Making VM survive host reboot
virsh # autostart vm4

Result now not working taking 100% cpu and hangs!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.