[Lenovo ThinkPad T530 2359] Specific KVM instances lock my machine

[Seth Arnold]

With my linux-generic package at 3.8.0.27.45 I had a lockup while running eight KVM instances; the soundcard continued to loop the little tiny sample of audio it was on.

The only lines that made it to /var/log/kern.log:

Jul 31 11:11:18 hunt kernel: [10420.003674] device vnet2 entered promiscuous mode
Jul 31 11:11:18 hunt kernel: [10420.007687] virbr0: topology change detected, propagating
Jul 31 11:11:18 hunt kernel: [10420.007696] virbr0: port 3(vnet2) entered forwarding state
Jul 31 11:11:18 hunt kernel: [10420.007719] virbr0: port 3(vnet2) entered forwarding state
Jul 31 11:11:24 hunt kernel: [10425.644923] device vnet3 entered promiscuous mode
Jul 31 11:11:24 hunt kernel: [10425.653002] virbr0: topology change detected, propagating
Jul 31 11:11:24 hunt kernel: [10425.653006] virbr0: port 4(vnet3) entered forwarding state
Jul 31 11:11:24 hunt kernel: [10425.653013] virbr0: port 4(vnet3) entered forwarding state
Jul 31 11:11:30 hunt kernel: [10431.693359] device vnet4 entered promiscuous mode
Jul 31 11:11:30 hunt kernel: [10431.703026] virbr0: topology change detected, propagating
Jul 31 11:11:30 hunt kernel: [10431.703031] virbr0: port 5(vnet4) entered forwarding state
Jul 31 11:11:30 hunt kernel: [10431.703038] virbr0: port 5(vnet4) entered forwarding state
Jul 31 11:11:36 hunt kernel: [10437.649982] device vnet5 entered promiscuous mode
Jul 31 11:11:36 hunt kernel: [10437.658017] virbr0: topology change detected, propagating
Jul 31 11:11:36 hunt kernel: [10437.658022] virbr0: port 6(vnet5) entered forwarding state
Jul 31 11:11:36 hunt kernel: [10437.658027] virbr0: port 6(vnet5) entered forwarding state
Jul 31 11:11:38 hunt kernel: [10439.551771] kvm [8865]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd
Jul 31 11:11:42 hunt kernel: [10443.846061] device vnet6 entered promiscuous mode
Jul 31 11:11:42 hunt kernel: [10443.850430] virbr0: topology change detected, propagating
Jul 31 11:11:42 hunt kernel: [10443.850440] virbr0: port 7(vnet6) entered forwarding state
Jul 31 11:11:42 hunt kernel: [10443.850489] virbr0: port 7(vnet6) entered forwarding state
Jul 31 11:11:48 hunt kernel: [10449.515394] device vnet7 entered promiscuous mode
Jul 31 11:11:48 hunt kernel: [10449.523544] virbr0: topology change detected, propagating
Jul 31 11:11:48 hunt kernel: [10449.523549] virbr0: port 8(vnet7) entered forwarding state
Jul 31 11:11:48 hunt kernel: [10449.523557] virbr0: port 8(vnet7) entered forwarding state
Jul 31 11:11:50 hunt kernel: [10451.419931] kvm [9063]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff

I've had ten or twelve KVM instances running before, it's a workload I know works.

I was watching 'top' at the moment of the hang, and I still had over a gigabyte of cached and over a gigabyte of buffered memory available, so it wasn't as simple as running out of memory.

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: linux-image-3.8.0-27-generic 3.8.0-27.40
ProcVersionSignature: Ubuntu 3.8.0-27.40-generic 3.8.13.4
Uname: Linux 3.8.0-27-generic x86_64
ApportVersion: 2.9.2-0ubuntu8.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: sarnold 3023 F.... pulseaudio
Date: Wed Jul 31 11:21:55 2013
HibernationDevice: RESUME=UUID=81b2976f-2830-4449-9dae-e1d44a43fa78
InstallationDate: Installed on 2012-10-18 (286 days ago)
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
MachineType: LENOVO 2359CTO
MarkForUpload: True
ProcEnviron:
 TERM=rxvt-unicode
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.8.0-27-generic root=UUID=7b8c2e1b-d2e6-47d9-9030-c078e9701a1d ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.8.0-27-generic N/A
 linux-backports-modules-3.8.0-27-generic N/A
 linux-firmware 1.106
SourcePackage: linux
UpgradeStatus: Upgraded to raring on 2013-03-18 (135 days ago)
dmi.bios.date: 09/13/2012
dmi.bios.vendor: LENOVO
dmi.bios.version: G4ET62WW (2.04 )
dmi.board.asset.tag: Not Available
dmi.board.name: 2359CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Defined
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrG4ET62WW(2.04):bd09/13/2012:svnLENOVO:pn2359CTO:pvrThinkPadT530:rvnLENOVO:rn2359CTO:rvrNotDefined:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2359CTO
dmi.product.version: ThinkPad T530
dmi.sys.vendor: LENOVO

[Brad Figg (brad-figg)]

This change was made by a bot.

[Seth Arnold (seth-arnold)]

This is reproducable without adding VMs nine and ten -- one or the other of those VMs on its own is sufficient to kill my machine.

And this time, we got lucky, I have stack traces. I've trimmed my log to show the crash up close, then the rest of a reboot.

[Seth Arnold (seth-arnold)]

Turning off ksm (echo 0 > /sys/kernel/mm/ksm/run) changed the crash I got.

With ksm off, the crash is now in Xorg. I had to use the vnc viewer to get the KVM instances past grub's boot prompt.

[Seth Arnold (seth-arnold)]

I've booted back into kernel 3.8.0-26-generic and have successfully had eight VMs, including two of the troublesome ones, running for at least ten minutes. This might be a regression in the -27 kernels.

[penalvch (penalvch)]

Seth Arnold, as per http://support.lenovo.com/en_US/downloads/default.page? an update is available for your BIOS (2.54). If you update to this, does it change anything?

If not, could you please both specify what happened, and provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Thank you for your understanding.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.11 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.11-rc4-saucy/

[Seth Arnold (seth-arnold)]

Joseph, I'm not sure how to record my results. 3.11.0-031100rc4-generic could not start my VMs at all:

    Error starting domain: unsupported configuration:
    Domain requires KVM, but it is not available. Check
    that virtualization is enabled in the host BIOS, and host
    configuration is setup to load the kvm modules.

I still get this error message after a manual "modprobe kvm" and "service libvirt-bin restart".

[Seth Arnold (seth-arnold)]

Joseph, because this bug report looks so similar to bug 1208532 I tested the kernels in http://kernel.ubuntu.com/~jsalisbury/lp1208532/509384a/ -- and those allowed me to start the problematic VMs (raring i386, raring amd64) without crashing.

I do not use my VMs as extensively as that other bug reporter, but I never got them more than three seconds or so into booting when using host kernel 3.8.0.27.45, so making it all the way to a login prompt seems like success.

[penalvch (penalvch)]

Seth Arnold, as per http://download.lenovo.com/express/ddfm.html an update is available for your BIOS (2.54). If you update to this, does it change anything?

If not, could you please both specify what happened, and provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Please note your current BIOS is already in the Bug Description, so posting this on the old BIOS would not be helpful.

Thank you for your understanding.

[Seth Arnold (seth-arnold)]

Christpher, thanks, but I need to buy a CD-ROM drive before I can test a BIOS update. :(

[penalvch (penalvch)]

Seth Arnold, assuming you have a USB port, would unpacking the iso to a bootable USB not work in substitute?

[Seth Arnold (seth-arnold)]

Christopher, that is a good question. The official documentation describes clearly that an optical media drive of some sort is needed (http://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles/g4uj17uc.txt). Perhaps that is the exact truth, or perhaps it is just something that is very nearly true, but the average Windows user will not have a 'dd' near to hand to try. (They also do not mention "hybrid images", as our Ubuntu images are.)

The annoying thing is I presume it just boots Linux to get the job done anyway, because it's free and Just Works.

I also don't have any USB sticks I trust at this point -- the last time I needed one to install Ubuntu, none worked.

Either way it's a shopping trip.

Thanks :)

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]