Server Freezes using kvm (openstack-nova)

[Adrian Moya]

I'm running an Oneiric x64 server as an Openstack controller, and my server is freezing every random amount of time. Checking the syslogs, I can see what it seems to be a KVM problem. I'm not sure if this is the right place to place this bug, but I'm out of ideas. I hope this info gives a bit of light about what's happening.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: linux-image-3.0.0-12-server 3.0.0-12.20
ProcVersionSignature: Ubuntu 3.0.0-12.20-server 3.0.4
Uname: Linux 3.0.0-12-server x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D2', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D1c', '/dev/snd/pcmC0D1p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info: Error: [Errno 2] No such file or directory
Card0.Amixer.values: Error: [Errno 2] No such file or directory
Date: Fri Nov 11 16:27:40 2011
HibernationDevice: RESUME=UUID=482070ae-caf7-4f92-a818-2c4edcbd417d
InstallationMedia: Ubuntu-Server 11.04 "Natty Narwhal" - Release amd64 (20110426)
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Gigabyte Technology Co., Ltd. G31M-S2L
ProcEnviron:
 LANGUAGE=en_US:en
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.0.0-12-server root=/dev/mapper/shcontroller-root ro quiet
RelatedPackageVersions:
 linux-restricted-modules-3.0.0-12-server N/A
 linux-backports-modules-3.0.0-12-server N/A
 linux-firmware 1.60
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: Upgraded to oneiric on 2011-11-10 (1 days ago)
dmi.bios.date: 06/05/2008
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F6
dmi.board.name: G31M-S2L
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF6:bd06/05/2008:svnGigabyteTechnologyCo.,Ltd.:pnG31M-S2L:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnG31M-S2L:rvr:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: G31M-S2L
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Adrian Moya (adrianmoya)]

These are the lines in the syslogs regarding the error:

Nov 11 16:08:22 shcloud-controller kernel: [118784.365233] general protection fault: 0000 [#1] SMP
Nov 11 16:08:22 shcloud-controller kernel: [118784.365276] CPU 2
Nov 11 16:08:22 shcloud-controller kernel: [118784.365291] Modules linked in: ebt_arp ebt_ip ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm snd_hda_codec_realtek snd_hda_intel snd_hda_codec i915 snd_hwdep snd_pcm snd_timer snd soundcore drm_kms_helper snd_page_alloc drm i2c_algo_bit video dm_multipath ppdev parport_pc lp parport r8169
Nov 11 16:08:22 shcloud-controller kernel: [118784.365653]
Nov 11 16:08:22 shcloud-controller kernel: [118784.365666] Pid: 10050, comm: bash Not tainted 3.0.0-12-server #20-Ubuntu Gigabyte Technology Co., Ltd. G31M-S2L/G31M-S2L
Nov 11 16:08:22 shcloud-controller kernel: [118784.365737] RIP: 0010:[<ffffffff81168ee6>] [<ffffffff81168ee6>] fget_light+0x96/0xe0
Nov 11 16:08:22 shcloud-controller kernel: [118784.365791] RSP: 0018:ffff880028fdff28 EFLAGS: 00010206
Nov 11 16:08:22 shcloud-controller kernel: [118784.365823] RAX: ffff880028e39c80 RBX: 0000000000000000 RCX: 0000000000000008
Nov 11 16:08:22 shcloud-controller kernel: [118784.365866] RDX: 0000880137e0a3c0 RSI: ffff880028fdff54 RDI: ffff880020d927f8
Nov 11 16:08:22 shcloud-controller kernel: [118784.365908] RBP: ffff880028fdff28 R08: 0000000000000001 R09: ffff880020d92000
Nov 11 16:08:22 shcloud-controller kernel: [118784.365951] R10: 0000000000000008 R11: 0000000000000246 R12: 00000000000000ff
Nov 11 16:08:22 shcloud-controller kernel: [118784.365994] R13: 0000000000005410 R14: 00007fff76816acc R15: 00000000ffffffff
Nov 11 16:08:22 shcloud-controller kernel: [118784.366037] FS: 00007fd3ee094720(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
Nov 11 16:08:22 shcloud-controller kernel: [118784.366084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 16:08:22 shcloud-controller kernel: [118784.366120] CR2: 00000000fb4710b8 CR3: 0000000028c10000 CR4: 00000000000026e0
Nov 11 16:08:22 shcloud-controller kernel: [118784.366162] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 16:08:22 shcloud-controller kernel: [118784.366204] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 11 16:08:22 shcloud-controller kernel: [118784.366247] Process bash (pid: 10050, threadinfo ffff880028fde000, task ffff880105824560)
Nov 11 16:08:22 shcloud-controller kernel: [118784.366294] Stack:
Nov 11 16:08:22 shcloud-controller kernel: [118784.366310] ffff880028fdff78 ffffffff8117937f dead000000200200 0000000000390000
Nov 11 16:08:22 shcloud-controller kernel: [118784.366364] ffff880028fdff78 00000000810742c8 0000000000000000 0000000000000000
Nov 11 16:08:22 shcloud-controller kernel: [118784.366418] 0000000000000004 0000000000000000 0000000000002742 ffffffff81606c02
Nov 11 16:08:22 shcloud-controller kernel: [118784.366472] Call Trace:
Nov 11 16:08:22 shcloud-controller kernel: [118784.366493] [<ffffffff811793...

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . If possible, please test the latest v3.2-rcN kernel (Not a kernel in the daily directory). Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the others). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed by the mainline kernel, please add the following tag 'kernel-fixed-upstream-KERNEL-VERSION'. For example, if kernel version 3.2-rc1 fixed and issue, the tag would be: 'kernel-fixed-upstream-v3.2-rc1'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Thanks in advance.

[Adrian Moya (adrianmoya)]

Yes, I'll do this test during this week and came back with some results. Thanks!

[Adrian Moya (adrianmoya)]

I installed v3.2-rc2-oneiric amd64, I'll be checking how it goes and report any issue. If I found no problem, I'll tag as instructed (kernel-fixed-upstream-v3.2-rc2. Else I'll just remove the needs-upstream-testing.

Thanks again.

[Adrian Moya (adrianmoya)]

After installing kernel v3.2-rc2, I have not experienced this problem, so I'm tagging this as fixed in the upstream kernel.