3.13 soft lockup up bug when running openstack tempest on KVM

[Aaron Rosen]

I'm running the openstack tempest tests that run on top of kvm. I frequently see this error reported in syslog which causes the machine to hang: http://logs2.aaronorosen.com/85/169585/1/check/dsvm-tempest-full-congress-nodepool/94f8441/logs/syslog.txt.gz#_Apr__1_02_43_44

jenkins@devstack-trusty-dib-nsx2-70:~$ uname -a
Linux devstack-trusty-dib-nsx2-70 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 1 22:36 seq
 crw-rw---- 1 root audio 116, 33 Apr 1 22:36 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.8
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
Ec2AMI: ami-000000bb
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: congress-ci
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-48-generic root=LABEL=cloudimg-rootfs ro nofb nomodeset vga=normal console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 3.13.0-48.80-generic 3.13.11-ckt16
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-48-generic N/A
 linux-backports-modules-3.13.0-48-generic N/A
 linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty ec2-images
Uname: Linux 3.13.0-48-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnOpenStackFoundation:pnOpenStackNova:pvr2013.2.2:cvnBochs:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2013.2.2
dmi.sys.vendor: OpenStack Foundation
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 1 22:36 seq
 crw-rw---- 1 root audio 116, 33 Apr 1 22:36 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.8
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
Ec2AMI: ami-000000bb
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: congress-ci
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-48-generic root=LABEL=cloudimg-rootfs ro nofb nomodeset vga=normal console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 3.13.0-48.80-generic 3.13.11-ckt16
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-48-generic N/A
 linux-backports-modules-3.13.0-48-generic N/A
 linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty ec2-images
Uname: Linux 3.13.0-48-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnOpenStackFoundation:pnOpenStackNova:pvr2013.2.2:cvnBochs:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2013.2.2
dmi.sys.vendor: OpenStack Foundation
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 1 22:36 seq
 crw-rw---- 1 root audio 116, 33 Apr 1 22:36 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.8
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
Ec2AMI: ami-000000bb
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: congress-ci
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-48-generic root=LABEL=cloudimg-rootfs ro nofb nomodeset vga=normal console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 3.13.0-48.80-generic 3.13.11-ckt16
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-48-generic N/A
 linux-backports-modules-3.13.0-48-generic N/A
 linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty ec2-images
Uname: Linux 3.13.0-48-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnOpenStackFoundation:pnOpenStackNova:pvr2013.2.2:cvnBochs:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2013.2.2
dmi.sys.vendor: OpenStack Foundation

[Aaron Rosen (arosen)]

Also the guest is configured to run with 9 gigs of ram and 6 processors so it should have plenty of resources.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1439394

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

apport information

[Aaron Rosen (arosen)]

Please look at the latest files that are uploaded. apport-collect 1439394 was run after the issue had occurred in that instance.

scopyarea ghash_clmulni_intel sysfillrect sysimgblt ttm aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul glue_helper ablk_helper drm cryptd psmouse floppy pata_acpi [last unloaded: kvm_intel]
[ 1860.158716] CPU: 5 PID: 5658 Comm: swift-object-au Not tainted 3.13.0-48-generic #80-Ubuntu
[ 1860.158718] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
[ 1860.158721] task: ffff880082fa6000 ti: ffff880078fba000 task.ti: ffff880078fba000
[ 1860.158723] RIP: 0010:[<ffffffff813708d5>] [<ffffffff813708d5>] copy_page_rep+0x5/0x10
[ 1860.158732] RSP: 0000:ffff880078fbbcf8 EFLAGS: 00010286
[ 1860.158734] RAX: 000000006d41b000 RBX: ffffffff81fa46a0 RCX: 0000000000000200
[ 1860.158736] RDX: ffff880082fa6000 RSI: ffff8801bce82000 RDI: ffff88006d41b000
[ 1860.158738] RBP: ffff880078fbbd70 R08: 0000000000000001 R09: 00000000fffffff7
[ 1860.158739] R10: ffff880271df9780 R11: ffff880271df9828 R12: ffff880082fa6000
[ 1860.158741] R13: 0000000000000000 R14: ffffea0001b6fec0 R15: ffff88026780cc00
[ 1860.158743] FS: 00007fa1cf7ef740(0000) GS:ffff880271b40000(0000) knlGS:0000000000000000
[ 1860.158745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1860.158747] CR2: 00000000029a0fd8 CR3: 0000000082e3a000 CR4: 00000000000407e0
[ 1860.158757] Stack:
[ 1860.158758] ffffffff81177a01 ffffea0001e35070 800000006dbfb867 00000000029d6000
[ 1860.158762] ffff880078d41d00 ffff8801bdd4dc00 ffffffff000000a8 0000000000000127
[ 1860.158765] 0000000000527000 ffffea0009c56b80 00000000029a0fd8 ffff880078fbc0a0
[ 1860.158768] Call Trace:
[ 1860.158776] [<ffffffff81177a01>] ? do_wp_page+0x121/0x7c0
[ 1860.158780] [<ffffffff8117a10c>] handle_mm_fault+0x65c/0xf10
[ 1860.158787] [<ffffffff8172d534>] __do_page_fault+0x184/0x560
[ 1860.158790] [<ffffffff8172d92a>] ? do_page_fault+0x1a/0x70
[ 1860.158793] [<ffffffff8172cfa9>] ? do_async_page_fault+0x29/0xe0
[ 1860.158797] [<ffffffff81729d98>] ? async_page_fault+0x28/0x30
[ 1860.158800] [<ffffffff8172d92a>] do_page_fault+0x1a/0x70
[ 1860.158803] [<ffffffff8172cfa9>] do_async_page_fault+0x29/0xe0
[ 1860.158806] [<ffffffff81729d98>] async_page_fault+0x28/0x30
[ 1860.158808] Code: 90 90 90 90 90 90 9c fa 65 48 3b 06 75 14 65 48 3b 56 08 75 0d 65 48 89 1e 65 48 89 4e 08 9d b0 01 c3 9d 30 c0 c3 b9 00 02 00 00 <f3> 48 a5 c3 0f 1f 80 00 00 00 00 eb ee 66 66 66 90 66 66 66 90
[ 1860.175004] BUG: soft lockup - CPU#0 stuck for 47s! [curl:5655]
[ 1860.177011] Modules linked in: dm_multipath scsi_dh xfs cmac rmd160 crypto_null xfrm_user ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm6_tunnel tunnel6 xfrm_ipcomp af_key xfrm_algo camellia_generic camellia_aesni_avx_x86_64 camellia_x86_64 cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common deflate cts ctr gcm ccm serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_generic twofish_avx_x86_64 twofish_x86_64_3way xts twof...

[Chris J Arges (arges)]

arosen:

It may be helpful to get more details about hang with a crashdump on a softlockup:
http://dinosaursareforever.blogspot.com/2014/10/getting-kernel-crashdumps-for-hung.html

Can you reproduce after ensuring you can obtain a crashdump?

In addition please collect the logs as described in #2 from an affected host machine.

Thanks,
--chris