Frequent Core Lock

[William Osborne]

I get these frequent CPU core locks. I got this on current CPU AMD Ryzen 1700 as well as AMD FX-8350. All hardware has been switched out except HD. Same issue. This seems to be a Kernal issue as far as i can tell. I started getting this issue in Ubuntu 14.04 with and without proprietary drivers.

May 30 21:08:39 TheSource kernel: [ 5707.856597] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kworker/8:0:11225]
May 30 21:08:39 TheSource kernel: [ 5707.859040] Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables xt_multiport iptable_filter ip_tables x_tables des_generic md4 nls_utf8 cifs fscache bi$
May 30 21:08:39 TheSource kernel: [ 5707.859105] CPU: 8 PID: 11225 Comm: kworker/8:0 Tainted: P W O L 4.4.0-78-generic #99-Ubuntu
May 30 21:08:39 TheSource kernel: [ 5707.859108] Hardware name: Micro-Star International Co., Ltd MS-7A32/X370 GAMING PRO CARBON (MS-7A32), BIOS 1.50 04/27/2017
May 30 21:08:39 TheSource kernel: [ 5707.859115] Workqueue: events once_deferred
May 30 21:08:39 TheSource kernel: [ 5707.859118] task: ffff8803d001c600 ti: ffff8803323e8000 task.ti: ffff8803323e8000
May 30 21:08:39 TheSource kernel: [ 5707.859120] RIP: 0010:[<ffffffff81104968>] [<ffffffff81104968>] smp_call_function_many+0x1f8/0x260
May 30 21:08:39 TheSource kernel: [ 5707.859127] RSP: 0018:ffff8803323ebc90 EFLAGS: 00000202
May 30 21:08:39 TheSource kernel: [ 5707.859129] RAX: 0000000000000003 RBX: 0000000000000200 RCX: 0000000000000001
May 30 21:08:39 TheSource kernel: [ 5707.859131] RDX: ffff88041ec5b160 RSI: 0000000000000200 RDI: ffff88041ee17d48
May 30 21:08:39 TheSource kernel: [ 5707.859132] RBP: ffff8803323ebcc8 R08: 0000000000000000 R09: 000000000000feff
May 30 21:08:39 TheSource kernel: [ 5707.859134] R10: 0000000000000008 R11: ffff88041ee17d48 R12: ffff88041ee17d48
May 30 21:08:39 TheSource kernel: [ 5707.859136] R13: ffff88041ee17d40 R14: ffffffff81035d30 R15: 0000000000000000
May 30 21:08:39 TheSource kernel: [ 5707.859139] FS: 00007f2b0f110700(0000) GS:ffff88041ee00000(0000) knlGS:000000000c4a1880
May 30 21:08:39 TheSource kernel: [ 5707.859141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 30 21:08:39 TheSource kernel: [ 5707.859142] CR2: 000055d6977db0a0 CR3: 00000003f7fae000 CR4: 00000000003406e0
May 30 21:08:39 TheSource kernel: [ 5707.859145] Stack:
May 30 21:08:39 TheSource kernel: [ 5707.859146] 0000000000017d00 01ff880300000001 ffffffff81773cae ffffffff81035d30
May 30 21:08:39 TheSource kernel: [ 5707.859149] 0000000000000000 ffffffff81773caf ffff8803ece2e280 ffff8803323ebcf0
May 30 21:08:39 TheSource kernel: [ 5707.859152] ffffffff81104aad ffffffff81773cae 0000000000000005 ffff8803323ebd43
May 30 21:08:39 TheSource kernel: [ 5707.859154] Call Trace:
May 30 21:08:39 TheSource kernel: [ 5707.859160] [<ffffffff81773cae>] ? ipqhashfn+0x2e/0xf0
May 30 21:08:39 TheSource kernel: [ 5707.859163] [<ffffffff81035d30>] ? arch_unregister_cpu+0x30/0x30
May 30 21:08:39 TheSource kernel: [ 5707.859166] [<ffffffff81773caf>] ? ipqhashfn+0x2f/0xf0
May 30 21:08:39 TheSource kernel: [ 5707.859170] [<ffffffff81104aad>] on_each_cpu+0x2d/0x60
May 30 21:08:39 TheSource kernel: [ 5707.859172] [<ffffffff81773cae>] ? ipqhashfn+0x2e/0xf0
May 30 21:08:39 TheSource kernel: [ 5707.859175] [<ffffffff81036d2a>] text_poke_bp+0x6a/0xf0
May 30 21:08:39 TheSource kernel: [ 5707.859178] [<ffffffff81773cae>] ? ipqhashfn+0x2e/0xf0
May 30 21:08:39 TheSource kernel: [ 5707.859182] [<ffffffff81033a1b>] arch_jump_label_transform+0x9b/0x120
May 30 21:08:39 TheSource kernel: [ 5707.859187] [<ffffffff8118c556>] __jump_label_update+0x76/0x90
May 30 21:08:39 TheSource kernel: [ 5707.859190] [<ffffffff8118c5f8>] jump_label_update+0x88/0x90
May 30 21:08:39 TheSource kernel: [ 5707.859194] [<ffffffff8118c8a1>] __static_key_slow_dec+0x41/0x90
May 30 21:08:39 TheSource kernel: [ 5707.859197] [<ffffffff8118c912>] static_key_slow_dec+0x22/0x60
May 30 21:08:39 TheSource kernel: [ 5707.859200] [<ffffffff81410a7a>] once_deferred+0x1a/0x30
May 30 21:08:39 TheSource kernel: [ 5707.859204] [<ffffffff8109a565>] process_one_work+0x165/0x480
May 30 21:08:39 TheSource kernel: [ 5707.859207] [<ffffffff8109a8cb>] worker_thread+0x4b/0x4c0
May 30 21:08:39 TheSource kernel: [ 5707.859210] [<ffffffff8109a880>] ? process_one_work+0x480/0x480
May 30 21:08:39 TheSource kernel: [ 5707.859214] [<ffffffff8109a880>] ? process_one_work+0x480/0x480
May 30 21:08:39 TheSource kernel: [ 5707.859217] [<ffffffff810a0bf8>] kthread+0xd8/0xf0
May 30 21:08:39 TheSource kernel: [ 5707.859220] [<ffffffff810a0b20>] ? kthread_create_on_node+0x1e0/0x1e0
May 30 21:08:39 TheSource kernel: [ 5707.859225] [<ffffffff81840dcf>] ret_from_fork+0x3f/0x70
May 30 21:08:39 TheSource kernel: [ 5707.859227] [<ffffffff810a0b20>] ? kthread_create_on_node+0x1e0/0x1e0
May 30 21:08:39 TheSource kernel: [ 5707.859229] Code: d2 e8 4d 92 30 00 3b 05 1b 56 e3 00 89 c1 0f 8d 99 fe ff ff 48 98 49 8b 55 00 48 03 14 c5 40 76 f3 81 8b 42 18 a8 01 74 ca f3 90 <8b> 42 18 a8 01 75 f7$

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-78-generic 4.4.0-78.99
ProcVersionSignature: Ubuntu 4.4.0-78.99-generic 4.4.62
Uname: Linux 4.4.0-78-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
AlsaVersion: Advanced Linux Sound Architecture Driver Version k4.4.0-78-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D3p', '/dev/snd/controlC0', '/dev/snd/by-path', '/dev/snd/hwC1D0', '/dev/snd/pcmC1D2c', '/dev/snd/pcmC1D1p', '/dev/snd/pcmC1D0c', '/dev/snd/pcmC1D0p', '/dev/snd/controlC1', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
Card1.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card1.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
Date: Wed May 31 07:31:16 2017
HibernationDevice: RESUME=UUID=1b6b0ecb-d0b1-4183-a44a-f4a091fe6a33
InstallationDate: Installed on 2016-08-02 (302 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Micro-Star International Co., Ltd MS-7A32
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 nouveaufb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-78-generic root=UUID=ae9408b8-f726-40a6-81c8-be3d25145890 ro
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-78-generic N/A
 linux-backports-modules-4.4.0-78-generic N/A
 linux-firmware 1.157.10
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/27/2017
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.50
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: X370 GAMING PRO CARBON (MS-7A32)
dmi.board.vendor: Micro-Star International Co., Ltd
dmi.board.version: 1.0
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Micro-Star International Co., Ltd
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1.50:bd04/27/2017:svnMicro-StarInternationalCo.,Ltd:pnMS-7A32:pvr1.0:rvnMicro-StarInternationalCo.,Ltd:rnX370GAMINGPROCARBON(MS-7A32):rvr1.0:cvnMicro-StarInternationalCo.,Ltd:ct3:cvr1.0:
dmi.product.name: MS-7A32
dmi.product.version: 1.0
dmi.sys.vendor: Micro-Star International Co., Ltd

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.12 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12-rc3

[William Osborne (syntax.intact)]

I have had this issue since i first built this machine and installed 14.04 when 14.04 was current version. I can't remember what update this occurred after it was a long time ago. I thought this was a hardware issue originally. I am happy to try upstream kernel however i can't get ZFS to run and is required for my workflow. If someone can provide an upstream kernel with ZFS support I would be happy to install it and give it a try.

[William Osborne (syntax.intact)]

FYI I have since reinstalled with 17.04 and have the same crashes.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1697132

[William Osborne (syntax.intact)]

I have removed zfs and migrated back to mdadm and was able to install an upstream kernel (4.10.17-041017-generic) and this issue persists.

[William Osborne (syntax.intact)]

I just noticed the recommended kernel of 4.12. I have installed the latest 4.12.2 and am currently testing

[Kai-Heng Feng (kaihengfeng)]

You need to file an upstream bug or raise the issue to mailing list.

[William Osborne (syntax.intact)]

I have installed kernel 4.12.2 and have zero crashes in over 24 hours with several TB of network traffic through it. This is a first time in over a year. Seems stable but will let it burn in before calling it resolved by the kernel upgrade.

[William Osborne (syntax.intact)]

I thought 4.12.2 fixed this issue but it just now crashed again.
I have replaced every single piece of hardware. CPU(fx and ryzen), mobo(asus and MSI), memory, hard drives(WD blue and re4 for boot disk, with red for storage array) , video card(nvidia and ATI).
The only commonality is Ubuntu with a large(7+TB) disk array on some kind a 64bit AMD CPU(i have not tried intel) with and without proprietary CPU code. I have used zfs and mdadm. I switched from samba to nfs for shares.

clean install with bind, and open ssh selected from tasksel and htop after the fact. Nothing else extra installed.

Issue started in Ubuntu 14.04 at some point later in it's life cycle.