NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [java:3427]

[Petri Airio]

Mar 15 13:51:28 airiot kernel: [78812.273886] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [java:3427]
Mar 15 13:51:28 airiot kernel: [78812.273929] Modules linked in: xt_multiport ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_conntrack nf_conntrack iptable_filter crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw glue_helper ablk_helper ppdev cryptd input_leds joydev i2c_piix4 mac_hid parport_pc serio_raw parport qemu_fw_cfg ip_tables x_tables autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid psmouse e1000 virtio_scsi pata_acpi floppy
Mar 15 13:51:28 airiot kernel: [78812.273935] CPU: 2 PID: 3427 Comm: java Tainted: G L 4.8.0-41-generic #44-Ubuntu
Mar 15 13:51:28 airiot kernel: [78812.273936] Hardware name: Hetzner vServer, BIOS 1.8.2 04/01/2014
Mar 15 13:51:28 airiot kernel: [78812.273938] task: ffff999f6ae2ac40 task.stack: ffff999f674f0000
Mar 15 13:51:28 airiot kernel: [78812.273946] RIP: 0010:[<ffffffffb070b4e1>] [<ffffffffb070b4e1>] smp_call_function_many+0x1f1/0x250
Mar 15 13:51:28 airiot kernel: [78812.273947] RSP: 0018:ffff999f674f3ca0 EFLAGS: 00000202
Mar 15 13:51:28 airiot kernel: [78812.273947] RAX: 0000000000000003 RBX: 0000000000000200 RCX: 0000000000000003
Mar 15 13:51:28 airiot kernel: [78812.273948] RDX: ffff999f7fd9d720 RSI: 0000000000000200 RDI: ffff999f7fd1a288
Mar 15 13:51:28 airiot kernel: [78812.273949] RBP: ffff999f674f3cd8 R08: fffffffffffffffe R09: 0000000000000009
Mar 15 13:51:28 airiot kernel: [78812.273949] R10: 0000000000000008 R11: 0000000000000206 R12: ffff999f7fd1a288
Mar 15 13:51:28 airiot kernel: [78812.273950] R13: ffff999f7fd1a280 R14: ffffffffb06723b0 R15: ffff999f674f3ce8
Mar 15 13:51:28 airiot kernel: [78812.273951] FS: 00007f50936dc700(0000) GS:ffff999f7fd00000(0000) knlGS:0000000000000000
Mar 15 13:51:28 airiot kernel: [78812.273952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 15 13:51:28 airiot kernel: [78812.273953] CR2: 00007fd9e89a0004 CR3: 000000042abc9000 CR4: 00000000000406e0
Mar 15 13:51:28 airiot kernel: [78812.273957] Stack:
Mar 15 13:51:28 airiot kernel: [78812.273962] 000000000001a240 01ff999f00000001 ffff999f6520a000 00007f50936cc000
Mar 15 13:51:28 airiot kernel: [78812.273964] ffff999f6520a2d8 00007f50936c7000 0000000000000005 ffff999f674f3d20
Mar 15 13:51:28 airiot kernel: [78812.273966] ffffffffb0672805 ffff999f6520a000 00007f50936c7000 00007f50936cc000
Mar 15 13:51:28 airiot kernel: [78812.273966] Call Trace:
Mar 15 13:51:28 airiot kernel: [78812.273971] [<ffffffffb0672805>] native_flush_tlb_others+0x65/0x130
Mar 15 13:51:28 airiot kernel: [78812.273973] [<ffffffffb06729e3>] flush_tlb_mm_range+0x63/0x150
Mar 15 13:51:28 airiot kernel: [78812.273986] [<ffffffffb07d7083>] tlb_flush_mmu_tlbonly+0x63/0xd0
Mar 15 13:51:28 airiot kernel: [78812.273988] [<ffffffffb07d83a4>] tlb_finish_mmu+0x14/0x50
Mar 15 13:51:28 airiot kernel: [78812.273990] [<ffffffffb07da52d>] zap_page_range+0xed/0x140
Mar 15 13:51:28 airiot kernel: [78812.273992] [<ffffffffb07e498e>] ? do_mmap+0x42e/0x510
Mar 15 13:51:28 airiot kernel: [78812.274007] [<ffffffffb09c0b28>] ? apparmor_mmap_file+0x18/0x20
Mar 15 13:51:28 airiot kernel: [78812.274009] [<ffffffffb07f016c>] SyS_madvise+0x3cc/0x8b0
Mar 15 13:51:28 airiot kernel: [78812.274014] [<ffffffffb07c5756>] ? vm_mmap_pgoff+0xc6/0xf0
Mar 15 13:51:28 airiot kernel: [78812.274019] [<ffffffffb069063b>] ? recalc_sigpending+0x1b/0x50
Mar 15 13:51:28 airiot kernel: [78812.274021] [<ffffffffb0693db6>] ? __set_current_blocked+0x36/0x60
Mar 15 13:51:28 airiot kernel: [78812.274039] [<ffffffffb0ea04f6>] entry_SYSCALL_64_fastpath+0x1e/0xa8
Mar 15 13:51:28 airiot kernel: [78812.274059] Code: d2 e8 04 ba 33 00 3b 05 c2 3a e5 00 89 c1 0f 8d 99 fe ff ff 48 98 49 8b 55 00 48 03 14 c5 e0 c5 55 b1 8b 42 18 a8 01 74 09 f3 90 <8b> 42 18 a8 01 75 f7 eb bf 0f b6 4d d0 4c 89 fa 4c 89 f6 44 89
Mar 15 13:51:28 airiot kernel: [78812.274227] [<ffffffffb07b0138>] clear_page_dirty_for_io+0x98/0x1b0
Mar 15 13:51:28 airiot kernel: [78812.274245] [<ffffffffb08c4f47>] mpage_submit_page+0x47/0x80
Mar 15 13:51:28 airiot kernel: [78812.274248] [<ffffffffb08c507b>] mpage_process_page_bufs+0xfb/0x110
Mar 15 13:51:28 airiot kernel: [78812.274251] [<ffffffffb08c63a3>] mpage_prepare_extent_to_map+0x203/0x310
Mar 15 13:51:28 airiot kernel: [78812.274260] [<ffffffffb08caa69>] ? ext4_writepages+0x499/0xd00
Mar 15 13:51:28 airiot kernel: [78812.274272] [<ffffffffb08fc62d>] ? __ext4_journal_start_sb+0x6d/0x120
Mar 15 13:51:28 airiot kernel: [78812.274275] [<ffffffffb08caa8a>] ext4_writepages+0x4ba/0xd00
Mar 15 13:51:28 airiot kernel: [78812.274289] [<ffffffffb0a3170f>] ? fprop_fraction_percpu+0x2f/0x80
Mar 15 13:51:28 airiot kernel: [78812.274292] [<ffffffffb07b140e>] do_writepages+0x1e/0x30
Mar 15 13:51:28 airiot kernel: [78812.274295] [<ffffffffb0863ed5>] __writeback_single_inode+0x45/0x320
Mar 15 13:51:28 airiot kernel: [78812.274316] [<ffffffffb08646d8>] writeback_sb_inodes+0x268/0x5f0
Mar 15 13:51:28 airiot kernel: [78812.274321] [<ffffffffb0864af2>] __writeback_inodes_wb+0x92/0xc0
Mar 15 13:51:28 airiot kernel: [78812.274324] [<ffffffffb0864e68>] wb_writeback+0x278/0x310
Mar 15 13:51:28 airiot kernel: [78812.274327] [<ffffffffb08657b4>] wb_workfn+0x234/0x410
Mar 15 13:51:28 airiot kernel: [78812.274335] [<ffffffffb069d96c>] process_one_work+0x1fc/0x4b0
Mar 15 13:51:28 airiot kernel: [78812.274338] [<ffffffffb069dc6b>] worker_thread+0x4b/0x500
Mar 15 13:51:28 airiot kernel: [78812.274339] [<ffffffffb069dc20>] ? process_one_work+0x4b0/0x4b0
Mar 15 13:51:28 airiot kernel: [78812.274343] [<ffffffffb06a40d8>] kthread+0xd8/0xf0
Mar 15 13:51:28 airiot kernel: [78812.274347] [<ffffffffb0ea071f>] ret_from_fork+0x1f/0x40
Mar 15 13:51:28 airiot kernel: [78812.274349] [<ffffffffb06a4000>] ? kthread_create_on_node+0x1e0/0x1e0

[Petri Airio (petria)]

$ cat /proc/version_signature
Ubuntu 4.8.0-41.44-generic 4.8.17

[Petri Airio (petria)]

$ lspci -vnvn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010] (prog-if 80 [Master])
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
        Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
        Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
        Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
        Region 4: I/O ports at c0c0 [size=16]
        Kernel driver in use: ata_piix
        Kernel modules: pata_acpi

00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] [8086:7020] (rev 01) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc QEMU Virtual Machine [1af4:1100]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin D routed to IRQ 11
        Region 4: I/O ports at c080 [size=32]
        Kernel driver in use: uhci_hcd

00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 03)
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 9
        Kernel driver in use: piix4_smbus
        Kernel modules: i2c_piix4

00:02.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) (prog-if 00 [VGA controller])
        Subsystem: Red Hat, Inc Device [1af4:1100]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Region 0: Memory at fd000000 (32-bit, prefetchable) [size=16M]
        Region 2: Memory at febf0000 (32-bit, non-prefetc...

[Petri Airio (petria)]

dmesg output

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1673057

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Petri Airio (petria)]

Made service request to provider (Hetzner Online) and they replied that there is nothing wrong with the host system ...

System is fresh Ubuntu 16.04 minimal

[Petri Airio (petria)]

apport-collect / apport-cli didn't find anything ?

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.11 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11-rc2

[Petri Airio (petria)]

Tried with upstream kernel:

 # uname -a
Linux airiot.fi 4.11.0-041100rc2-generic #201703121831 SMP Sun Mar 12 22:33:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Still happens:

Mar 15 20:45:44 airiot kernel: [ 139.629684] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [java:2124]
Mar 15 20:45:44 airiot kernel: [ 139.630151] Modules linked in: xt_multiport crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc ppdev aesni_intel aes_x86_64 crypto_simd glue_helper cry
ptd input_leds joydev serio_raw i2c_piix4 parport_pc parport mac_hid qemu_fw_cfg ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 xt_limit
 xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrac
k iptable_filter ip_tables x_tables autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic u
sbhid hid virtio_scsi psmouse e1000 floppy pata_acpi
Mar 15 20:45:44 airiot kernel: [ 139.630479] CPU: 1 PID: 2124 Comm: java Not tainted 4.11.0-041100rc2-generic #201703121831
Mar 15 20:45:44 airiot kernel: [ 139.630481] Hardware name: Hetzner vServer, BIOS 1.8.2 04/01/2014
Mar 15 20:45:44 airiot kernel: [ 139.630483] task: ffff9c61a95f2d00 task.stack: ffffb7d082cd4000
Mar 15 20:45:44 airiot kernel: [ 139.630528] RIP: 0010:clear_page+0xc/0x10
Mar 15 20:45:44 airiot kernel: [ 139.630534] RSP: 0000:ffffb7d082cd7c10 EFLAGS: 00010246 ORIG_RAX: ffffffffffffff10
Mar 15 20:45:44 airiot kernel: [ 139.630537] RAX: 0000000000000000 RBX: ffff9c61a95f2d00 RCX: 0000000000000200
Mar 15 20:45:44 airiot kernel: [ 139.630538] RDX: 000000000001efac RSI: 0000000000000012 RDI: ffff9c60bf9ed000
Mar 15 20:45:44 airiot kernel: [ 139.630539] RBP: ffffb7d082cd7cc8 R08: 0000000000000000 R09: 0000000000000020
Mar 15 20:45:44 airiot kernel: [ 139.630540] R10: 00007fb11971afe0 R11: 0000000000000081 R12: 0000000000000000
Mar 15 20:45:44 airiot kernel: [ 139.630542] R13: ffffecfd4cfe7b40 R14: ffff9c61bffc9d00 R15: ffffecfd4cfe7b40
Mar 15 20:45:44 airiot kernel: [ 139.630544] FS: 00007fb148ea2700(0000) GS:ffff9c61bfc80000(0000) knlGS:0000000000000000
Mar 15 20:45:44 airiot kernel: [ 139.630546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 15 20:45:44 airiot kernel: [ 139.630547] CR2: 00007fb11971b000 CR3: 00000004293af000 CR4: 00000000000406e0
Mar 15 20:45:44 airiot kernel: [ 139.630553] Call Trace:
Mar 15 20:45:44 airiot kernel: [ 139.630584] ? get_page_from_freelist+0x489/0xad0
Mar 15 20:45:44 airiot kernel: [ 139.630602] __alloc_pages_nodemask+0xdf/0x240
Mar 15 20:45:44 airiot kernel: [ 139.630617] alloc_pages_vma+0xab/0x250
Mar 15 20:45:44 airiot kernel: [ 139.630630] __handle_mm_fault+0xc69/0x10f0
Mar 15 20:45:44 airiot kernel: [ 139.630633] ? _copy_to_user+0x54/0x60
Mar 15 20:45:44 airiot kernel: [ 139.630637] handle_mm_fault+0xd0/0x240
Mar 15 20:45:44 airiot kernel: [ 139.630655] __do_page_fault+0x23e/0x4e0
Mar 15 20:45:44 airiot kernel: [ 139.630659] trace_do_page_fault+0x37/0xd0
Mar 15 20:45:44 a...

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]