divide error: 0000 [#1] SMP in task_numa_migrate - handle_mm_fault

[Markus Schade]

While running qemu 2.5 on a trusty host running 4.4.0-15.31~14.04.1 the host system has crashed (load > 200) 3 times in the last 3 days.

Always with this stack trace:

Apr 9 19:01:09 cnode9.0 kernel: [197071.195577] divide error: 0000 [#1] SMP
Apr 9 19:01:09 cnode9.0 kernel: [197071.195633] Modules linked in: vhost_net vhost macvtap macvlan arc4 md4 nls_utf8 ci
fs nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 xt_NFQUEUE xt_CLASSIFY ip6table_mangle sch_sfq sch_htb veth dccp_diag
 dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ebtable_filter ebtables nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_fil
ter ip6_tables iptable_mangle xt_CT iptable_raw xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter ip_tables x_tables dum
my bridge stp llc ipmi_ssif ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm dcdbas irqbypass crct10dif_p
clmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds nf_nat_ftp sb_edac nf_conntrack_ftp e
dac_core cdc_ether nf_nat_pptp usbnet nf_conntrack_pptp mii nf_nat_proto_gre lpc_ich nf_nat_sip ioatdma nf_nat nf_conntrack_sip nfsd ipmi_si
8250_fintek nf_conntrack_proto_gre ipmi_msghandler acpi_pad wmi shpchp nf_conntrack acpi_power_meter mac_hid auth_rpcgss nfs_acl bonding nfs
lp lockd parport grace sunrpc fscache tcp_htcp xfs btrfs hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_
xor async_tx xor ixgbe raid6_pq libcrc32c igb vxlan raid1 i2c_algo_bit ip6_udp_tunnel dca udp_tunnel ahci raid0 ptp libahci megaraid_sas mult
ipath pps_core mdio linear fjes
Apr 9 19:01:09 cnode9.0 kernel: [197071.197014] CPU: 13 PID: 3147726 Comm: ceph-osd Not tainted 4.4.0-15-generic #31~14
.04.1-Ubuntu
Apr 9 19:01:09 cnode9.0 kernel: [197071.197085] Hardware name: Dell Inc. PowerEdge R720/0XH7F2, BIOS 2.5.2 01/28/2015
Apr 9 19:01:09 cnode9.0 kernel: [197071.197154] task: ffff88252be1ee00 ti: ffff8824fc0d4000 task.ti: ffff8824fc0d4000
Apr 9 19:01:09 cnode9.0 kernel: [197071.197221] RIP: 0010:[<ffffffff810afec8>] [<ffffffff810afec8>] task_numa_find_cpu+0x238/0x700
Apr 9 19:01:09 cnode9.0 kernel: [197071.197300] RSP: 0000:ffff8824fc0d7ba8 EFLAGS: 00010257
Apr 9 19:01:09 cnode9.0 kernel: [197071.197340] RAX: 0000000000000000 RBX: ffff8824fc0d7c48 RCX: 0000000000000000
Apr 9 19:01:09 cnode9.0 kernel: [197071.197406] RDX: 0000000000000000 RSI: ffff88479f180000 RDI: ffff884782a47600
Apr 9 19:01:09 cnode9.0 kernel: [197071.197473] RBP: ffff8824fc0d7c10 R08: 0000000102eea157 R09: 00000000000001a8
Apr 9 19:01:09 cnode9.0 kernel: [197071.197540] R10: 000000000002404b R11: 000000000000023f R12: ffff882380930000
Apr 9 19:01:09 cnode9.0 kernel: [197071.197606] R13: 0000000000000008 R14: 000000000000008c R15: 0000000000000124
Apr 9 19:01:09 cnode9.0 kernel: [197071.197673] FS: 00007f19aab5b700(0000) GS:ffff88479f180000(0000) knlGS:0000000000000000
Apr 9 19:01:09 cnode9.0 kernel: [197071.197741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 9 19:01:09 cnode9.0 kernel: [197071.197782] CR2: 0000000025469600 CR3: 00000023846bc000 CR4: 00000000000426e0
Apr 9 19:01:09 cnode9.0 kernel: [197071.197848] Stack:
Apr 9 19:01:09 cnode9.0 kernel: [197071.197880] ffffffff817425fb ffff8829af3e9e00 00000000000000f6 ffff88252be1ee00
Apr 9 19:01:09 cnode9.0 kernel: [197071.197965] 000000000000008d 0000000000000225 0000000000016d40 000000000000008d
Apr 9 19:01:09 cnode9.0 kernel: [197071.198047] ffff88252be1ee00 00000000000001ad ffff8824fc0d7c48 00000000000000e1
Apr 9 19:01:09 cnode9.0 kernel: [197071.198132] Call Trace:
Apr 9 19:01:09 cnode9.0 kernel: [197071.198172] [<ffffffff817425fb>] ? tcp_schedule_loss_probe+0x12b/0x1b0
Apr 9 19:01:09 cnode9.0 kernel: [197071.198219] [<ffffffff810b0830>] task_numa_migrate+0x4a0/0x930
Apr 9 19:01:09 cnode9.0 kernel: [197071.198264] [<ffffffff816d2957>] ? release_sock+0x117/0x160
Apr 9 19:01:09 cnode9.0 kernel: [197071.198306] [<ffffffff810b0d39>] numa_migrate_preferred+0x79/0x80
Apr 9 19:01:09 cnode9.0 kernel: [197071.198350] [<ffffffff810b557d>] task_numa_fault+0x91d/0xcc0
Apr 9 19:01:09 cnode9.0 kernel: [197071.198395] [<ffffffff811d35ae>] ? mpol_misplaced+0x14e/0x190
Apr 9 19:01:09 cnode9.0 kernel: [197071.198439] [<ffffffff811b06b8>] handle_pte_fault+0x5a8/0x14c0
Apr 9 19:01:09 cnode9.0 kernel: [197071.198485] [<ffffffff810f8531>] ? futex_wake+0x81/0x150
Apr 9 19:01:09 cnode9.0 kernel: [197071.198526] [<ffffffff810b0de4>] ? set_next_entity+0xa4/0x700
Apr 9 19:01:09 cnode9.0 kernel: [197071.198569] [<ffffffff810fab44>] ? do_futex+0xf4/0x4d0
Apr 9 19:01:09 cnode9.0 kernel: [197071.198610] [<ffffffff811b2440>] handle_mm_fault+0x250/0x540
Apr 9 19:01:09 cnode9.0 kernel: [197071.198654] [<ffffffff81067d19>] __do_page_fault+0x199/0x430
Apr 9 19:01:09 cnode9.0 kernel: [197071.198696] [<ffffffff81067fd2>] do_page_fault+0x22/0x30
Apr 9 19:01:09 cnode9.0 kernel: [197071.198740] [<ffffffff817ef878>] page_fault+0x28/0x30
Apr 9 19:01:09 cnode9.0 kernel: [197071.198775] Code: 4d b0 4c 89 f7 e8 29 d5 ff ff 48 8b 4d b0 49 8b 86 b0 00 00 00 31 d2 48 0f af 81 d8 01 00 00 49 8b 4e 78 4c 8b 73 78 48 83 c1 01 <48> f7 f1 48 8b 4b 20 49 89 c1 48 29 c1 4c 03 4b 48 4c 39 7d d0
Apr 9 19:01:09 cnode9.0 kernel: [197071.199217] RIP [<ffffffff810afec8>] task_numa_find_cpu+0x238/0x700
Apr 9 19:01:09 cnode9.0 kernel: [197071.199264] RSP <ffff8824fc0d7ba8>
Apr 9 19:01:09 cnode9.0 kernel: [197071.199900] ---[ end trace e938a840610a79f7 ]---

This is appears to be the same bug as reported upstream in
http://lkml.iu.edu/hypermail/linux/kernel/1603.2/01659.html

According to this thread the issue is:

27: 48 83 c1 01 add $0x1,%rcx
2b:* 48 f7 f1 div %rcx <-- trapping instruction

This suggests the CONFIG_FAIR_GROUP_SCHED version of task_h_load:

update_cfs_rq_h_load(cfs_rq);
return div64_ul(p->se.avg.load_avg * cfs_rq->h_load,
cfs_rq_load_avg(cfs_rq) + 1);

So the load avg is -1, thus after adding 1 we get division by 0

The fix of the LKML reporter was to include the patches to kernel/sched/fair.c up to 4.5
A specific patch was not identified.

Please backport these patches for Xenial and lts-xenial kernel in trusty.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1568729

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jonas Keidel (jonas-keidel)]

Encountered the same issue on some machines while running Qemu 2.5 on lts-xenial kernel in trusty. The machine died with nearly the same calltrace as above and a very high load. Downgrading to the latest wily kernel fixed the issue.

[Tim Gardner (timg-tpi)]

Please try the test kernel at http://people.canonical.com/~rtg/4.4.0-fair-sched/

wget http://people.canonical.com/~rtg/4.4.0-fair-sched/linux-image-4.4.0-19-generic_4.4.0-19.35_amd64.deb
wget http://people.canonical.com/~rtg/4.4.0-fair-sched/linux-image-extra-4.4.0-19-generic_4.4.0-19.35_amd64.deb
sudo dpkg -i linux-image-4.4.0-19-generic_4.4.0-19.35_amd64.deb linux-image-extra-4.4.0-19-generic_4.4.0-19.35_amd64.deb

[Markus Schade (lp-markusschade)]

Thanks. Will test and report back in a few days.

[Markus Schade (lp-markusschade)]

Unfortunately, the issue is still present.

Apr 14 00:34:43 cnode17 kernel: [204922.475156] divide error: 0000 [#1] SMP
Apr 14 00:34:43 cnode17 kernel: [204922.475185] Modules linked in: cpuid arc4 md4 nls_utf8 cifs vhost_net vhost macvtap macvlan nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 xt_NFQUEUE xt_CLASSIFY ip6table_mangle sch_sfq sch_htb veth dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ebtable_filter ebtables nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables iptable_mangle xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter xt_CT iptable_raw ip_tables x_tables dummy bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache ipmi_ssif ipmi_devintf x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds sb_edac edac_core cdc_ether nf_nat_ftp usbnet mii nf_conntrack_ftp nf_nat_pptp lpc_ich nf_conntrack_pptp nf_nat_proto_gre nf_nat_sip nf_nat nf_conntrack_sip ioatdma ipmi_si shpchp ipmi_msghandler 8250_fintek nf_conntrack_proto_gre wmi acpi_pad nf_conntrack acpi_power_meter mac_hid bonding lp parport tcp_htcp xfs btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid ixgbe igb raid6_pq vxlan i2c_algo_bit libcrc32c ahci ip6_udp_tunnel dca udp_tunnel libahci raid1 ptp pps_core megaraid_sas raid0 mdio multipath fjes linear
Apr 14 00:34:43 cnode17 kernel: [204922.476423] CPU: 2 PID: 5905 Comm: ceph-osd Not tainted 4.4.0-19-generic #35
Apr 14 00:34:43 cnode17 kernel: [204922.476469] Hardware name: Dell Inc. PowerEdge R720/0W7JN5, BIOS 2.5.2 01/28/2015
Apr 14 00:34:43 cnode17 kernel: [204922.476538] task: ffff8822ea962940 ti: ffff8822eab44000 task.ti: ffff8822eab44000
Apr 14 00:34:43 cnode17 kernel: [204922.476606] RIP: 0010:[<ffffffff810b575c>] [<ffffffff810b575c>] task_numa_find_cpu+0x23c/0x710
Apr 14 00:34:43 cnode17 kernel: [204922.476682] RSP: 0000:ffff8822eab47bd8 EFLAGS: 00010206
Apr 14 00:34:43 cnode17 kernel: [204922.476723] RAX: 0000000000000000 RBX: ffff8822eab47c78 RCX: 0000000000000000
Apr 14 00:34:43 cnode17 kernel: [204922.476789] RDX: 0000000000000000 RSI: ffff88239f840000 RDI: ffff8823847efa00
Apr 14 00:34:43 cnode17 kernel: [204922.476856] RBP: ffff8822eab47c40 R08: 00000001030c9163 R09: 0000000000000262
Apr 14 00:34:43 cnode17 kernel: [204922.476923] R10: 0000000000000185 R11: 000000000001b936 R12: ffff88470dcf6e00
Apr 14 00:34:43 cnode17 kernel: [204922.477086] R13: 000000000000000d R14: 0000000000000089 R15: 0000000000000166
Apr 14 00:34:43 cnode17 kernel: [204922.477153] FS: 00007f86a7caa700(0000) GS:ffff88239f840000(0000) knlGS:0000000000000000
Apr 14 00:34:43 cnode17 kernel: [204922.477222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 14 00:34:43 cnode17 kernel: [204922.477265] CR2: 0000000021ba73f0 CR3: 0000002384352000 CR4: 00000000001426e0
Apr 14 00:34:43 cnode17 kernel: [204922.477331] Stack:
Apr 14 00:34:43 cnode17 kernel: [204922.477364] ffff8847706be600 ffff88479f016d70 00000000002619b8 ffff8822ea962940
Apr 14 00:3...

[Jonas Keidel (jonas-keidel)]

Here another call trace:

[Thu Apr 14 13:53:29 2016] divide error: 0000 [#1] SMP
[Thu Apr 14 13:53:29 2016] Modules linked in: cpuid arc4 md4 nls_utf8 cifs vhost_net vhost macvtap macvlan nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 xt_NFQUEUE xt_CLASSIFY ip6table_mangle sch_sfq sch_htb veth dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ebtable_filter ebtables xt_CT iptable_raw iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables dummy bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache ipmi_ssif ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel dcdbas kvm irqbypass crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac input_leds joydev edac_core nf_nat_ftp nf_conntrack_ftp cdc_ether usbnet nf_nat_pptp mii nf_conntrack_pptp lpc_ich nf_nat_proto_gre ioatdma nf_nat_sip nf_nat nf_conntrack_sip ipmi_si nf_conntrack_proto_gre 8250_fintek ipmi_msghandler shpchp nf_conntrack acpi_pad mac_hid wmi acpi_power_meter bonding lp parport tcp_htcp xfs btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic usbhid hid xor ixgbe raid6_pq libcrc32c igb vxlan raid1 i2c_algo_bit ip6_udp_tunnel ahci dca udp_tunnel raid0 libahci ptp multipath pps_core megaraid_sas mdio linear fjes
[Thu Apr 14 13:53:29 2016] CPU: 22 PID: 9581 Comm: ceph-osd Not tainted 4.4.0-19-generic #35
[Thu Apr 14 13:53:29 2016] Hardware name: Dell Inc. PowerEdge R720/0XH7F2, BIOS 2.5.2 01/28/2015
[Thu Apr 14 13:53:29 2016] task: ffff8822ce772940 ti: ffff882300380000 task.ti: ffff882300380000
[Thu Apr 14 13:53:29 2016] RIP: 0010:[<ffffffff810b57ed>] [<ffffffff810b57ed>] task_numa_find_cpu+0x2cd/0x710
[Thu Apr 14 13:53:29 2016] RSP: 0000:ffff882300383bd8 EFLAGS: 00010257
[Thu Apr 14 13:53:29 2016] RAX: 0000000000000000 RBX: ffff882300383c78 RCX: 0000000000000311
[Thu Apr 14 13:53:29 2016] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff882376c5ea00
[Thu Apr 14 13:53:29 2016] RBP: ffff882300383c40 R08: 0000000000000004 R09: 000000000000000d
[Thu Apr 14 13:53:29 2016] R10: 0000000000000294 R11: 0000000000000005 R12: ffff8822d8e62940
[Thu Apr 14 13:53:29 2016] R13: 0000000000000015 R14: ffff882376c5ea00 R15: 000000000000026b
[Thu Apr 14 13:53:29 2016] FS: 00007f5fac546700(0000) GS:ffff88239fac0000(0000) knlGS:0000000000000000
[Thu Apr 14 13:53:29 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu Apr 14 13:53:29 2016] CR2: 00007f5fac545b60 CR3: 0000002381838000 CR4: 00000000000406e0
[Thu Apr 14 13:53:29 2016] Stack:
[Thu Apr 14 13:53:29 2016] ffff882300383c38 ffffffff8102d66c ffff882300000001 0000000000000311
[Thu Apr 14 13:53:29 2016] 0000000000000081 00000000000001ad 0000000000016d00 0000000000000004
[Thu Apr 14 13:53:29 2016] ffff8822ce772940 ffff882300383c78 000000000000011d 00000000000001b3
[Thu Apr 14 13:53:29 2016] Call Trace:
[Thu Apr 14 13:53:29 2016] [<ffffffff8102d66c>] ? __switch_to+0x1dc/0x5a0
[Thu Apr 14 13:53:29 2016] [<ffffffff810b606e>] task_numa_migrate+0...

[Jonas Keidel (jonas-keidel)]

And again. This time with upstream kernel (linux-image-4.5.1-040501-generic):
[Fri Apr 15 13:26:56 2016] divide error: 0000 [#1] SMP
[Fri Apr 15 13:26:56 2016] Modules linked in: vhost_net vhost macvtap macvlan ip6table_mangle nfnetlink_queue nfnetlink xt_CLASSIFY xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 xt_NFQUEUE sch_sfq sch_htb veth dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ebtable_filter ebtables nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_CT iptable_raw xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter iptable_mangle ip_tables x_tables dummy bridge stp llc ipmi_ssif ipmi_devintf x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac input_leds joydev edac_core nf_nat_ftp cdc_ether usbnet nf_conntrack_ftp mii nf_nat_pptp lpc_ich nf_conntrack_pptp nf_nat_proto_gre ioatdma nf_nat_sip nf_nat nfsd ipmi_si nf_conntrack_sip ipmi_msghandler 8250_fintek nf_conntrack_proto_gre acpi_pad nf_conntrack wmi acpi_power_meter shpchp mac_hid auth_rpcgss nfs_acl bonding nfs lp lockd parport grace sunrpc fscache tcp_htcp xfs btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid igb raid6_pq ixgbe libcrc32c raid1 i2c_algo_bit vxlan ahci ip6_udp_tunnel dca raid0 libahci udp_tunnel ptp megaraid_sas pps_core multipath mdio fjes linear
[Fri Apr 15 13:26:56 2016] CPU: 10 PID: 9261 Comm: ceph-osd Not tainted 4.5.1-040501-generic #201604121331
[Fri Apr 15 13:26:56 2016] Hardware name: Dell Inc. PowerEdge R720/08RW36, BIOS 2.5.2 01/28/2015
[Fri Apr 15 13:26:56 2016] task: ffff8846b4611c80 ti: ffff8846b4704000 task.ti: ffff8846b4704000
[Fri Apr 15 13:26:56 2016] RIP: 0010:[<ffffffff810b5d3c>] [<ffffffff810b5d3c>] task_numa_find_cpu+0x23c/0x710
[Fri Apr 15 13:26:56 2016] RSP: 0000:ffff8846b4707bd8 EFLAGS: 00010206
[Fri Apr 15 13:26:56 2016] RAX: 0000000000000000 RBX: ffff8846b4707c78 RCX: 0000000000000000
[Fri Apr 15 13:26:56 2016] RDX: 0000000000000000 RSI: ffff88239f940000 RDI: ffff88237a510200
[Fri Apr 15 13:26:56 2016] RBP: ffff8846b4707c40 R08: 0000000101378ff0 R09: 0000000000000012
[Fri Apr 15 13:26:56 2016] R10: 00000000000000ee R11: 0000000000000003 R12: ffff8846b46b0e40
[Fri Apr 15 13:26:56 2016] R13: 0000000000000001 R14: 0000000000000000 R15: 00000000000000e0
[Fri Apr 15 13:26:56 2016] FS: 00007f95c52c8700(0000) GS:ffff88239f940000(0000) knlGS:0000000000000000
[Fri Apr 15 13:26:56 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Apr 15 13:26:56 2016] CR2: 000000002b75cc00 CR3: 00000047850e4000 CR4: 00000000001426e0
[Fri Apr 15 13:26:56 2016] Stack:
[Fri Apr 15 13:26:56 2016] ffff8846b4707c38 ffffffff8101667e 0000000100016b00 ffff8846b4611c80
[Fri Apr 15 13:26:56 2016] 000000000000014d fffffffffffffe53 0000000000016b00 000000000000014d
[Fri Apr 15 13:26:56 2016] ffff8846b4611c80 ffff8846b4707c78 00000000000002ca 000000000000014d
[Fri Apr 15 13:26:56 2016] Call Trace:
[Fri Apr 15 13:26:56 2016] [<ffffffff8101667e>] ? __switch_to+0x1de/0x5d0
[Fri Apr 15 13:26:56 2...

[Dongwon Cho (dongwoncho)]

dmesg -T
[Fri Jun 3 01:07:11 2016] divide error: 0000 [#1] SMP
[Fri Jun 3 01:07:11 2016] Modules linked in: iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat 8021q garp mrp binfmt_misc veth vhost_net vhost macvtap macvlan ebtable_filter ebtables ip6table_filter ip6_tables openvswitch nf_defrag_ipv6 nf_conntrack xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables bonding zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) ipmi_ssif ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal coretemp sb_edac edac_core mei_me mei shpchp ipmi_si ipmi_msghandler 8250_fintek lpc_ich mac_hid acpi_power_meter kvm_intel kvm irqbypass ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
[Fri Jun 3 01:07:11 2016] raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul igb aesni_intel aes_x86_64 lrw dca gf128mul glue_helper ptp ahci ablk_helper pps_core mxm_wmi cryptd libahci megaraid_sas i2c_algo_bit fjes wmi
[Fri Jun 3 01:07:11 2016] CPU: 4 PID: 4915 Comm: vhost-4913 Tainted: P O 4.4.0-22-generic #40-Ubuntu
[Fri Jun 3 01:07:11 2016] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 2.0.2 03/15/2016
[Fri Jun 3 01:07:11 2016] task: ffff8807e6a80000 ti: ffff880046a3c000 task.ti: ffff880046a3c000
[Fri Jun 3 01:07:11 2016] RIP: 0010:[<ffffffff810b593d>] [<ffffffff810b593d>] task_numa_find_cpu+0x2cd/0x710
[Fri Jun 3 01:07:11 2016] RSP: 0018:ffff880046a3f7d8 EFLAGS: 00010257
[Fri Jun 3 01:07:11 2016] RAX: 0000000000000000 RBX: ffff880046a3f878 RCX: 0000000000000001
[Fri Jun 3 01:07:11 2016] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880dc42efa00
[Fri Jun 3 01:07:12 2016] RBP: ffff880046a3f840 R08: 0000000000000001 R09: 0000000000aaaaaa
[Fri Jun 3 01:07:12 2016] R10: 0000000000000335 R11: 0000000000000000 R12: ffff880e2fbeee00
[Fri Jun 3 01:07:12 2016] R13: 0000000000000001 R14: ffff880dc42efa00 R15: 0000000000000335
[Fri Jun 3 01:07:12 2016] FS: 0000000000000000(0000) GS:ffff88085e680000(0000) knlGS:0000000000000000
[Fri Jun 3 01:07:12 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Jun 3 01:07:12 2016] CR2: 00007f442814d400 CR3: 0000000e6d094000 CR4: 00000000001426e0
[Fri Jun 3 01:07:12 2016] Stack:
[Fri Jun 3 01:07:12 2016] 00000000d08a1709 ffff8807e6a806ac ffff8807e6a80000 0000000000000001
[Fri Jun 3 01:07:12 2016] 0000000000000335 000000000000030d 0000000000016d00 0000000000000001
[Fri Jun 3 01:07:12 2016] ffff8807e6a80000 ffff880046a3f878 000000000000006d 0000000000000059
[Fri Jun 3 01:07:12 2016] Call Trace:
[Fri Jun 3 01:07:12 2016] [<ffffffff810b61be>] task_numa_migrate+0x43e/0x9b0
[Fri Jun 3 01:07:12 2016] [<ffffffff810b67a9>] numa_migrate_preferred+0x79/0x80
[Fri Jun 3 01:07:12 2016] [<ffffffff810badc4>] task_numa_fault+0x7f4/0xd40
[Fri Jun 3 01:07:12 2016] [<ffffffff810ba435>] ? should_numa_migrate_memory+0x55/0x130
[Fri Jun 3 01:07:13 2016] [<ffffffff811bf860>] handle_mm_fault+0xbc0/0x1820
[Fri Jun 3 01:07:13 2016] [<ffffffff8105a50e>] ? physflat_send...

[youshotwhointhatwhatnow (moloney-brendan)]

There is one person on the Ceph mailing list who thinks this is fixed in 4.7rc6 (http://thread.gmane.org/gmane.comp.file-systems.ceph.user/30793/focus=30987). Unfortunately, I haven't been able to figure out a precise patch set that can be applied to 4.4 to fix it.

[Peter Sabaini (peter-sabaini)]

On a system running 4.4.0-28-generic I get something similar-looking:

foonode kernel: [595908.569972] divide error: 0000 [#1] SMP
foonode kernel: [595908.571257] Modules linked in: ip6table_raw ip6table_mangle nf_conntrack_ipv6 xt_CT xt_connmark xt_mac xt_comment xt_physdev br_netfilter xt_set xt_multiport ip_set_hash_net ip_set nfnetlink veth iptable_raw xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vport_gre ip_gre ip_tunnel gre openvswitch nf_defrag_ipv6 nf_conntrack dm_crypt bonding ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp ipmi_ssif coretemp kvm_intel dcdbas kvm irqbypass dm_multipath joydev input_leds sb_edac mei_me edac_core shpchp mei lpc_ich ipmi_si 8250_fintek ipmi_msghandler mac_hid acpi_power_meter xfs libcrc32c btrfs xor raid6_pq bcache crct10dif_pclmul crc32_pclmul ixgbe hid_generic igb vxlan ip6_udp_tunnel usbhid aesni_intel udp_tunnel dca aes_x86_64 ptp lrw gf128mul glue_helper ablk_helper hid pps_core cryptd i2c_algo_bit mdio megaraid_sas wmi fjes
foonode kernel: [595908.607022] CPU: 30 PID: 3173122 Comm: ms_pipe_write Not tainted 4.4.0-28-generic #47~14.04.1-Ubuntu
foonode kernel: [595908.609874] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 1.0.4 08/28/2014
foonode kernel: [595908.612265] task: ffff881efbe78000 ti: ffff8816f3410000 task.ti: ffff8816f3410000
foonode kernel: [595908.614601] RIP: 0010:[<ffffffff810aff78>] [<ffffffff810aff78>] task_numa_find_cpu+0x238/0x700
foonode kernel: [595908.617346] RSP: 0000:ffff8816f3413bb0 EFLAGS: 00010257
foonode kernel: [595908.619007] RAX: 0000000000000000 RBX: ffff8816f3413c50 RCX: 0000000000000000
foonode kernel: [595908.733387] RDX: 0000000000000000 RSI: ffff881ffefc0000 RDI: ffff881ffefd6d70
foonode kernel: [595908.852154] RBP: ffff8816f3413c18 R08: 0000000108dfe960 R09: 0000000000000042
foonode kernel: [595908.973271] R10: 000000000000001e R11: 00000000000001dc R12: ffff883e649ea940
foonode kernel: [595909.093668] R13: 0000000000000019 R14: 000000000000011f R15: 00000000000001bf
foonode kernel: [595909.213849] FS: 00007fd7eb7f6700(0000) GS:ffff881ffefc0000(0000) knlGS:0000000000000000
foonode kernel: [595909.334056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
foonode kernel: [595909.394132] CR2: 0000562170fdde00 CR3: 00000025c0ba8000 CR4: 00000000001406e0
foonode kernel: [595909.512605] Stack:
foonode kernel: [595909.569642] 000000304e62e962 0000000000000054 00000000ffffffff ffff881efbe78000
foonode kernel: [595909.684440] 0000000000000039 0000000000000181 0000000000016d00 0000000000000039
foonode kernel: [595909.798807] ffff881efbe78000 00000000000001d7 ffff8816f3413c50 0000000000000133
foonode kernel: [595909.913067] Call Trace:
foonode kernel: [595909.968501] [<ffffffff810b08e0>] task_numa_migrate+0x4a0/0x930
foonode kernel: [595910.024588] [<ffffffff816d5217>] ? release_sock+0x117/0x160
foo...

[Stefan Bader (smb)]

Not sure which patch(es) Tim had in the test kernel. Following the various leads from the thread on the ceph mailing list from comment #9 it might be that the 3 patches to pick might be:

2b8c41d sched/fair: Initiate a new task's util avg to a bounded value
b7fa30c sched/fair: Fix post_init_entity_util_avg() serialization
8974189 sched/fair: Fix cfs_rq avg tracking underflow

[Stefan Bader (smb)]

A new set of test kernel packages can be found at: http://people.canonical.com/~smb/lp1568729/ (those include backports of the three patches mentioned above). If someone could check whether that helps? Thanks.

[youshotwhointhatwhatnow (moloney-brendan)]

According to this message in the ceph thread those three patches are not sufficient:

https://<email address hidden>/msg30390.html

In a follow up Stefan Priebe mentions he has about 20 other patches applied, and that must have contributed to having the problem solved on his systems:

https://<email address hidden>/msg30392.html

[Stefan Bader (smb)]

The whole discussion seems to be going back and forth and is rather confusing. On one side there is the ceph discussion where Stefan Priebe indeed mentions that he has many more patches in his tree. On the other side there is the LKML discussion which ends in GregKH getting exactly one patch submitted for 4.4 and 4.6 (https://lkml.org/lkml/2016/7/12/297).
This is the backport of the last of the 3 patches in comment #11. I am not sure whether Peter Zijlstra just as well got confused in the end or it turned out that in older kernels one only needed the last one. From the references in the patches, the last one refers to the second and the second refers to the first. Practically the last one works around certain values becoming 0 in certain corner cases. Which causes the divide errors. So maybe it is enough.

As for the last test kernel done by Tim: although is answer when I asked about what was in there sounded like it might be the same set, I am starting to doubt that from the time-line. That kernel was done in April and all three patches I had were 4.7-rc1 or later which looks to be from May. So I cannot really believe that those were in discussion back then.

So what I would like to propose: I created a PPA and uploaded a kernel with just the last patch applied (https://launchpad.net/~smb/+archive/ubuntu/xenial/+packages). This is the state that is queued upstream for 4.4.x longterm (but not yet released). Someone who can see the issue should actually test this kernel. Regardless of any reports on the ceph list. Because if this does not fix the problem this should be discussed upstream.

[Stefan Bader (smb)]

For reference, this is the one patch picked from upstream stable queue for 4.4.

[David Medberry (med)]

Stefan,
Can you add a trusty (Xenial HWE) version of this to the PPA? I'm trying a rebuild of this as part of a Trusty PPA and getting a failure.

[Peter Sabaini (peter-sabaini)]

We had something similar-looking again, filed as Bug #1606098

[Stefan Bader (smb)]

Added a linux-lts-xenial variant to the PPA.

[Markus Schade (lp-markusschade)]

I have been running the -29 kernel from http://people.canonical.com/~smb/lp1568729/ for about two weeks now, so far without triggering the bug. With the kernel from Tim, the bug could still be triggered. So it seems there were different patches.

I'll try the -33 kernel from ppa as well, to see if there is any difference.

[Piotr Parczewski (pparczewski)]

Same issue with 4.4.0-31-generic, stack trace:

[443830.036000] divide error: 0000 [#1] SMP
[443830.036583] Modules linked in: nf_conntrack_netlink xt_multiport xt_CT xt_mac xt_physdev xt_set ip_set_hash_net ip_set nfnetlink vhost_net vhost macvtap macvlan xt_REDIRECT nf_nat_redirect xt_mark vport_vxlan xt_CHECKSUM ip6table_raw nf_conntrack_ipv6 ip6table_mangle xt_connmark xt_comment iptable_raw iptable_mangle dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag ebtable_filter ebtables ip6table_filter ip6_tables ip_vs openvswitch nf_defrag_ipv6 xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge aufs mpt3sas raid_class scsi_transport_sas mptctl mptbase binfmt_misc bonding xfs nls_iso8859_1 intel_rapl ipmi_ssif joydev input_leds x86_pkg_temp_thermal
[443830.039865] intel_powerclamp coretemp sb_edac mei_me mei edac_core lpc_ich ioatdma ipmi_si ipmi_msghandler shpchp 8250_fintek acpi_power_meter mac_hid kvm_intel acpi_pad kvm irqbypass ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp mrp stp llc sunrpc autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure hid_generic crct10dif_pclmul ast crc32_pclmul i2c_algo_bit ttm ixgbe drm_kms_helper aesni_intel dca aes_x86_64 vxlan lrw ip6_udp_tunnel gf128mul syscopyarea glue_helper udp_tunnel ablk_helper sysfillrect cryptd usbhid sysimgblt fb_sys_fops hid ahci ptp drm libahci megaraid_sas pps_core mdio wmi fjes
[443830.045336] CPU: 10 PID: 13866 Comm: ceph-osd Not tainted 4.4.0-31-generic #50-Ubuntu
[443830.046219] Hardware name: Supermicro PIO-628U-TR4T+-ST031/X10DRU-i+, BIOS 2.0 12/17/2015
[443830.047112] task: ffff881fc1260dc0 ti: ffff881fc126c000 task.ti: ffff881fc126c000
[443830.048073] RIP: 0010:[<ffffffff810b5adc>] [<ffffffff810b5adc>] task_numa_find_cpu+0x23c/0x710
[443830.049023] RSP: 0000:ffff881fc126fbd8 EFLAGS: 00010206
[443830.050042] RAX: 0000000000000000 RBX: ffff881fc126fc78 RCX: 0000000000000000
[443830.051023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff881fef0c0800
[443830.052067] RBP: ffff881fc126fc40 R08: 00000001069be1a6 R09: 0000000000000015
[443830.053080] R10: 00000000000003d2 R11: 0000000000000df4 R12: ffff883b2e116e00
[443830.054174] R13: 000000000000000c R14: 0000000000000000 R15: fffffffffffffca6
[443830.055146] FS: 00007fe71a644700(0000) GS:ffff881fffa80000(0000) knlGS:0000000000000000
[443830.056197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[443830.057206] CR2: 000000000aea4980 CR3: 0000001feed55000 CR4: 00000000001426e0
[443830.058296] Stack:
[443830.059327] ffff881fc126fbf8 ffffffff813f0f4f 0000000000000100 ffff881fc1260dc0
[443830.060462] 0000000000000063 0000000000000129 0000000000016d00 0000000000000063
[443830.061607] ffff881fc1260dc0 ffff881fc126fc78 000000000000015f 00000000000001c2
[443830.062721] Call Trace:
[443830.063831] [<ffffffff813f0f4f>] ? cpumask_next_and+0x2f/0x40
[443830.064965] [<ffffffff810b63ee>]...

[Markus Schade (lp-markusschade)]

But that is not the -31 from the PPA. Could you please try the kernel from the PPA that Stefan has posted?

[chrone (chrone81)]

Just happened two days ago on Trusty 14.04.4 with Linux 4.4.0-31 on one of Ceph Jewel OSD server. It ran fine for 8 days though and suddenly the CPU load spiked to 600.

The server is from SuperMicro SuperStorage Server SSG-6048R-E1CR36L with these following specs:
2x Intel Xeon E5-2630 v3 @ 2.4GHz 16C/32T
128GB DDR3 ECC
2x 80GB Intel SSD S3500 series for OS Drive in mdraid1 mode
2x 800GB Intel PCIe SSD S3700 series for Ceph OSD Journal
36x 6TB Samsung NAS 7200rpm SAS drives for Ceph OSDs
4x 10GbE SFP+ Intel 82599ES ethernet with LACP bonding mode

The apport-collect log is attached. Hope this helps.

/chrone

[Bryan Quigley (bryanquigley)]

The fixes mentioned are also part of the 4.4.16 stable series, which will land in the next kernel cycle (August 29th ish). That should be 4.4.0-35 (or higher) in Ubuntu kernel versions.
See- https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1607404

[Markus Schade (lp-markusschade)]

The -31 Kernel from Stefan's PPA with the patch has been running stable for 2 weeks and the previous version based on -29 for 3 weeks. So if that patch will make its way into Ubuntu kernel via the 4.4.16 stable series update, the bug can be closed from my side at that point

[Philipp Noack (philipp-noack-b)]

Had the same divide error and a hanging server myself. Ubuntu 16.04.1 LTS with "linux-image-4.4.0-31-generic". Now running "linux-image-4.4.0-34-generic".

[Stefan Bader (smb)]

The required change will be in the 4.4.0-36 (includes the 4.4.0-35 updates) which is currently in -proposed and waiting for verification and regression testing to finish. If there are no problems found this is supposed to get released by Aug-29th.

https://launchpad.net/ubuntu/+source/linux/4.4.0-36.55

[Bert JW Regeer (bregeer-ctl)]

Just ran into this issue on Ubuntu 16.04:

Aug 21 02:49:14 doddering-fransisca kernel: [2532635.918673] divide error: 0000 [#1] SMP
Aug 21 02:49:14 doddering-fransisca kernel: [2532635.935386] Modules linked in: bridge stp llc bonding nls_iso8859_1 ipmi_ssif ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass sb_edac edac_core 8250_fintek ipmi_si ipmi_msghandler mei_me mei lpc_ich shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 crct10dif_pclmul crc32_pclmul mxm_wmi aesni_intel aes_x86_64 lrw ixgbe i40e gf128mul glue_helper igb ablk_helper vxlan cryptd ip6_udp_tunnel udp_tunnel dca mdio ahci ptp libahci pps_core i2c_algo_bit fjes wmi
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.036341] CPU: 20 PID: 95997 Comm: jujud Not tainted 4.4.0-31-generic #50-Ubuntu
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.052997] Hardware name: Dell Inc. PowerEdge C6320/082F9M, BIOS 2.0.2 02/16/2016
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.069486] task: ffff883f0cbd9b80 ti: ffff883f5ae5c000 task.ti: ffff883f5ae5c000
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.085837] RIP: 0010:[<ffffffff810b5b6d>] [<ffffffff810b5b6d>] task_numa_find_cpu+0x2cd/0x710
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.102963] RSP: 0000:ffff883f5ae5fbd8 EFLAGS: 00010257
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.118987] RAX: 0000000000000000 RBX: ffff883f5ae5fc78 RCX: 0000000000000414
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.134881] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88547de36200
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.150581] RBP: ffff883f5ae5fc40 R08: 000000000000012a R09: 0000000000075d4a
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.166034] R10: 0000000000000084 R11: 000000000003eab1 R12: ffff887c1fc6e040
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.181262] R13: 000000000000001b R14: ffff88547de36200 R15: 000000000000007c
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.196284] FS: 00007f0888ff9700(0000) GS:ffff883f7e880000(0000) knlGS:0000000000000000
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.211275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.225943] CR2: 000000d884976e48 CR3: 0000003f56296000 CR4: 00000000003406e0
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.240452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.254703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.268649] Stack:
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.282271] 000000000000001d 0000000000000100 000000000000012a 0000000000000414
Aug 21 02:49:14 doddering-fransisca kernel: [2532636.295952] 0000000000000065 00000000000000c3 0000000000016d00 000000000000012a
Aug 21 02:49:14 doddering-fransisca...

[David Medberry (med)]

So jujud exercises the same code path as ceph-osd? curious....

[Stefan Bader (smb)]

The new Xenail/16.04 kernel should now be in updates.