Server reboots every 4.1 weeks

[emmecerre]

Every 4.1 uptime weeks (more or less), our 34 servers reboots with the logs below.

Description: Ubuntu 16.04.1 LTS
Release: 16.04

The servers hosts the stack flanneld (0.5.5) docker (1.11.2, build b9f10c9) kubernetes (v1.3.6) plus etcd (2.3.7)

Jan 02 06:40:32 prd-node021 kernel: ------------[ cut here ]------------
Jan 02 06:40:32 prd-node021 kernel: kernel BUG at /build/linux-xHzv4a/linux-4.4.0/include/linux/fs.h:2569!
Jan 02 06:40:32 prd-node021 kernel: invalid opcode: 0000 [#1] SMP
Jan 02 06:40:32 prd-node021 kernel: Modules linked in: nf_conntrack_netlink nfnetlink veth xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_tcpudp tcp_diag inet_diag
Jan 02 06:40:32 prd-node021 kernel: raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc
Jan 02 06:40:32 prd-node021 kernel: CPU: 46 PID: 22749 Comm: iptables-restor Not tainted 4.4.0-47-generic #68-Ubuntu
Jan 02 06:40:32 prd-node021 kernel: Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 09/13/2016
Jan 02 06:40:32 prd-node021 kernel: task: ffff882f7ccb44c0 ti: ffff882fcb810000 task.ti: ffff882fcb810000
Jan 02 06:40:32 prd-node021 kernel: RIP: 0010:[<ffffffff8120f9ed>] [<ffffffff8120f9ed>] __fput+0x21d/0x220
Jan 02 06:40:32 prd-node021 kernel: RSP: 0018:ffff882fcb813e68 EFLAGS: 00010246
Jan 02 06:40:32 prd-node021 kernel: RAX: 0000000000000000 RBX: ffff8829dbd92700 RCX: 00000000308c6396
Jan 02 06:40:32 prd-node021 kernel: RDX: 0000000000000001 RSI: ffff88301f299f60 RDI: 0000000000000000
Jan 02 06:40:32 prd-node021 kernel: RBP: ffff882fcb813ea0 R08: 0000000000019f60 R09: ffffffff811b3a1d
Jan 02 06:40:32 prd-node021 kernel: R10: ffffea0064756680 R11: ffff8829dbd92710 R12: 0000000000000010
Jan 02 06:40:32 prd-node021 kernel: R13: ffff8817d3520518 R14: ffff881021a34da0 R15: ffff8817d3542540
Jan 02 06:40:32 prd-node021 kernel: FS: 00007fcdaa753700(0000) GS:ffff88301f280000(0000) knlGS:0000000000000000
Jan 02 06:40:32 prd-node021 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 02 06:40:32 prd-node021 kernel: CR2: 00007fcdaa758000 CR3: 0000001917993000 CR4: 00000000003406e0
Jan 02 06:40:32 prd-node021 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 02 06:40:32 prd-node021 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 02 06:40:32 prd-node021 kernel: Stack:
Jan 02 06:40:32 prd-node021 kernel: ffff8817d3520518 ffff8829dbd92710 ffff882f7ccb44c0 ffffffff82103a30
Jan 02 06:40:32 prd-node021 kernel: ffff8829dbd92700 0000000000000000 ffff882f7ccb4b38 ffff882fcb813eb0
Jan 02 06:40:32 prd-node021 kernel: ffffffff8120fa2e ffff882fcb813ef0 ffffffff8109ee01 ffff882f7ccb4b6c
Jan 02 06:40:32 prd-node021 kernel: Call Trace:
Jan 02 06:40:32 prd-node021 kernel: [<ffffffff8120fa2e>] ____fput+0xe/0x10
Jan 02 06:40:32 prd-node021 kernel: [<ffffffff8109ee01>] task_work_run+0x81/0xa0
Jan 02 06:40:32 prd-node021 kernel: [<ffffffff81003242>] exit_to_usermode_loop+0xc2/0xd0
Jan 02 06:40:32 prd-node021 kernel: [<ffffffff81003c6e>] syscall_return_slowpath+0x4e/0x60
Jan 02 06:40:32 prd-node021 kernel: [<ffffffff81835150>] int_ret_from_sys_call+0x25/0x8f
Jan 02 06:40:32 prd-node021 kernel: Code: 0f 84 cf fe ff ff 48 8b 43 28 48 8b 80 80 00 00 00 48 85 c0 0f 84 bb fe ff ff 31 d2 48 89 de bf ff ff ff ff ff d0 e9 aa fe ff ff <0f>
Jan 02 06:40:32 prd-node021 kernel: RIP [<ffffffff8120f9ed>] __fput+0x21d/0x220
-- Reboot --
Jan 02 06:42:56 prd-node021 systemd-journald[819]: Runtime journal (/run/log/journal/) is 8.0M, max 1.8G, 1.8G free.
Jan 02 06:42:56 prd-node021 kernel: Initializing cgroup subsys cpuset

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-47-generic 4.4.0-47.68
ProcVersionSignature: Ubuntu 4.4.0-47.68-generic 4.4.24
Uname: Linux 4.4.0-47-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 2 06:42 seq
 crw-rw---- 1 root audio 116, 33 Jan 2 06:42 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Mon Jan 2 10:49:15 2017
HibernationDevice: RESUME=/dev/mapper/vg00-swap
InstallationDate: Installed on 2016-09-13 (110 days ago)
InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Beta amd64 (20160803)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
PciMultimedia:

ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-47-generic root=/dev/mapper/vg00-root ro cgroup_enable=memory swapaccount=1
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-47-generic N/A
 linux-backports-modules-4.4.0-47-generic N/A
 linux-firmware 1.157.5
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/13/2016
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.board.name: ProLiant DL360 Gen9
dmi.board.vendor: HP
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP89:bd09/13/2016:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL360 Gen9
dmi.sys.vendor: HP

[Brad Figg (brad-figg)]

This change was made by a bot.

[Colin Ian King (colin-king)]

A possible interesting data point is that /proc/interrupts CAL interrupts (Function call interrupts) are reaching a 32 bit wrap around.

[emmecerre (manuelcarlo-ranieri)]

I thought about this, but I've no idea which buffer fills

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.10 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc2

[emmecerre (manuelcarlo-ranieri)]

Hi Joseph Salisbury
I installed the upstream kernel but this kernel AUFS miss the support that's needed by our stack.
I also tried to compile the kernel with the upstream AUFS but the online AUFS is too old and patches do not apply correctly and the process goes in error.

I dunno if make sense to try some older kernel v4.8 or v4.9 (if the compile works... )

[emmecerre (manuelcarlo-ranieri)]

In the meantime, I compiled the upstream 4.9.2
kernel + ubuntu patches + aufs
Let me know if make sense the test with this kernel version

[From-Nibly (thatoneemail)]

What is the latest on this. We are running into a similar problem with kubernetes only the most recent occurrence was in 13 days. Did upgrading the kernel to 4.9.2 work?

[emmecerre (manuelcarlo-ranieri)]

We installed the 4.9.2 kernel a few days ago. We are still waiting for crash :)
For now, we have more or less 3.1 weeks of uptime.
I'll keep you posted.

[emmecerre (manuelcarlo-ranieri)]

The server with kernel 4.9.2 reboots after 2.874 weeks.

[emmecerre (manuelcarlo-ranieri)]

kernel-bug-exists-upstream

[Andrii (angr)]

Hello,

We have the same issue on our k8s cluster.

Description: Ubuntu 16.04.3 LTS
Release: 16.04

Docker: 17.03.2-ce
Kubernetes: v1.10.11
Kernel: 4.4.0-145-generic, 4.4.0-109-generic

[5493817.706210] ------------[ cut here ]------------
[5493817.707602] kernel BUG at /build/linux-6VmqmP/linux-4.4.0/include/linux/fs.h:2585!
[5493817.708284] invalid opcode: 0000 [#1] SMP
[5493817.708935] Modules linked in: xt_set xt_multiport iptable_raw iptable_mangle ip_set_hash_net veth nf_conntrack_netlink xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_set nfnetlink ip_vs ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs vmw_balloon vmw_vsock_vmci_transport vsock joydev input_leds serio_raw shpchp i2c_piix4 vmw_vmci mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
[5493817.714295] async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel vmwgfx aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ttm psmouse drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops vmxnet3 drm vmw_pvscsi ahci libahci pata_acpi fjes
[5493817.716942] CPU: 5 PID: 13487 Comm: iptables-save Not tainted 4.4.0-145-generic #171-Ubuntu
[5493817.717597] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
[5493817.718885] task: ffff8800bb16d400 ti: ffff88045f38c000 task.ti: ffff88045f38c000
[5493817.719627] RIP: 0010:[<ffffffff8121df13>] [<ffffffff8121df13>] __fput+0x223/0x230
[5493817.720314] RSP: 0018:ffff88045f38fe78 EFLAGS: 00010246
[5493817.721030] RAX: 0000000000000000 RBX: ffff88005f20e200 RCX: 00000000b86bfbc9
[5493817.721714] RDX: 0000000000000001 RSI: ffffe8ffffd51350 RDI: 0000000000000000
[5493817.722383] RBP: ffff88045f38feb0 R08: 000060f7c0011350 R09: ffffffff811beded
[5493817.723038] R10: ffffea000a9f3f00 R11: ffff88005f20e210 R12: 0000000000000010
[5493817.723695] R13: ffff880816bd11a8 R14: ffff8808130a1520 R15: ffff880816aaed80
[5493817.724371] FS: 00007f6087f8b700(0000) GS:ffff88083fd40000(0000) knlGS:0000000000000000
[5493817.725147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[5493817.725835] CR2: 00007f6085f6d1f0 CR3: 00000004621e2000 CR4: 0000000000060670
[5493817.726518] Stack:
[5493817.727156] ffff880816bd11a8 ffff88005f20e210 ffffffff82113f30 ffff8800bb16daa8
[5493817.727806] ffff8800bb16d400 0000000000000000 ffff8800bb16d400 ffff88045f38fec0
[5493817.728508] ffffffff8121df5e ffff88045f38fef0 ffffffff810a4895 0000000000000002
[5493817.729138] Call Trace:
[5493817.729845] [<ffffffff8121df5e>] ____fput+0xe/0x10
[5493817.730511] [<ffffffff810a4895>] task_work_run+0x95/0xb0
[5493817.731222] [<ffffffff81003532>] exit_to_usermode_loop+0xc2/0xd0
[5493817.731875] [<ffffffff81003c7e>...

[Andrii (angr)]

The similar problem is described on github: https://github.com/kubernetes/kubernetes/issues/70229

[chudihuang (chudihuang)]

Hi,

We have the same issue on our k8s cluster.

Description: ubuntu16.04.1 LTSx86_64
Release: 16.04.1

Kernel: 4.4.0-104-generic

the dump file can be downloaded via following way:
wget http://129.226.115.161/dump.202006231820.tar.gz

I did some analysis, however i still didnot find the root cause:

Load the vmcore in crash (please refer to the hyperlink above). Crash should present details similar to the following:
crash> bt
PID: 11388 TASK: ffff880eb1f79e00 CPU: 29 COMMAND: "heartbeat"
 #0 [ffff8809131a7b08] machine_kexec at ffffffff8105c22b
 #1 [ffff8809131a7b68] crash_kexec at ffffffff8110e852
 #2 [ffff8809131a7c38] oops_end at ffffffff81031c49
 #3 [ffff8809131a7c60] die at ffffffff810320fb
 #4 [ffff8809131a7c90] do_trap at ffffffff8102f121
 #5 [ffff8809131a7ce0] do_error_trap at ffffffff8102f4a9
 #6 [ffff8809131a7da0] do_invalid_op at ffffffff8102fa10
 #7 [ffff8809131a7db0] invalid_op at ffffffff8184638e
    [exception RIP: __fput+541]
    RIP: ffffffff812126ad RSP: ffff8809131a7e68 RFLAGS: 00010246
    RAX: 0000000000000000 RBX: ffff880ef6915700 RCX: 0000000365fb1705
    RDX: 0000000000000001 RSI: ffff880fff55a020 RDI: 0000000000000000
    RBP: ffff8809131a7ea0 R8: 000000000001a020 R9: ffffffff811b591d
    R10: ffffea002b69b300 R11: ffff880ef6915710 R12: 0000000000000010
    R13: ffff880ed152aef8 R14: ffff8800bba18aa0 R15: ffff880ed1513a40
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #8 [ffff8809131a7e60] __fput at ffffffff812125ac
 #9 [ffff8809131a7ea8] ____fput at ffffffff812126ee
#10 [ffff8809131a7eb8] task_work_run at ffffffff8109f101
#11 [ffff8809131a7ef8] exit_to_usermode_loop at ffffffff81003242
#12 [ffff8809131a7f30] syscall_return_slowpath at ffffffff81003c6e
#13 [ffff8809131a7f50] int_ret_from_sys_call at ffffffff818449d0
    RIP: 000000000047f704 RSP: 000000c423b77c98 RFLAGS: 00000246
    RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000047f704
    RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000000000ca
    RBP: 000000c423b77ce0 R8: 0000000000000000 R9: 0000000000000000
    R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
    R13: 0000000000000000 R14: 000000c423b78ee0 R15: 0000000000000008
    ORIG_RAX: 0000000000000003 CS: 0033 SS: 002b
crash>

crash> log
[19156101.592212] ------------[ cut here ]------------
[19156101.593103] kernel BUG at /build/linux-SwhOyu/linux-4.4.0/include/linux/fs.h:2582!
[19156101.594385] invalid opcode: 0000 [#1] SMP
[19156101.595083] Modules linked in: binfmt_misc af_packet_diag netlink_diag dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag veth br_netfilter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_set xt_mark ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport ip_set dummy xt_comment xt_addrtype iptable_nat nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_tcpudp bridge stp llc nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack aufs isofs ppdev crct10dif_pclmul parport_pc crc32_pclmul input_leds joydev ghash_clmulni_intel parport serio_raw ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ...