Client (1Gbps Link):
[ 3038.818986] nfs: server 10.0.0.200 not responding, timed out
[ 3038.818991] nfs: server 10.0.0.200 not responding, timed out
[ 3038.818996] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819001] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819006] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819012] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819017] nfs: server 10.0.0.200 not responding, timed out
[ 3038.958559] nfs: server 10.0.0.200 not responding, timed out
Server (10Gbps NIC)
Full of these errors:
[114310.563718] RPC request reserved 156 but used 212
[114310.565495] RPC request reserved 156 but used 176
[114310.569816] RPC request reserved 156 but used 176
[114310.576001] RPC request reserved 156 but used 176
[114310.580087] RPC request reserved 156 but used 176
[115206.967835] RPC request reserved 156 but used 212
[115206.981548] RPC request reserved 156 but used 176
[115811.134896] RPC request reserved 156 but used 176
[115811.136346] RPC request reserved 156 but used 176
All machines are Dell R410s and Dell R430s with Broadcom NICs.
Still happening in 14.04 LTS
Client (1Gbps Link):
[ 3038.818986] nfs: server 10.0.0.200 not responding, timed out
[ 3038.818991] nfs: server 10.0.0.200 not responding, timed out
[ 3038.818996] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819001] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819006] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819012] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819017] nfs: server 10.0.0.200 not responding, timed out
[ 3038.958559] nfs: server 10.0.0.200 not responding, timed out
Pings are under 1ms
Crash: 540>] 0xffffea00084c2540 netlink nf_conntrack nfnetlink intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw ipmi_devintf serio_raw joydev gf128mul glue_helper dcdbas ablk_helper i7core_edac acpi_power_meter gpio_ich lpc_ich ipmi_si edac_core cryptd ipmi_msghandler shpchp mac_hid lp parport tcp_htcp hid_generic mptsas mptscsih usbhid mptbase psmouse hid scsi_transport_sas pata_acpi bnx2 ffffea00084c254 0>] [<ffffea00084c2 540>] 0xffffea00084c2540 207a40 EFLAGS: 00010282 0(0000) GS:ffff88102fc6 0000(0000) knlGS:000000000 0000000 751>] ? shrink_ page_list+ 0x241/0xaa0 645>] shrink_ inactive_ list+0x1c5/ 0x560 343>] shrink_ lruvec+ 0x523/0x710 5ac>] shrink_ zone+0x7c/ 0x1b0 6e5>] balance_ pgdat+0x3b5/ 0x620 aab>] kswapd+0x15b/0x3f0 0e0>] ? prepare_ to_wait_ event+0x100/ 0x100 950>] ? balance_ pgdat+0x620/ 0x620 5a2>] kthread+0xd2/0xf0 4d0>] ? kthread_ create_ on_node+ 0x1c0/0x1c0 2d8>] ret_from_ fork+0x58/ 0x90 4d0>] ? kthread_ create_ on_node+ 0x1c0/0x1c0 540>] 0xffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.799988] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.815847] BUG: unable to handle kernel paging request at ffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.824363] IP: [<ffffea00084c2
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.832785] PGD 82fff5067 PUD 82fff4067 PMD 80000008176001e3
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.841143] Oops: 0011 [#2] SMP
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.849239] Modules linked in: mptctl xt_comment iptable_filter xt_multiport ip_tables x_tables rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nf_conntrack_
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.919101] CPU: 6 PID: 210 Comm: kswapd0 Tainted: G D W 3.16.0-51-generic #69~14.04.1-Ubuntu
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.937875] Hardware name: Dell Inc. PowerEdge R410/01V648, BIOS 1.6.3 02/07/2011
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.957034] task: ffff8808043c1e90 ti: ffff880802204000 task.ti: ffff880802204000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.976975] RIP: 0010:[<
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.997565] RSP: 0018:ffff880802
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.007973] RAX: ffff8807f94c8848 RBX: ffff880802207db0 RCX: 0000000000000000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.028679] RDX: ffffea00084c2540 RSI: 0000000000000002 RDI: ffffea001623df80
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.049907] RBP: ffff880802207b40 R08: ffff880002d078e8 R09: ffff880005eaf478
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.071886] R10: ffff8808022079c8 R11: ffffea003f7e0980 R12: ffffea002f1b4b60
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.094261] R13: ffff880802207bc8 R14: ffffea002f1b4b40 R15: 0000000000000001
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.116663] FS: 000000000000000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.139733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.151330] CR2: ffffea00084c2540 CR3: 0000000001c13000 CR4: 00000000000007e0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.173960] Stack:
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.184901] ffffffff81174751 ffff8808043c1e90 ffff8808043c1e90 ffff8808043c1e90
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.206832] 000000010348c640 ffff880802207bb0 ffff880802207ba0 ffff88102fff9f00
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.228830] 0000000000000000 000000000000001d ffff8808043c1e90 0000000000000000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.250885] Call Trace:
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.261761] [<ffffffff81174
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.272587] [<ffffffff81175
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.283274] [<ffffffff81176
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.293916] [<ffffffff81176
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.304157] [<ffffffff81177
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.314106] [<ffffffff81177
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.323960] [<ffffffff810b5
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.333809] [<ffffffff81177
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.343503] [<ffffffff81091
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.352951] [<ffffffff81091
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.362610] [<ffffffff8176f
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.372355] [<ffffffff81091
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.381537] Code: 00 00 00 ff ff ff ff 01 00 00 00 60 d1 ef 0d 00 ea ff ff e0 20 b7 0b 00 ea ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <68> 00 00 00 00 ff ff 06 28 c8 96 71 06 88 ff ff 2e 00 00 00 00
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.408965] RIP [<ffffea00084c2
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.417611] RSP <ffff880802207a40>
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.425778] CR2: ffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.433893] ---[ end trace 0021a14ede94c8d6 ]---
Server (10Gbps NIC)
Full of these errors:
[114310.563718] RPC request reserved 156 but used 212
[114310.565495] RPC request reserved 156 but used 176
[114310.569816] RPC request reserved 156 but used 176
[114310.576001] RPC request reserved 156 but used 176
[114310.580087] RPC request reserved 156 but used 176
[115206.967835] RPC request reserved 156 but used 212
[115206.981548] RPC request reserved 156 but used 176
[115811.134896] RPC request reserved 156 but used 176
[115811.136346] RPC request reserved 156 but used 176
All machines are Dell R410s and Dell R430s with Broadcom NICs.