Comment 57 for bug 585657

Still happening in 14.04 LTS

Client (1Gbps Link):
[ 3038.818986] nfs: server 10.0.0.200 not responding, timed out
[ 3038.818991] nfs: server 10.0.0.200 not responding, timed out
[ 3038.818996] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819001] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819006] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819012] nfs: server 10.0.0.200 not responding, timed out
[ 3038.819017] nfs: server 10.0.0.200 not responding, timed out
[ 3038.958559] nfs: server 10.0.0.200 not responding, timed out

Pings are under 1ms

Crash:
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.799988] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.815847] BUG: unable to handle kernel paging request at ffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.824363] IP: [<ffffea00084c2540>] 0xffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.832785] PGD 82fff5067 PUD 82fff4067 PMD 80000008176001e3
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.841143] Oops: 0011 [#2] SMP
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.849239] Modules linked in: mptctl xt_comment iptable_filter xt_multiport ip_tables x_tables rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nf_conntrack_netlink nf_conntrack nfnetlink intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw ipmi_devintf serio_raw joydev gf128mul glue_helper dcdbas ablk_helper i7core_edac acpi_power_meter gpio_ich lpc_ich ipmi_si edac_core cryptd ipmi_msghandler shpchp mac_hid lp parport tcp_htcp hid_generic mptsas mptscsih usbhid mptbase psmouse hid scsi_transport_sas pata_acpi bnx2
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.919101] CPU: 6 PID: 210 Comm: kswapd0 Tainted: G D W 3.16.0-51-generic #69~14.04.1-Ubuntu
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.937875] Hardware name: Dell Inc. PowerEdge R410/01V648, BIOS 1.6.3 02/07/2011
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.957034] task: ffff8808043c1e90 ti: ffff880802204000 task.ti: ffff880802204000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.976975] RIP: 0010:[<ffffea00084c2540>] [<ffffea00084c2540>] 0xffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888544.997565] RSP: 0018:ffff880802207a40 EFLAGS: 00010282
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.007973] RAX: ffff8807f94c8848 RBX: ffff880802207db0 RCX: 0000000000000000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.028679] RDX: ffffea00084c2540 RSI: 0000000000000002 RDI: ffffea001623df80
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.049907] RBP: ffff880802207b40 R08: ffff880002d078e8 R09: ffff880005eaf478
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.071886] R10: ffff8808022079c8 R11: ffffea003f7e0980 R12: ffffea002f1b4b60
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.094261] R13: ffff880802207bc8 R14: ffffea002f1b4b40 R15: 0000000000000001
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.116663] FS: 0000000000000000(0000) GS:ffff88102fc60000(0000) knlGS:0000000000000000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.139733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.151330] CR2: ffffea00084c2540 CR3: 0000000001c13000 CR4: 00000000000007e0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.173960] Stack:
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.184901] ffffffff81174751 ffff8808043c1e90 ffff8808043c1e90 ffff8808043c1e90
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.206832] 000000010348c640 ffff880802207bb0 ffff880802207ba0 ffff88102fff9f00
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.228830] 0000000000000000 000000000000001d ffff8808043c1e90 0000000000000000
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.250885] Call Trace:
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.261761] [<ffffffff81174751>] ? shrink_page_list+0x241/0xaa0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.272587] [<ffffffff81175645>] shrink_inactive_list+0x1c5/0x560
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.283274] [<ffffffff81176343>] shrink_lruvec+0x523/0x710
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.293916] [<ffffffff811765ac>] shrink_zone+0x7c/0x1b0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.304157] [<ffffffff811776e5>] balance_pgdat+0x3b5/0x620
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.314106] [<ffffffff81177aab>] kswapd+0x15b/0x3f0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.323960] [<ffffffff810b50e0>] ? prepare_to_wait_event+0x100/0x100
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.333809] [<ffffffff81177950>] ? balance_pgdat+0x620/0x620
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.343503] [<ffffffff810915a2>] kthread+0xd2/0xf0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.352951] [<ffffffff810914d0>] ? kthread_create_on_node+0x1c0/0x1c0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.362610] [<ffffffff8176f2d8>] ret_from_fork+0x58/0x90
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.372355] [<ffffffff810914d0>] ? kthread_create_on_node+0x1c0/0x1c0
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.381537] Code: 00 00 00 ff ff ff ff 01 00 00 00 60 d1 ef 0d 00 ea ff ff e0 20 b7 0b 00 ea ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <68> 00 00 00 00 ff ff 06 28 c8 96 71 06 88 ff ff 2e 00 00 00 00
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.408965] RIP [<ffffea00084c2540>] 0xffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.417611] RSP <ffff880802207a40>
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.425778] CR2: ffffea00084c2540
Nov 2 06:14:49 rtd-lin-nnrpd04 kernel: [888545.433893] ---[ end trace 0021a14ede94c8d6 ]---

Server (10Gbps NIC)
Full of these errors:
[114310.563718] RPC request reserved 156 but used 212
[114310.565495] RPC request reserved 156 but used 176
[114310.569816] RPC request reserved 156 but used 176
[114310.576001] RPC request reserved 156 but used 176
[114310.580087] RPC request reserved 156 but used 176
[115206.967835] RPC request reserved 156 but used 212
[115206.981548] RPC request reserved 156 but used 176
[115811.134896] RPC request reserved 156 but used 176
[115811.136346] RPC request reserved 156 but used 176

All machines are Dell R410s and Dell R430s with Broadcom NICs.