Kernel OOPS

[Pat]

I am receiving kernel OOPS while testing openvswitch with openflow to redirect traffic to a proxy.

Things work fine if I run a slow tcp connection rate (10conn/s) on our test framework.

As soon as I increase the tcp connection rate (1000 conn/s) the kernel OOPS almost immediately.

Here are a couple of the kernel OOPS:

===============================================================
ovs1604 login: [ 264.174741] BUG: unable to handle kernel paging request at ffffffffb99e2b0c
[ 264.518279] IP: [<ffffffff8171615d>] __netdev_pick_tx+0x8d/0x140
[ 264.813723] PGD 2e0d067 PUD 2e0e063 PMD 0
[ 265.013608] Oops: 0000 [#1] SMP
[ 265.170788] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[ 269.038220] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 4.4.0-22-generic #39-Ubuntu
[ 269.411556] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 269.729466] task: ffff881038ba1b80 ti: ffff881038bb0000 task.ti: ffff881038bb0000
[ 269.952910] ------------[ cut here ]------------
[ 269.952918] WARNING: CPU: 0 PID: 4 at /build/linux-UbQGH5/linux-4.4.0/net/sched/sch_generic.c:303 dev_watchdog+0x237/0x240()
[ 269.952919] NETDEV WATCHDOG: eth5 (ixgbe): transmit queue 18 timed out
[ 269.952972] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[ 269.952984] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.4.0-22-generic #39-Ubuntu
[ 269.952985] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 269.952993] Workqueue: events dbs_timer
[ 269.952997] 0000000000000286 00000000d6830222 ffff88103f603d98 ffffffff813e9c53
[ 269.953000] ffff88103f603de0 ffffffff81d62a90 ffff88103f603dd0 ffffffff81080fb2
[ 269.953004] 0000000000000012 ffff8810309ddf00 0000000000000000 ffff881030720000
[ 269.953005] Call Trace:
[ 269.953015] <IRQ> [<ffffffff813e9c53>] dump_stack+0x63/0x90
[ 269.953022] [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
[ 269.953025] [<ffffffff8108104c>] warn_slowpath_fmt+0x5c/0x80
[ 269.953030] [<ffffffff81741df7>] dev_watchdog+0x237/0x240
[ 269.953032] [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[ 269.953039] [<ffffffff810ec3b5>] call_timer_fn+0x35/0x120
[ 269.953041] [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[ 269.953045] [<ffffffff810ecd6a>] run_timer_softirq+0x23a/0x2f0
[ 269.953050] [<ffffffff810859f1>] __do_softirq+0x101/0x290
[ 269.953054] [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[ 269.953061] [<ffffffff81827da2>] smp_apic_timer_interrupt+0x42/0x50
[ 269.953064] [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
[ 269.953070] <EOI> [<ffffffff810ca84d>] ? native_queued_spin_lock_slowpath+0x10d/0x170
[ 269.953075] [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
[ 269.953079] [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
[ 269.953083] [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
[ 269.953089] [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
[ 269.953093] [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
[ 269.953096] [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
[ 269.953100] [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
[ 269.953103] [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
[ 269.953109] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 269.953112] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 269.953115] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 269.953119] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 269.953122] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 269.953124] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 269.953127] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 269.953129] ---[ end trace 23deee586868b6e2 ]---
[ 269.953134] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[ 270.688938] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[ 270.704938] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[ 274.961065] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[ 275.697087] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[ 275.697089] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[ 279.953212] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[ 280.689235] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[ 280.705234] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[ 284.961358] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[ 285.697381] ixgbe 0000:05:00.0 eth4: initiating reset due to tx timeout
[ 285.697383] ixgbe 0000:0b:00.0 eth6: initiating reset due to tx timeout
[ 288.421453] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:365]
[ 288.421500] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[ 288.421512] CPU: 1 PID: 365 Comm: kworker/1:1 Tainted: G W 4.4.0-22-generic #39-Ubuntu
[ 288.421513] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 288.421520] Workqueue: events dbs_timer
[ 288.421523] task: ffff881037a22940 ti: ffff881033cc4000 task.ti: ffff881033cc4000
[ 288.421529] RIP: 0010:[<ffffffff810ca84d>] [<ffffffff810ca84d>] native_queued_spin_lock_slowpath+0x10d/0x170
[ 288.421531] RSP: 0018:ffff881033cc7c48 EFLAGS: 00000246
[ 288.421533] RAX: 0000000000000000 RBX: 0000000008fd0680 RCX: ffff88103fa97840
[ 288.421534] RDX: ffff88103f657840 RSI: 0000000000080000 RDI: ffffffff821ed04c
[ 288.421536] RBP: ffff881033cc7c48 R08: 0000000000000101 R09: 0000000000000002
[ 288.421537] R10: ffff882033502a20 R11: ffff882033502a00 R12: ffffe8efffe44ef0
[ 288.421538] R13: ffff882033502a00 R14: 0000000000000001 R15: 0000000000000100
[ 288.421541] FS: 0000000000000000(0000) GS:ffff88103f640000(0000) knlGS:0000000000000000
[ 288.421542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 288.421544] CR2: 00007f4b2f1f9c20 CR3: 0000000002e0a000 CR4: 00000000001406e0
[ 288.421545] Stack:
[ 288.421548] ffff881033cc7c58 ffffffff81825061 ffff881033cc7cb0 ffffffff816b9d23
[ 288.421551] 0000000000000001 ffff881033cc7d80 0014726000000100 ffff8802001702a0
[ 288.421554] 000000008a595031 ffff882033502a00 00000000001702a0 0000000000000002
[ 288.421554] Call Trace:
[ 288.421561] [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
[ 288.421564] [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
[ 288.421567] [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
[ 288.421572] [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
[ 288.421575] [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
[ 288.421579] [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
[ 288.421582] [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
[ 288.421585] [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
[ 288.421589] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 288.421592] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 288.421595] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 288.421598] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 288.421601] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 288.421604] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 288.421606] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 288.421609] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 288.421640] Code: 12 48 c1 e9 0c 83 e8 01 83 e1 30 48 98 48 81 c1 40 78 01 00 48 03 0c c5 80 42 f3 81 48 89 11 8b 42 08 85 c0 75 0d f3 90 8b 42 08 <85> c0 74 f7 eb 02 f3 90 8b 0f 66 85 c9 75 f7 39 f1 75 0f 89 c8
[ 288.429453] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:20]
[ 288.429497] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds 8250_fintek sb_edac edac_core ipmi_si hpilo ipmi_msghandler lpc_ich ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul usbhid hid ixgbe aesni_intel dca aes_x86_64 vxlan lrw gf128mul ip6_udp_tunnel tg3 glue_helper udp_tunnel hpsa ablk_helper ptp cryptd pps_core scsi_transport_sas mdio wmi fjes
[ 288.429508] CPU: 2 PID: 20 Comm: kworker/2:0 Tainted: G W L 4.4.0-22-generic #39-Ubuntu
[ 288.429509] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 288.429514] Workqueue: events dbs_timer
[ 288.429516] task: ffff881038466e00 ti: ffff881038044000 task.ti: ffff881038044000
[ 288.429521] RIP: 0010:[<ffffffff810ca84d>] [<ffffffff810ca84d>] native_queued_spin_lock_slowpath+0x10d/0x170
[ 288.429523] RSP: 0018:ffff881038047c48 EFLAGS: 00000246
[ 288.429524] RAX: 0000000000000000 RBX: 000000000a52d280 RCX: ffff88103f717840
[ 288.429526] RDX: ffff88103f697840 RSI: 00000000000c0000 RDI: ffffffff821ed04c
[ 288.429527] RBP: ffff881038047c48 R08: 0000000000000101 R09: 0000000000000004
[ 288.429528] R10: ffff882033502c20 R11: ffff882033502c00 R12: ffffe8efffe84ef0
[ 288.429530] R13: ffff882033502c00 R14: 0000000000000002 R15: 0000000000000100
[ 288.429532] FS: 0000000000000000(0000) GS:ffff88103f680000(0000) knlGS:0000000000000000
[ 288.429533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 288.429535] CR2: 0000000000604180 CR3: 0000000002e0a000 CR4: 00000000001406e0
[ 288.429536] Stack:
[ 288.429539] ffff881038047c58 ffffffff81825061 ffff881038047cb0 ffffffff816b9d23
[ 288.429542] ffff88103f696d00 0000000000000020 0018817000000100 ffff8802001a6da0
[ 288.429544] 0000000070d943b3 ffff882033502c00 00000000001a6da0 0000000000000002
[ 288.429545] Call Trace:
[ 288.429551] [<ffffffff81825061>] _raw_spin_lock+0x21/0x30
[ 288.429553] [<ffffffff816b9d23>] pcc_cpufreq_target+0x73/0x220
[ 288.429556] [<ffffffff816afe38>] __cpufreq_driver_target+0x88/0x370
[ 288.429559] [<ffffffff810fe28a>] ? get_cpu_idle_time_us+0x5a/0xd0
[ 288.429563] [<ffffffff816b3fe1>] od_check_cpu+0xc1/0xd0
[ 288.429566] [<ffffffff816b5d6e>] dbs_check_cpu+0x16e/0x1d0
[ 288.429569] [<ffffffff816b42a2>] od_dbs_timer+0xa2/0xf0
[ 288.429572] [<ffffffff816b5fa4>] dbs_timer+0xb4/0xe0
[ 288.429575] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 288.429578] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 288.429581] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 288.429583] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 288.429586] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 288.429588] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 288.429590] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 288.429620] Code: 12 48 c1 e9 0c 83 e8 01 83 e1 30 48 98 48 81 c1 40 78 01 00 48 03 0c c5 80 42 f3 81 48 89 11 8b 42 08 85 c0 75 0d f3 90 8b 42 08 <85> c0 74 f7 eb 02 f3 90 8b 0f 66 85 c9 75 f7 39 f1 75 0f 89 c8
================================================================

and another:

================================================================

ovs1604 login: [ 1138.011941] BUG: unable to handle kernel NULL pointer dereference at 0000000000000400
[ 1138.394597] IP: [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
[ 1138.688303] PGD 0
[ 1138.785779] Oops: 0000 [#1] SMP
[ 1138.943399] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1142.819687] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.4.0-22-generic #39-Ubuntu
[ 1143.193063] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1143.511272] task: ffff882038bc2940 ti: ffff881038404000 task.ti: ffff881038404000
[ 1143.876349] RIP: 0010:[<ffffffff81716136>] [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
[ 1144.289553] RSP: 0018:ffff88103f9c36b0 EFLAGS: 00010206
[ 1144.548342] RAX: ffff88202e15ec00 RBX: ffff88100dddd400 RCX: ffffffff817160d0
[ 1144.898612] RDX: 000000000000007e RSI: ffff88100dddd400 RDI: ffff88102fda0000
[ 1145.247504] RBP: ffff88103f9c36e0 R08: 0000000000000074 R09: ffff88100dbe2818
[ 1145.595710] R10: ffff88100deba06c R11: 00000000000001b0 R12: ffff88102fda0000
[ 1145.944483] R13: 0000000000000000 R14: 00000000ffffffff R15: 0000000000000400
[ 1146.293274] FS: 0000000000000000(0000) GS:ffff88103f9c0000(0000) knlGS:0000000000000000
[ 1146.688266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1146.968958] CR2: 0000000000000400 CR3: 0000000002e0a000 CR4: 00000000001406e0
[ 1147.322813] Stack:
[ 1147.421064] ffff881033968e00 ffff88100dddd400 ffff88102fda0000[ 1147.692532] ------------[ cut here ]------------
[ 1147.692542] WARNING: CPU: 24 PID: 0 at /build/linux-UbQGH5/linux-4.4.0/net/sched/sch_generic.c:303 dev_watchdog+0x237/0x240()
[ 1147.692543] NETDEV WATCHDOG: eth5 (ixgbe): transmit queue 13 timed out
[ 1147.692544] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1147.696361] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.4.0-22-generic #39-Ubuntu
[ 1147.696362] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1147.696364] 0000000000000286 f29cb0207d8e6510 ffff88203f083d98 ffffffff813e9c53
[ 1147.696367] ffff88203f083de0 ffffffff81d62a90 ffff88203f083dd0 ffffffff81080fb2
[ 1147.696566] 000000000000000d ffff881030505f00 0000000000000018 ffff88102fda0000
[ 1147.696758] Call Trace:
[ 1147.696760] <IRQ> [<ffffffff813e9c53>] dump_stack+0x63/0x90
[ 1147.697762] [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
[ 1147.697974] [<ffffffff8108104c>] warn_slowpath_fmt+0x5c/0x80
[ 1147.698185] [<ffffffff81741df7>] dev_watchdog+0x237/0x240
[ 1147.698188] [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[ 1147.698193] [<ffffffff810ec3b5>] call_timer_fn+0x35/0x120
[ 1147.698195] [<ffffffff81741bc0>] ? qdisc_rcu_free+0x40/0x40
[ 1147.698199] [<ffffffff810ecd6a>] run_timer_softirq+0x23a/0x2f0
[ 1147.698203] [<ffffffff810859f1>] __do_softirq+0x101/0x290
[ 1147.698206] [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[ 1147.698213] [<ffffffff81827da2>] smp_apic_timer_interrupt+0x42/0x50
[ 1147.698216] [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
[ 1147.698217] <EOI> [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
[ 1147.698226] [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
[ 1147.698233] [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
[ 1147.698236] [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
[ 1147.698239] [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
[ 1147.698245] [<ffffffff81051714>] start_secondary+0x154/0x190
[ 1147.698247] ---[ end trace 640662c79e82e205 ]---
[ 1147.698252] ixgbe 0000:05:00.1 eth5: initiating reset due to tx timeout
[ 1147.698277] ixgbe 0000:05:00.1 eth5: Reset adapter
[ 1148.258500] ixgbe 0000:05:00.1 eth5: detected SFP+: 5
[ 1148.468518] ixgbe 0000:05:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1161.520473] ffff88102fda0000
[ 1161.710452] 0000000000000000 ffffffff817160d0 ffff88103f9c3718 ffffffffc0457ce2
[ 1162.090787] 0000000000000000 ffff88102fda0000 ffff88100dddd400 0000000000000000
[ 1162.466909] Call Trace:
[ 1162.616986] <IRQ>
[ 1162.710449] [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
[ 1163.027595] [<ffffffffc0457ce2>] ixgbe_select_queue+0xb2/0x130 [ixgbe]
[ 1163.366984] [<ffffffff8171d821>] netdev_pick_tx+0x61/0xe0
[ 1163.658701] [<ffffffff8171d967>] __dev_queue_xmit+0xc7/0x590
[ 1163.955472] [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1164.239831] [<ffffffffc03567e8>] vlan_dev_hard_start_xmit+0x98/0x120 [8021q]
[ 1164.602400] [<ffffffff8171d5d9>] dev_hard_start_xmit+0x249/0x3d0
[ 1164.914965] [<ffffffff8171ddc6>] __dev_queue_xmit+0x526/0x590
[ 1165.218331] [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1165.499745] [<ffffffffc0332cba>] ovs_vport_send+0x4a/0xc0 [openvswitch]
[ 1165.839193] [<ffffffffc03246a3>] do_output.isra.28+0x43/0x170 [openvswitch]
[ 1166.196050] [<ffffffffc0325a09>] do_execute_actions+0x729/0x8b0 [openvswitch]
[ 1166.562138] [<ffffffffc0325d73>] ovs_execute_actions+0x33/0xd0 [openvswitch]
[ 1166.921220] [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1167.295247] [<ffffffffc0325dee>] ovs_execute_actions+0xae/0xd0 [openvswitch]
[ 1167.654452] [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1168.026460] [<ffffffffc032a354>] ? key_extract+0x994/0xc80 [openvswitch]
[ 1168.371007] [<ffffffffc03325ac>] ovs_vport_receive+0x6c/0xd0 [openvswitch]
[ 1168.722503] [<ffffffff8171db26>] ? __dev_queue_xmit+0x286/0x590
[ 1169.027910] [<ffffffffc01d16c0>] ? nf_nat_ipv4_fn+0x60/0x220 [nf_nat_ipv4]
[ 1169.378769] [<ffffffff8175dd09>] ? ip_finish_output2+0x159/0x340
[ 1169.688345] [<ffffffffc02fd23d>] ? ipv4_confirm+0x7d/0x100 [nf_conntrack_ipv4]
[ 1170.059626] [<ffffffff8175ec56>] ? ip_finish_output+0x136/0x1f0
[ 1170.364133] [<ffffffff817524d3>] ? nf_hook_slow+0x73/0xd0
[ 1170.639235] [<ffffffff8175b2ed>] ? ip_forward_finish+0x4d/0x70
[ 1170.939768] [<ffffffffc03337cd>] netdev_frame_hook+0xed/0x160 [openvswitch]
[ 1171.296146] [<ffffffff8171af74>] __netif_receive_skb_core+0x364/0xa60
[ 1171.626731] [<ffffffff81794900>] ? inet_gro_receive+0x180/0x200
[ 1171.931280] [<ffffffff8171bf38>] ? dev_gro_receive+0x228/0x3c0
[ 1172.241496] [<ffffffff8171b688>] __netif_receive_skb+0x18/0x60
[ 1172.541952] [<ffffffff8171b702>] netif_receive_skb_internal+0x32/0xa0
[ 1172.621141] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [kworker/14:2:2021]
[ 1172.621183] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1172.621194] CPU: 14 PID: 2021 Comm: kworker/14:2 Tainted: G W 4.4.0-22-generic #39-Ubuntu
[ 1172.621195] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1172.621202] Workqueue: events linkwatch_event
[ 1172.621204] task: ffff882032295280 ti: ffff882027830000 task.ti: ffff882027830000
[ 1172.621215] RIP: 0010:[<ffffffff81103a2d>] [<ffffffff81103a2d>] smp_call_function_single+0x8d/0x130
[ 1172.621217] RSP: 0018:ffff882027833c40 EFLAGS: 00000202
[ 1172.621218] RAX: 0000000000000001 RBX: 000000000000001f RCX: 0000000000000000
[ 1172.621219] RDX: ffffffff81e52580 RSI: ffff88203ee17c00 RDI: 000000000000001f
[ 1172.621220] RBP: ffff882027833c88 R08: 0000000000000000 R09: 0000000000000000
[ 1172.621221] R10: 0000000000000004 R11: ffff8820322952e0 R12: ffffffff810e4d50
[ 1172.621222] R13: 0000000080000000 R14: 00dffeff5ff26fd7 R15: 000000000000001f
[ 1172.621224] FS: 0000000000000000(0000) GS:ffff88203ee00000(0000) knlGS:0000000000000000
[ 1172.621225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1172.621226] CR2: 00007f7c0be726d0 CR3: 0000000002e0a000 CR4: 00000000001406e0
[ 1172.621227] Stack:
[ 1172.621230] ffff88103f007480 ffff88100ddc6600 0000000000000000 0000000000000000
[ 1172.621232] 0000000000000000 0000000000000003 00000000b542f39f ffffffff81e52580
[ 1172.621234] 000000000000dca0 ffff882027833d08 ffffffff810e53d9 ffffffff81e52700
[ 1172.621234] Call Trace:
[ 1172.621240] [<ffffffff810e53d9>] synchronize_sched_expedited+0x339/0x880
[ 1172.621244] [<ffffffff810ec4f4>] ? lock_timer_base.isra.22+0x54/0x70
[ 1172.621247] [<ffffffff810e59ae>] synchronize_rcu_expedited+0xe/0x10
[ 1172.621250] [<ffffffff81716fd3>] synchronize_net+0x23/0x30
[ 1172.621253] [<ffffffff817429eb>] dev_deactivate_many+0x23b/0x240
[ 1172.621255] [<ffffffff81742a35>] dev_deactivate+0x45/0x70
[ 1172.621258] [<ffffffff817310d0>] linkwatch_do_dev+0x30/0x50
[ 1172.621261] [<ffffffff81731427>] __linkwatch_run_queue+0x167/0x190
[ 1172.621263] [<ffffffff81731475>] linkwatch_event+0x25/0x30
[ 1172.621269] [<ffffffff8109a052>] process_one_work+0x162/0x480
[ 1172.621271] [<ffffffff8109a3bb>] worker_thread+0x4b/0x4c0
[ 1172.621274] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 1172.621277] [<ffffffff8109a370>] ? process_one_work+0x480/0x480
[ 1172.621280] [<ffffffff810a0588>] kthread+0xd8/0xf0
[ 1172.621283] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 1172.621285] [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
[ 1172.621287] [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
[ 1172.621309] Code: c0 74 0c 9c 58 0f 1f 44 00 00 f6 c4 02 74 71 85 c9 75 48 48 c7 c6 00 7c 01 00 65 48 03 35 64 67 f0 7e 8b 46 18 a8 01 74 09 f3 90 <8b> 46 18 a8 01 75 f7 83 4e 18 01 48 89 d1 89 df 4c 89 e2 e8 3b
[ 1189.966808] [<ffffffff8171b85f>] napi_gro_complete+0x7f/0xa0
[ 1190.263283] [<ffffffff8171b8da>] napi_gro_flush+0x5a/0x80
[ 1190.546867] [<ffffffffc045a32e>] ixgbe_poll+0x58e/0x8d0 [ixgbe]
[ 1190.855476] [<ffffffff8171bbce>] net_rx_action+0x21e/0x360
[ 1191.142638] [<ffffffff810859f1>] __do_softirq+0x101/0x290
[ 1191.424157] [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[ 1191.679965] [<ffffffff81827ce4>] do_IRQ+0x54/0xd0
[ 1191.926377] [<ffffffff81825dc2>] common_interrupt+0x82/0x82
[ 1192.228819] <EOI>
[ 1192.322267] [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
[ 1192.660739] [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
[ 1192.943489] [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
[ 1193.215091] [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
[ 1193.499665] [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
[ 1193.801670] [<ffffffff81051714>] start_secondary+0x154/0x190
[ 1194.092982] Code: 87 e8 03 00 00 49 89 fc 48 85 c0 0f 84 da 00 00 00 8b 96 ac 00 00 00 83 ea 01 48 8d 44 d0 10 4c 8b 38 4d 85 ff 0f 84 c0 00 00 00 <41> 8b 1f 83 fb 01 0f 84 8d 00 00 00 f6 86 91 00 00 00 30 0f 84
[ 1195.043812] RIP [<ffffffff81716136>] __netdev_pick_tx+0x66/0x140
[ 1195.352932] RSP <ffff88103f9c36b0>
[ 1195.534578] CR2: 0000000000000400
[ 1195.742792] ---[ end trace 640662c79e82e206 ]---
[ 1195.981707] Kernel panic - not syncing: Fatal exception in interrupt
[ 1196.303793] Kernel Offset: disabled
[ 1196.493128] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[ 1196.854462] ------------[ cut here ]------------
[ 1197.090915] WARNING: CPU: 29 PID: 0 at /build/linux-UbQGH5/linux-4.4.0/arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x60/0x70()
[ 1197.706261] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 8021q garp mrp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables x_tables openvswitch nf_defrag_ipv6 nf_conntrack nls_iso8859_1 ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass input_leds sb_edac edac_core hpilo lpc_ich ipmi_si ipmi_msghandler 8250_fintek ioatdma shpchp acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ixgbe crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 dca lrw vxlan gf128mul ip6_udp_tunnel glue_helper tg3 ablk_helper udp_tunnel cryptd hpsa usbhid ptp pps_core hid scsi_transport_sas mdio wmi fjes
[ 1201.718155] CPU: 29 PID: 0 Comm: swapper/29 Tainted: G D W L 4.4.0-22-generic #39-Ubuntu
[ 1202.164963] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 1202.499892] 0000000000000086 f9c2da801aceb89c ffff88103f9c3168 ffffffff813e9c53
[ 1202.880066] 0000000000000000 ffffffff81ca35b0 ffff88103f9c31a0 ffffffff81080fb2
[ 1203.259124] 000000000000000f ffff88203ee56d00 000000000000001d ffff882038bc2940
[ 1203.636609] Call Trace:
[ 1203.773175] <IRQ> [<ffffffff813e9c53>] dump_stack+0x63/0x90
[ 1204.070769] [<ffffffff81080fb2>] warn_slowpath_common+0x82/0xc0
[ 1204.383768] [<ffffffff810810fa>] warn_slowpath_null+0x1a/0x20
[ 1204.685193] [<ffffffff81050970>] native_smp_send_reschedule+0x60/0x70
[ 1205.020826] [<ffffffff810be7a3>] trigger_load_balance+0x133/0x210
[ 1205.334824] [<ffffffff810ad3a6>] scheduler_tick+0xa6/0xd0
[ 1205.619034] [<ffffffff810fe560>] ? tick_sched_handle.isra.14+0x60/0x60
[ 1205.959021] [<ffffffff810ee961>] update_process_times+0x51/0x60
[ 1206.270252] [<ffffffff810fe525>] tick_sched_handle.isra.14+0x25/0x60
[ 1206.601615] [<ffffffff810fe59d>] tick_sched_timer+0x3d/0x70
[ 1206.894935] [<ffffffff810ef282>] __hrtimer_run_queues+0x102/0x290
[ 1207.209571] [<ffffffff810efa48>] hrtimer_interrupt+0xa8/0x1a0
[ 1207.510638] [<ffffffff81052fa8>] local_apic_timer_interrupt+0x38/0x60
[ 1207.845707] [<ffffffff81827d9d>] smp_apic_timer_interrupt+0x3d/0x50
[ 1208.172391] [<ffffffff81826062>] apic_timer_interrupt+0x82/0x90
[ 1208.482226] [<ffffffff8118bbb9>] ? panic+0x1d5/0x215
[ 1208.745324] [<ffffffff8118bbb2>] ? panic+0x1ce/0x215
[ 1209.008398] [<ffffffff81031c4a>] oops_end+0xca/0xd0
[ 1209.263560] [<ffffffff8106ad45>] no_context+0x135/0x380
[ 1209.538824] [<ffffffff8106b010>] __bad_area_nosemaphore+0x80/0x1f0
[ 1209.860782] [<ffffffff8106b193>] bad_area_nosemaphore+0x13/0x20
[ 1210.170022] [<ffffffff8106b457>] __do_page_fault+0xb7/0x400
[ 1210.463123] [<ffffffffc0462a02>] ? ixgbe_xmit_frame+0x42/0x90 [ixgbe]
[ 1210.797385] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[ 1211.076463] [<ffffffff81827478>] page_fault+0x28/0x30
[ 1211.342835] [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
[ 1211.633287] [<ffffffff81716136>] ? __netdev_pick_tx+0x66/0x140
[ 1211.940942] [<ffffffff817160d0>] ? __skb_tx_hash+0xa0/0xa0
[ 1212.230203] [<ffffffffc0457ce2>] ixgbe_select_queue+0xb2/0x130 [ixgbe]
[ 1212.566347] [<ffffffff8171d821>] netdev_pick_tx+0x61/0xe0
[ 1212.851473] [<ffffffff8171d967>] __dev_queue_xmit+0xc7/0x590
[ 1213.152225] [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1213.437482] [<ffffffffc03567e8>] vlan_dev_hard_start_xmit+0x98/0x120 [8021q]
[ 1213.801585] [<ffffffff8171d5d9>] dev_hard_start_xmit+0x249/0x3d0
[ 1214.115849] [<ffffffff8171ddc6>] __dev_queue_xmit+0x526/0x590
[ 1214.416730] [<ffffffff8171de40>] dev_queue_xmit+0x10/0x20
[ 1214.699175] [<ffffffffc0332cba>] ovs_vport_send+0x4a/0xc0 [openvswitch]
[ 1215.038336] [<ffffffffc03246a3>] do_output.isra.28+0x43/0x170 [openvswitch]
[ 1215.394550] [<ffffffffc0325a09>] do_execute_actions+0x729/0x8b0 [openvswitch]
[ 1215.758636] [<ffffffffc0325d73>] ovs_execute_actions+0x33/0xd0 [openvswitch]
[ 1216.118188] [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1216.490791] [<ffffffffc0325dee>] ovs_execute_actions+0xae/0xd0 [openvswitch]
[ 1216.849706] [<ffffffffc0329564>] ovs_dp_process_packet+0x84/0x130 [openvswitch]
[ 1217.222344] [<ffffffffc032a354>] ? key_extract+0x994/0xc80 [openvswitch]
[ 1217.566366] [<ffffffffc03325ac>] ovs_vport_receive+0x6c/0xd0 [openvswitch]
[ 1217.918259] [<ffffffff8171db26>] ? __dev_queue_xmit+0x286/0x590
[ 1218.222395] [<ffffffffc01d16c0>] ? nf_nat_ipv4_fn+0x60/0x220 [nf_nat_ipv4]
[ 1218.572934] [<ffffffff8175dd09>] ? ip_finish_output2+0x159/0x340
[ 1218.881368] [<ffffffffc02fd23d>] ? ipv4_confirm+0x7d/0x100 [nf_conntrack_ipv4]
[ 1219.248697] [<ffffffff8175ec56>] ? ip_finish_output+0x136/0x1f0
[ 1219.552761] [<ffffffff817524d3>] ? nf_hook_slow+0x73/0xd0
[ 1219.832000] [<ffffffff8175b2ed>] ? ip_forward_finish+0x4d/0x70
[ 1220.131812] [<ffffffffc03337cd>] netdev_frame_hook+0xed/0x160 [openvswitch]
[ 1220.486445] [<ffffffff8171af74>] __netif_receive_skb_core+0x364/0xa60
[ 1220.816076] [<ffffffff81794900>] ? inet_gro_receive+0x180/0x200
[ 1221.119917] [<ffffffff8171bf38>] ? dev_gro_receive+0x228/0x3c0
[ 1221.419505] [<ffffffff8171b688>] __netif_receive_skb+0x18/0x60
[ 1221.718992] [<ffffffff8171b702>] netif_receive_skb_internal+0x32/0xa0
[ 1222.048204] [<ffffffff8171b85f>] napi_gro_complete+0x7f/0xa0
[ 1222.330734] [<ffffffff8171b8da>] napi_gro_flush+0x5a/0x80
[ 1222.609174] [<ffffffffc045a32e>] ixgbe_poll+0x58e/0x8d0 [ixgbe]
[ 1222.914199] [<ffffffff8171bbce>] net_rx_action+0x21e/0x360
[ 1223.196818] [<ffffffff810859f1>] __do_softirq+0x101/0x290
[ 1223.475092] [<ffffffff81085cf3>] irq_exit+0xa3/0xb0
[ 1223.728540] [<ffffffff81827ce4>] do_IRQ+0x54/0xd0
[ 1223.973680] [<ffffffff81825dc2>] common_interrupt+0x82/0x82
[ 1224.260746] <EOI> [<ffffffff816bc411>] ? cpuidle_enter_state+0x111/0x2b0
[ 1224.607877] [<ffffffff816bc5e7>] cpuidle_enter+0x17/0x20
[ 1224.886873] [<ffffffff810c3db2>] call_cpuidle+0x32/0x60
[ 1225.158221] [<ffffffff816bc5c3>] ? cpuidle_select+0x13/0x20
[ 1225.447849] [<ffffffff810c4070>] cpu_startup_entry+0x290/0x350
[ 1225.748616] [<ffffffff81051714>] start_secondary+0x154/0x190
[ 1226.047825] ---[ end trace 640662c79e82e207 ]---

=======================================================

I'm not sure if providing the current openvswitch configuration will help. The setup is fairly complicated. Here is the basic configuration:

# ovs-vsctl show
1a7ea432-8b96-445e-8734-d71d10533302
    Bridge "ovsbr0"
        Port "eth6"
            tag: 998
            Interface "eth6"
        Port "eth5.110"
            tag: 110
            Interface "eth5.110"
        Port "eth4"
            tag: 999
            Interface "eth4"
        Port int-servers
            tag: 998
            Interface int-servers
                type: internal
        Port client
            tag: 999
            Interface client
                type: internal
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port ingress
            tag: 110
            Interface ingress
                type: internal
        Port "eth1.110"
            tag: 110
            Interface "eth1.110"
    ovs_version: "2.5.0"

# ovs-ofctl -O Openflow15 dump-groups ovsbr0
OFPST_GROUP_DESC reply (OF1.5) (xid=0x2):
 group_id=200,type=select,selection_method=hash,fields(ip_dst,tcp_dst),bucket=bucket_id:0,actions=set_field:fa:16:3e:17:2f:f1->eth_dst,output:9,bucket=bucket_id:1,actions=set_field:fa:16:3e:2f:51:99->eth_dst,output:9,bucket=bucket_id:2,actions=set_field:fa:16:3e:76:f3:42->eth_dst,output:9,bucket=bucket_id:3,actions=set_field:fa:16:3e:c0:cf:10->eth_dst,output:9,bucket=bucket_id:4,actions=set_field:fa:16:3e:ba:6f:b7->eth_dst,output:9
 group_id=100,type=select,selection_method=hash,fields(ip_src,tcp_src),bucket=bucket_id:0,actions=set_field:fa:16:3e:17:2f:f1->eth_dst,output:9,bucket=bucket_id:1,actions=set_field:fa:16:3e:2f:51:99->eth_dst,output:9,bucket=bucket_id:2,actions=set_field:fa:16:3e:76:f3:42->eth_dst,output:9,bucket=bucket_id:3,actions=set_field:fa:16:3e:c0:cf:10->eth_dst,output:9,bucket=bucket_id:4,actions=set_field:fa:16:3e:ba:6f:b7->eth_dst,output:9

# ovs-ofctl -O Openflow15 dump-flows ovsbr0
OFPST_FLOW reply (OF1.5) (xid=0x2):
 cookie=0x0, duration=206.013s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=2 actions=resubmit(,100)
 cookie=0x0, duration=206.013s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=10 actions=resubmit(,101)
 cookie=0x0, duration=206.009s, table=0, n_packets=8, n_bytes=632, priority=100,in_port=9 actions=resubmit(,200)
 cookie=0x0, duration=206.013s, table=0, n_packets=8, n_bytes=560, priority=0 actions=NORMAL
 cookie=0x0, duration=206.012s, table=100, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_src=10.100.0.0/16,tp_dst=80 actions=ct(table=110)
 cookie=0x0, duration=206.012s, table=100, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
 cookie=0x0, duration=206.012s, table=101, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_dst=10.100.0.0/16,tp_src=80 actions=ct(table=120,zone=200)
 cookie=0x0, duration=206.012s, table=101, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
 cookie=0x0, duration=206.011s, table=110, n_packets=0, n_bytes=0, priority=100,ct_state=+new+trk,ip actions=ct(commit,exec(set_field:0x64->ct_mark)),group:100
 cookie=0x0, duration=206.010s, table=110, n_packets=0, n_bytes=0, priority=100,ct_state=+est-rpl+trk,ip actions=ct(commit),group:100
 cookie=0x0, duration=206.010s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=206.010s, table=120, n_packets=0, n_bytes=0, priority=101,ct_state=+est+rpl+trk,ct_zone=200,ip actions=ct(commit,zone=200),group:200
 cookie=0x0, duration=206.010s, table=120, n_packets=0, n_bytes=0, priority=100,ct_state=+est+rpl+trk,ip actions=ct(commit,zone=200),group:200
 cookie=0x0, duration=206.009s, table=120, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=206.009s, table=200, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_src=10.100.0.0/16,tp_dst=80 actions=ct(table=210,zone=200)
 cookie=0x0, duration=206.008s, table=200, n_packets=0, n_bytes=0, priority=100,ct_state=-trk,tcp,nw_dst=10.100.0.0/16,tp_src=80 actions=ct(table=220)
 cookie=0x0, duration=206.008s, table=200, n_packets=8, n_bytes=632, priority=0 actions=NORMAL
 cookie=0x0, duration=206.008s, table=210, n_packets=0, n_bytes=0, priority=100,ct_state=+new+trk,ip actions=ct(commit,zone=200),NORMAL
 cookie=0x0, duration=206.007s, table=210, n_packets=0, n_bytes=0, priority=100,ct_state=+est-rpl+trk,ip actions=ct(commit,zone=200),NORMAL
 cookie=0x0, duration=206.007s, table=210, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=100,ct_state=+est+rpl+trk,ip actions=ct(commit),NORMAL
 cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=100,ct_state=+rpl+trk,ip actions=ct(commit),NORMAL
 cookie=0x0, duration=206.006s, table=220, n_packets=0, n_bytes=0, priority=0 actions=drop

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-22-generic 4.4.0-22.39
ProcVersionSignature: Ubuntu 4.4.0-22.39-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 9 17:05 seq
 crw-rw---- 1 root audio 116, 33 May 9 17:05 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Mon May 9 17:12:43 2016
HibernationDevice: RESUME=/dev/mapper/ovs1604--vg-swap
InstallationDate: Installed on 2016-05-09 (0 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL380 Gen9
PciMultimedia:

ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic.efi.signed root=/dev/mapper/hostname--vg-root ro net.ifnames=0 biosdevname=0 console=ttyS0,9600 console=tty1
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-22-generic N/A
 linux-backports-modules-4.4.0-22-generic N/A
 linux-firmware 1.157
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/06/2015
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP89:bd05/06/2015:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL380 Gen9
dmi.sys.vendor: HP

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.6 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc7-wily/

[Pat (patrick-bergamin)]

I have tested with the suggested upstream v4.6 kernel, running up to 12000 tcp conn/s without hitting the problem.

# uname -a
Linux ovs1604 4.6.0-040600rc7-generic #201605081830 SMP Sun May 8 22:32:57 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

As a side note I was also able to reproduce the problem with the 4.4.9-040409 kernel.