Kernel Bug at skb_segment + 0x95a

[Dave Chiluk]

SRU Justification:
Impact: A crash occurs with VXLAN packets when using GSO capable hardware such as the case with openstack neutron.

Fix: Upstream fix 5882a07c7 fixes a locking error that previously allowed
certain timings to hit this bug.

Testcase: Create and run a Neutron gateway with VXLANs, and stress the network for an extended period of time.

Kernel Bug affects 3.11-3.15 on machines using gso offload.

Stack trace
[ 871.025524] ------------[ cut here ]------------
[ 871.030742] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:2903!
[ 871.038669] invalid opcode: 0000 [#1] SMP
[ 871.043318] Modules linked in: xt_nat xt_REDIRECT dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conn
track nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables veth openvswitch gre vxlan ip_tunnel
 dm_crypt 8021q garp stp mrp llc bonding nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache sb_edac gpio_ich joydev edac_core x86_pkg_temp_thermal intel_powerclamp coretemp lpc_ich acpi_power_m
eter kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ipmi_si mac_hid lp parport btrfs xor raid6_pq libcrc32c
hid_generic qla2xxx fnic libfcoe megaraid_sas
[ 871.124025] igb usbhid libfc hid dca scsi_transport_fc ptp enic scsi_tgt pps_core i2c_algo_bit wmi
[ 871.132967] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.13.0-29-generic #53-Ubuntu
[ 871.141500] Hardware name: Cisco Systems Inc UCSC-C220-M3S/UCSC-C220-M3S, BIOS C220M3.1.5.4f.0.111320130449 11/13/2013
[ 871.153541] task: ffff881fd2d117f0 ti: ffff881fd2d18000 task.ti: ffff881fd2d18000
[ 871.161975] RIP: 0010:[<ffffffff81612a6a>] [<ffffffff81612a6a>] skb_segment+0x95a/0x980
[ 871.171124] RSP: 0018:ffff881fffce3498 EFLAGS: 00010206
[ 871.177099] RAX: 0000000000000000 RBX: ffff881fb8929700 RCX: ffff881fcfba32f0
[ 871.185120] RDX: 0000000000000050 RSI: ffff881fcfba3200 RDI: ffff881fcfba2200
[ 871.193140] RBP: ffff881fffce3560 R08: 0000000000000042 R09: 0000000000000000
[ 871.201160] R10: ffff881fb8929e00 R11: 00000000000005ea R12: ffff881fcfba22f0
[ 871.209171] R13: 0000000000000000 R14: ffff881fd09a8400 R15: 0000000000000050
[ 871.217191] FS: 0000000000000000(0000) GS:ffff881fffce0000(0000) knlGS:0000000000000000
[ 871.226307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 871.232772] CR2: 00007fbd1ccb4000 CR3: 0000000001c0e000 CR4: 00000000001407e0
[ 871.240793] Stack:
[ 871.243079] 0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
[ 871.251470] 00000001000005ea 0000000000000040 ffff881fb8929e00 0000000100000000
[ 871.259864] ffffffffffffffce 0000000000000074 00000032000005a8 ffff881fb8929700
[ 871.268257] Call Trace:
[ 871.271032] <IRQ>
[ 871.273193]
[ 871.274913] [<ffffffff8167cfbd>] tcp_gso_segment+0x10d/0x3f0
[ 871.279926] [<ffffffffa03740f0>] ? vxlan_set_owner+0x60/0x60 [vxlan]
[ 871.287178] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
[ 871.293749] [<ffffffffa045c701>] ? bond_mode_name+0x21/0x30 [bonding]
[ 871.301097] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
[ 871.307858] [<ffffffff816840d7>] skb_udp_tunnel_segment+0xd7/0x390
[ 871.314906] [<ffffffff816847f0>] udp4_ufo_fragment+0x120/0x130
[ 871.321566] [<ffffffff8168d422>] inet_gso_segment+0x132/0x360
[ 871.328132] [<ffffffffa0563f67>] ? ipt_do_table+0x317/0x6aa [ip_tables]
[ 871.335669] [<ffffffffa05a3500>] ? nf_conntrack_hash_check_insert+0x280/0x2e0 [nf_conntrack]
[ 871.345268] [<ffffffff8161fdac>] skb_mac_gso_segment+0x9c/0x180
[ 871.352026] [<ffffffff8161feed>] __skb_gso_segment+0x5d/0xb0
[ 871.358494] [<ffffffff816201fa>] dev_hard_start_xmit+0x18a/0x560
[ 871.365351] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
[ 871.371910] [<ffffffffa05c0238>] ? ipv4_confirm+0x78/0x100 [nf_conntrack_ipv4]
[ 871.380156] [<ffffffff8163e03b>] ? eth_header+0x2b/0xd0
[ 871.386135] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
[ 871.392315] [<ffffffff81629367>] neigh_connected_output+0xb7/0x100
[ 871.399366] [<ffffffff81658eb0>] ip_finish_output+0x1b0/0x3b0
[ 871.405922] [<ffffffff8165a418>] ip_output+0x58/0x90
[ 871.411609] [<ffffffff81659b75>] ip_local_out+0x25/0x30
[ 871.417596] [<ffffffff8169c43e>] iptunnel_xmit+0xee/0x110
[ 871.423772] [<ffffffffa0376581>] vxlan_xmit_skb+0x1d1/0x350 [vxlan]
[ 871.430928] [<ffffffffa055956b>] vxlan_tnl_send+0x11b/0x190 [openvswitch]
[ 871.438666] [<ffffffffa055895d>] ovs_vport_send+0x1d/0x80 [openvswitch]
[ 871.446203] [<ffffffffa054f19a>] do_output+0x2a/0x50 [openvswitch]
[ 871.453255] [<ffffffffa054f643>] do_execute_actions+0x2e3/0xa90 [openvswitch]
[ 871.461394] [<ffffffff8163f1ff>] ? sch_direct_xmit+0x5f/0x1c0
[ 871.467960] [<ffffffffa054fe1b>] ovs_execute_actions+0x2b/0x30 [openvswitch]
[ 871.475979] [<ffffffffa05526e2>] ovs_dp_process_received_packet+0x92/0x120 [openvswitch]
[ 871.485198] [<ffffffff8136ced1>] ? csum_partial+0x11/0x20
[ 871.491376] [<ffffffffa055889a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
[ 871.499206] [<ffffffffa0558ced>] internal_dev_xmit+0x1d/0x30 [openvswitch]
[ 871.507035] [<ffffffff81620388>] dev_hard_start_xmit+0x318/0x560
[ 871.513890] [<ffffffff816208e8>] __dev_queue_xmit+0x318/0x500
[ 871.520454] [<ffffffff81620ae0>] dev_queue_xmit+0x10/0x20
[ 871.526630] [<ffffffff81658fc9>] ip_finish_output+0x2c9/0x3b0
[ 871.533193] [<ffffffff8165a418>] ip_output+0x58/0x90
[ 871.538884] [<ffffffff8165639b>] ip_forward_finish+0x8b/0x170
[ 871.545446] [<ffffffff816567d5>] ip_forward+0x355/0x410
[ 871.551427] [<ffffffff816544ed>] ip_rcv_finish+0x7d/0x350
[ 871.557600] [<ffffffff81654e38>] ip_rcv+0x298/0x3d0
[ 871.563191] [<ffffffff8161e9b6>] __netif_receive_skb_core+0x666/0x840
[ 871.570532] [<ffffffff8161eba8>] __netif_receive_skb+0x18/0x60
[ 871.586559] [<ffffffff8161f6ce>] process_backlog+0xae/0x1a0
[ 871.592928] [<ffffffff8161ef92>] net_rx_action+0x152/0x250
[ 871.599211] [<ffffffff8106caec>] __do_softirq+0xec/0x2c0
[ 871.605288] [<ffffffff8106d035>] irq_exit+0x105/0x110
[ 871.611081] [<ffffffff8172cf16>] do_IRQ+0x56/0xc0
[ 871.616485] [<ffffffff817226ad>] common_interrupt+0x6d/0x6d
[ 871.622850] <EOI>
[ 871.625006]
[ 871.626727] [<ffffffff815cd0f2>] ? cpuidle_enter_state+0x52/0xc0
[ 871.632122] [<ffffffff815cd219>] cpuidle_idle_call+0xb9/0x1f0
[ 871.638694] [<ffffffff8101ce9e>] arch_cpu_idle+0xe/0x30
[ 871.644681] [<ffffffff810beb95>] cpu_startup_entry+0xc5/0x290
[ 871.651257] [<ffffffff81040fb8>] start_secondary+0x218/0x2c0
[ 871.657722] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 18 ae ae 81 44 89 ce 31 c0 e8 75 14 10 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 d0 a5 10 00 48 c7 c0 ea f
f ff ff e9 c7 fd ff ff
[ 871.679616] RIP [<ffffffff81612a6a>] skb_segment+0x95a/0x980
[ 871.686097] RSP <ffff881fffce3498>

Fix has been committed in the upstream linux tree at 5882a07c72093dc3a18e2d2b129fb200686bb6ee

Netdev conversation can be viewed here. http://patchwork.ozlabs.org/patch/357291/

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1331219

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Dave Chiluk (chiluk)]

I have sent an e-mail to the linux-stable mailing list requesting that this patch be included in the stable trees.

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for linux-lts-trusty has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-33.58

---------------
linux (3.13.0-33.58) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1349897

  [ Upstream Kernel Changes ]

  * mm: numa: do not automatically migrate KSM pages
    - LP: #1346917
  * net: fix UDP tunnel GSO of frag_list GRO packets
    - LP: #1331219
  * auditsc: audit_krule mask accesses need bounds checking
    - LP: #1347088
  * n_tty: Fix buffer overruns with larger-than-4k pastes
    - LP: #1208740
 -- Tim Gardner <email address hidden> Fri, 18 Jul 2014 14:57:50 +0000