| 2017-03-13 18:15:13 |
Dan Streetman |
bug |
|
|
added bug |
| 2017-03-13 18:19:56 |
Dan Streetman |
linux (Ubuntu): assignee |
|
Dan Streetman (ddstreet) |
|
| 2017-03-13 18:20:00 |
Dan Streetman |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2017-03-13 18:20:05 |
Dan Streetman |
linux (Ubuntu): status |
New |
In Progress |
|
| 2017-03-13 18:20:47 |
Dan Streetman |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-03-13 18:20:47 |
Dan Streetman |
nominated for series |
|
Ubuntu Trusty |
|
| 2017-03-13 18:20:47 |
Dan Streetman |
nominated for series |
|
Ubuntu Vivid |
|
| 2017-03-13 18:20:47 |
Dan Streetman |
nominated for series |
|
Ubuntu Zesty |
|
| 2017-03-13 18:20:47 |
Dan Streetman |
nominated for series |
|
Ubuntu Yakkety |
|
| 2017-03-13 18:21:08 |
Dan Streetman |
tags |
|
sts |
|
| 2017-03-13 19:20:39 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Trusty) |
|
| 2017-03-13 19:21:03 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Vivid) |
|
| 2017-03-13 19:21:39 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Xenial) |
|
| 2017-03-13 19:21:48 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Yakkety) |
|
| 2017-03-13 19:22:06 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Zesty) |
|
| 2017-03-13 19:22:16 |
Joseph Salisbury |
linux (Ubuntu Yakkety): importance |
Undecided |
Medium |
|
| 2017-03-13 19:22:18 |
Joseph Salisbury |
linux (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2017-03-13 19:22:21 |
Joseph Salisbury |
linux (Ubuntu Vivid): importance |
Undecided |
Medium |
|
| 2017-03-13 19:22:23 |
Joseph Salisbury |
linux (Ubuntu Trusty): importance |
Undecided |
Medium |
|
| 2017-03-13 19:22:28 |
Joseph Salisbury |
linux (Ubuntu Yakkety): status |
New |
Confirmed |
|
| 2017-03-13 19:22:31 |
Joseph Salisbury |
linux (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2017-03-13 19:22:33 |
Joseph Salisbury |
linux (Ubuntu Vivid): status |
New |
Confirmed |
|
| 2017-03-13 19:22:35 |
Joseph Salisbury |
linux (Ubuntu Trusty): status |
New |
Confirmed |
|
| 2017-03-13 21:02:16 |
Dan Streetman |
bug task deleted |
linux (Ubuntu Vivid) |
|
|
| 2017-03-14 01:29:44 |
Yoshi Kadokawa |
bug |
|
|
added subscriber Yoshi Kadokawa |
| 2017-03-14 06:55:54 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2017-03-17 14:09:16 |
Dan Streetman |
linux (Ubuntu Trusty): status |
Confirmed |
Invalid |
|
| 2017-03-17 14:10:37 |
Dan Streetman |
linux (Ubuntu Xenial): status |
Confirmed |
In Progress |
|
| 2017-03-17 14:10:39 |
Dan Streetman |
linux (Ubuntu Yakkety): status |
Confirmed |
In Progress |
|
| 2017-03-17 14:14:09 |
Dan Streetman |
linux (Ubuntu Yakkety): assignee |
|
Dan Streetman (ddstreet) |
|
| 2017-03-17 14:14:11 |
Dan Streetman |
linux (Ubuntu Xenial): assignee |
|
Dan Streetman (ddstreet) |
|
| 2017-03-17 14:23:12 |
Dan Streetman |
description |
The br_netfilter module processes packets traveling through its bridge, and while processing each skb it places a special fake dst onto the skb. When the skb leaves the bridge, it removes the fake dst and places a real dst onto it. However, it uses a hook to do this, and when the br_netfilter module is unloading it unregisters that hook. Any skbs that are currently being processed in the bridge by the br_netfilter module, but that leave the bridge after the hook is unregistered (or, during br_netfilter module load, before the hook is registered) will still have the fake dst; when other code then tries to process that dst, it causes a kernel panic because the dst is invalid.
Recent upstream discussion:
https://www.spinics.net/lists/netdev/msg416912.html
Upstream patch (not yet merged into net-next):
https://patchwork.ozlabs.org/patch/738275/
example panic report:
[ 214.518262] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 214.612199] IP: [< (null)>] (null)
[ 214.672744] PGD 0 [ 214.696887] Oops: 0010 [#1] SMP [ 214.735697] Modules linked in: br_netfilter(+) tun 8021q bridge stp llc bonding iTCO_wdt iTCO_vendor_support tpm_tis tpm kvm_intel kvm irqbypass sb_edac edac_core ixgbe mdio ipmi_si ipmi_msghandler lpc_ich mfd_core mousedev evdev igb dca procmemro(O) nokeyctl(O) noptrace(O)
[ 215.029240] CPU: 34 PID: 0 Comm: swapper/34 Tainted: G O 4.4.39 #1
[ 215.116720] Hardware name: Cisco Systems Inc UCSC-C220-M3L/UCSC-C220-M3L, BIOS C220M3.2.0.13a.0.0713160937 07/13/16
[ 215.241644] task: ffff882038fb4380 ti: ffff8810392b0000 task.ti: ffff8810392b0000
[ 215.331207] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 215.420877] RSP: 0018:ffff88103fec3880 EFLAGS: 00010286
[ 215.484436] RAX: ffff881011631000 RBX: ffff881011067100 RCX: 0000000000000000
[ 215.569836] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff881011067100
[ 215.655234] RBP: ffff88103fec38a8 R08: 0000000000000008 R09: ffff8810116300a0
[ 215.740629] R10: 0000000000000000 R11: 0000000000000000 R12: ffff881018917dce
[ 215.826030] R13: ffffffff81c9be00 R14: ffffffff81c9be00 R15: ffff881011630078
[ 215.911432] FS: 0000000000000000(0000) GS:ffff88103fec0000(0000) knlGS:0000000000000000
[ 216.008274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 216.077032] CR2: 0000000000000000 CR3: 0000001011b9d000 CR4: 00000000001406e0
[ 216.162430] Stack:
[ 216.186461] ffffffff8157d7f9 ffff881011067100 ffff881018917dce ffff881011630000
[ 216.275407] ffffffff81c9be00 ffff88103fec3918 ffffffff8157e0db 0000000000000000
[ 216.364352] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 216.453301] Call Trace:
[ 216.482536] <IRQ> [ 216.505533] [<ffffffff8157d7f9>] ? ip_rcv_finish+0x99/0x320
[ 216.575442] [<ffffffff8157e0db>] ip_rcv+0x25b/0x370
[ 216.634842] [<ffffffff81540e0b>] __netif_receive_skb_core+0x2cb/0xa20
[ 216.712965] [<ffffffff81541578>] __netif_receive_skb+0x18/0x60
[ 216.783801] [<ffffffff815415e3>] netif_receive_skb_internal+0x23/0x80
[ 216.861921] [<ffffffff8154165c>] netif_receive_skb+0x1c/0x70
[ 216.930686] [<ffffffffa02f6439>] br_handle_frame_finish+0x1b9/0x5b0 [bridge]
[ 217.016091] [<ffffffff81187a00>] ? ___slab_alloc+0x1d0/0x440
[ 217.084849] [<ffffffffa0584074>] br_nf_pre_routing_finish+0x174/0x3d0 [br_netfilter]
[ 217.178568] [<ffffffffa0584c07>] ? br_nf_pre_routing+0x97/0x470 [br_netfilter]
[ 217.266052] [<ffffffffa02f6280>] ? br_handle_local_finish+0x80/0x80 [bridge]
[ 217.351450] [<ffffffffa0584d17>] br_nf_pre_routing+0x1a7/0x470 [br_netfilter]
[ 217.437891] [<ffffffff81572f6d>] nf_iterate+0x5d/0x70
[ 217.499367] [<ffffffff81572fe4>] nf_hook_slow+0x64/0xc0
[ 217.562928] [<ffffffffa02f69e9>] br_handle_frame+0x1b9/0x290 [bridge]
[ 217.641048] [<ffffffffa02f6280>] ? br_handle_local_finish+0x80/0x80 [bridge]
[ 217.726446] [<ffffffff81540e82>] __netif_receive_skb_core+0x342/0xa20
[ 217.804566] [<ffffffff815a7916>] ? tcp4_gro_receive+0x126/0x1d0
[ 217.876445] [<ffffffff815b7446>] ? inet_gro_receive+0x1c6/0x250
[ 217.948322] [<ffffffff81541578>] __netif_receive_skb+0x18/0x60
[ 218.019161] [<ffffffff815415e3>] netif_receive_skb_internal+0x23/0x80
[ 218.097281] [<ffffffff81542213>] napi_gro_receive+0xc3/0x110
[ 218.166051] [<ffffffffa00a801f>] ixgbe_clean_rx_irq+0x52f/0xa70 [ixgbe]
[ 218.246255] [<ffffffffa00a9248>] ixgbe_poll+0x438/0x790 [ixgbe]
[ 218.318131] [<ffffffff81541a6e>] net_rx_action+0x1ee/0x320
[ 218.384813] [<ffffffff8109c837>] ? handle_irq_event_percpu+0x167/0x1d0
[ 218.463973] [<ffffffff8105c3c1>] __do_softirq+0x101/0x280
[ 218.529608] [<ffffffff8105c69e>] irq_exit+0x8e/0x90
[ 218.589007] [<ffffffff816dd504>] do_IRQ+0x54/0xd0
[ 218.646323] [<ffffffff816dba02>] common_interrupt+0x82/0x82 |
[Impact]
When using iptables rules affecting bridge traffic, and if affected traffic is flowing through bridge while br_netfilter module is loaded or unloaded, a kernel panic may occur.
[Test Case]
It's difficult to reproduce because of a very small race condition window during br_netfilter load/unload when the module is receiving traffic but has not yet registered its hooks (or, has unregistered its hooks but still has traffic it's processing). A system must be set up using a bridge, and iptable netfilter rules must be set up to process the bridge traffic. Then the system should be rebooted until the problem occurs, or the br_netfilter module should be loaded/unloaded until the problem occurs.
[Regression Potential]
Changing how the br_netfilter module switches its fake dst for a real dst may, if done incorrectly, result in more kernel panics if other code tries to process the br_netfilter module's fake dst.
[Other Info]
The br_netfilter module processes packets traveling through its bridge, and while processing each skb it places a special fake dst onto the skb. When the skb leaves the bridge, it removes the fake dst and places a real dst onto it. However, it uses a hook to do this, and when the br_netfilter module is unloading it unregisters that hook. Any skbs that are currently being processed in the bridge by the br_netfilter module, but that leave the bridge after the hook is unregistered (or, during br_netfilter module load, before the hook is registered) will still have the fake dst; when other code then tries to process that dst, it causes a kernel panic because the dst is invalid.
Recent upstream discussion:
https://www.spinics.net/lists/netdev/msg416912.html
Upstream patch:
https://patchwork.ozlabs.org/patch/738275/
upstream commit is a13b2082ece95247779b9995c4e91b4246bed023
example panic report:
[ 214.518262] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 214.612199] IP: [< (null)>] (null)
[ 214.672744] PGD 0 [ 214.696887] Oops: 0010 [#1] SMP [ 214.735697] Modules linked in: br_netfilter(+) tun 8021q bridge stp llc bonding iTCO_wdt iTCO_vendor_support tpm_tis tpm kvm_intel kvm irqbypass sb_edac edac_core ixgbe mdio ipmi_si ipmi_msghandler lpc_ich mfd_core mousedev evdev igb dca procmemro(O) nokeyctl(O) noptrace(O)
[ 215.029240] CPU: 34 PID: 0 Comm: swapper/34 Tainted: G O 4.4.39 #1
[ 215.116720] Hardware name: Cisco Systems Inc UCSC-C220-M3L/UCSC-C220-M3L, BIOS C220M3.2.0.13a.0.0713160937 07/13/16
[ 215.241644] task: ffff882038fb4380 ti: ffff8810392b0000 task.ti: ffff8810392b0000
[ 215.331207] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 215.420877] RSP: 0018:ffff88103fec3880 EFLAGS: 00010286
[ 215.484436] RAX: ffff881011631000 RBX: ffff881011067100 RCX: 0000000000000000
[ 215.569836] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff881011067100
[ 215.655234] RBP: ffff88103fec38a8 R08: 0000000000000008 R09: ffff8810116300a0
[ 215.740629] R10: 0000000000000000 R11: 0000000000000000 R12: ffff881018917dce
[ 215.826030] R13: ffffffff81c9be00 R14: ffffffff81c9be00 R15: ffff881011630078
[ 215.911432] FS: 0000000000000000(0000) GS:ffff88103fec0000(0000) knlGS:0000000000000000
[ 216.008274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 216.077032] CR2: 0000000000000000 CR3: 0000001011b9d000 CR4: 00000000001406e0
[ 216.162430] Stack:
[ 216.186461] ffffffff8157d7f9 ffff881011067100 ffff881018917dce ffff881011630000
[ 216.275407] ffffffff81c9be00 ffff88103fec3918 ffffffff8157e0db 0000000000000000
[ 216.364352] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 216.453301] Call Trace:
[ 216.482536] <IRQ> [ 216.505533] [<ffffffff8157d7f9>] ? ip_rcv_finish+0x99/0x320
[ 216.575442] [<ffffffff8157e0db>] ip_rcv+0x25b/0x370
[ 216.634842] [<ffffffff81540e0b>] __netif_receive_skb_core+0x2cb/0xa20
[ 216.712965] [<ffffffff81541578>] __netif_receive_skb+0x18/0x60
[ 216.783801] [<ffffffff815415e3>] netif_receive_skb_internal+0x23/0x80
[ 216.861921] [<ffffffff8154165c>] netif_receive_skb+0x1c/0x70
[ 216.930686] [<ffffffffa02f6439>] br_handle_frame_finish+0x1b9/0x5b0 [bridge]
[ 217.016091] [<ffffffff81187a00>] ? ___slab_alloc+0x1d0/0x440
[ 217.084849] [<ffffffffa0584074>] br_nf_pre_routing_finish+0x174/0x3d0 [br_netfilter]
[ 217.178568] [<ffffffffa0584c07>] ? br_nf_pre_routing+0x97/0x470 [br_netfilter]
[ 217.266052] [<ffffffffa02f6280>] ? br_handle_local_finish+0x80/0x80 [bridge]
[ 217.351450] [<ffffffffa0584d17>] br_nf_pre_routing+0x1a7/0x470 [br_netfilter]
[ 217.437891] [<ffffffff81572f6d>] nf_iterate+0x5d/0x70
[ 217.499367] [<ffffffff81572fe4>] nf_hook_slow+0x64/0xc0
[ 217.562928] [<ffffffffa02f69e9>] br_handle_frame+0x1b9/0x290 [bridge]
[ 217.641048] [<ffffffffa02f6280>] ? br_handle_local_finish+0x80/0x80 [bridge]
[ 217.726446] [<ffffffff81540e82>] __netif_receive_skb_core+0x342/0xa20
[ 217.804566] [<ffffffff815a7916>] ? tcp4_gro_receive+0x126/0x1d0
[ 217.876445] [<ffffffff815b7446>] ? inet_gro_receive+0x1c6/0x250
[ 217.948322] [<ffffffff81541578>] __netif_receive_skb+0x18/0x60
[ 218.019161] [<ffffffff815415e3>] netif_receive_skb_internal+0x23/0x80
[ 218.097281] [<ffffffff81542213>] napi_gro_receive+0xc3/0x110
[ 218.166051] [<ffffffffa00a801f>] ixgbe_clean_rx_irq+0x52f/0xa70 [ixgbe]
[ 218.246255] [<ffffffffa00a9248>] ixgbe_poll+0x438/0x790 [ixgbe]
[ 218.318131] [<ffffffff81541a6e>] net_rx_action+0x1ee/0x320
[ 218.384813] [<ffffffff8109c837>] ? handle_irq_event_percpu+0x167/0x1d0
[ 218.463973] [<ffffffff8105c3c1>] __do_softirq+0x101/0x280
[ 218.529608] [<ffffffff8105c69e>] irq_exit+0x8e/0x90
[ 218.589007] [<ffffffff816dd504>] do_IRQ+0x54/0xd0
[ 218.646323] [<ffffffff816dba02>] common_interrupt+0x82/0x82 |
|
| 2017-03-27 13:27:52 |
Tim Gardner |
linux (Ubuntu Zesty): status |
In Progress |
Fix Committed |
|
| 2017-03-29 11:30:22 |
Thadeu Lima de Souza Cascardo |
linux (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2017-03-29 12:02:24 |
Thadeu Lima de Souza Cascardo |
linux (Ubuntu Yakkety): status |
In Progress |
Fix Committed |
|
| 2017-04-04 15:21:23 |
Launchpad Janitor |
linux (Ubuntu Zesty): status |
Fix Committed |
Fix Released |
|
| 2017-04-04 15:21:23 |
Launchpad Janitor |
cve linked |
|
2017-5986 |
|
| 2017-04-13 16:13:27 |
Kleber Sacilotto de Souza |
tags |
sts |
sts verification-needed-xenial |
|
| 2017-04-13 16:15:03 |
Kleber Sacilotto de Souza |
tags |
sts verification-needed-xenial |
sts verification-needed-xenial verification-needed-yakkety |
|
| 2017-04-17 18:36:13 |
Dan Streetman |
tags |
sts verification-needed-xenial verification-needed-yakkety |
sts verification-done-xenial verification-done-yakkety |
|
| 2017-04-24 14:09:16 |
Launchpad Janitor |
linux (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-04-24 14:09:16 |
Launchpad Janitor |
cve linked |
|
2017-6353 |
|
| 2017-04-24 14:11:40 |
Launchpad Janitor |
linux (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
