ip_rcv_finish() NULL pointer kernel panic

[Dan Streetman]

[Impact]

When using iptables rules affecting bridge traffic, and if affected traffic is flowing through bridge while br_netfilter module is loaded or unloaded, a kernel panic may occur.

[Test Case]

It's difficult to reproduce because of a very small race condition window during br_netfilter load/unload when the module is receiving traffic but has not yet registered its hooks (or, has unregistered its hooks but still has traffic it's processing). A system must be set up using a bridge, and iptable netfilter rules must be set up to process the bridge traffic. Then the system should be rebooted until the problem occurs, or the br_netfilter module should be loaded/unloaded until the problem occurs.

[Regression Potential]

Changing how the br_netfilter module switches its fake dst for a real dst may, if done incorrectly, result in more kernel panics if other code tries to process the br_netfilter module's fake dst.

[Other Info]

The br_netfilter module processes packets traveling through its bridge, and while processing each skb it places a special fake dst onto the skb. When the skb leaves the bridge, it removes the fake dst and places a real dst onto it. However, it uses a hook to do this, and when the br_netfilter module is unloading it unregisters that hook. Any skbs that are currently being processed in the bridge by the br_netfilter module, but that leave the bridge after the hook is unregistered (or, during br_netfilter module load, before the hook is registered) will still have the fake dst; when other code then tries to process that dst, it causes a kernel panic because the dst is invalid.

Recent upstream discussion:
https://www.spinics.net/lists/netdev/msg416912.html

Upstream patch:
https://patchwork.ozlabs.org/patch/738275/
upstream commit is a13b2082ece95247779b9995c4e91b4246bed023

example panic report:

[ 214.518262] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 214.612199] IP: [< (null)>] (null)
[ 214.672744] PGD 0 [ 214.696887] Oops: 0010 [#1] SMP [ 214.735697] Modules linked in: br_netfilter(+) tun 8021q bridge stp llc bonding iTCO_wdt iTCO_vendor_support tpm_tis tpm kvm_intel kvm irqbypass sb_edac edac_core ixgbe mdio ipmi_si ipmi_msghandler lpc_ich mfd_core mousedev evdev igb dca procmemro(O) nokeyctl(O) noptrace(O)
[ 215.029240] CPU: 34 PID: 0 Comm: swapper/34 Tainted: G O 4.4.39 #1
[ 215.116720] Hardware name: Cisco Systems Inc UCSC-C220-M3L/UCSC-C220-M3L, BIOS C220M3.2.0.13a.0.0713160937 07/13/16
[ 215.241644] task: ffff882038fb4380 ti: ffff8810392b0000 task.ti: ffff8810392b0000
[ 215.331207] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 215.420877] RSP: 0018:ffff88103fec3880 EFLAGS: 00010286
[ 215.484436] RAX: ffff881011631000 RBX: ffff881011067100 RCX: 0000000000000000
[ 215.569836] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff881011067100
[ 215.655234] RBP: ffff88103fec38a8 R08: 0000000000000008 R09: ffff8810116300a0
[ 215.740629] R10: 0000000000000000 R11: 0000000000000000 R12: ffff881018917dce
[ 215.826030] R13: ffffffff81c9be00 R14: ffffffff81c9be00 R15: ffff881011630078
[ 215.911432] FS: 0000000000000000(0000) GS:ffff88103fec0000(0000) knlGS:0000000000000000
[ 216.008274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 216.077032] CR2: 0000000000000000 CR3: 0000001011b9d000 CR4: 00000000001406e0
[ 216.162430] Stack:
[ 216.186461] ffffffff8157d7f9 ffff881011067100 ffff881018917dce ffff881011630000
[ 216.275407] ffffffff81c9be00 ffff88103fec3918 ffffffff8157e0db 0000000000000000
[ 216.364352] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 216.453301] Call Trace:
[ 216.482536] <IRQ> [ 216.505533] [<ffffffff8157d7f9>] ? ip_rcv_finish+0x99/0x320
[ 216.575442] [<ffffffff8157e0db>] ip_rcv+0x25b/0x370
[ 216.634842] [<ffffffff81540e0b>] __netif_receive_skb_core+0x2cb/0xa20
[ 216.712965] [<ffffffff81541578>] __netif_receive_skb+0x18/0x60
[ 216.783801] [<ffffffff815415e3>] netif_receive_skb_internal+0x23/0x80
[ 216.861921] [<ffffffff8154165c>] netif_receive_skb+0x1c/0x70
[ 216.930686] [<ffffffffa02f6439>] br_handle_frame_finish+0x1b9/0x5b0 [bridge]
[ 217.016091] [<ffffffff81187a00>] ? ___slab_alloc+0x1d0/0x440
[ 217.084849] [<ffffffffa0584074>] br_nf_pre_routing_finish+0x174/0x3d0 [br_netfilter]
[ 217.178568] [<ffffffffa0584c07>] ? br_nf_pre_routing+0x97/0x470 [br_netfilter]
[ 217.266052] [<ffffffffa02f6280>] ? br_handle_local_finish+0x80/0x80 [bridge]
[ 217.351450] [<ffffffffa0584d17>] br_nf_pre_routing+0x1a7/0x470 [br_netfilter]
[ 217.437891] [<ffffffff81572f6d>] nf_iterate+0x5d/0x70
[ 217.499367] [<ffffffff81572fe4>] nf_hook_slow+0x64/0xc0
[ 217.562928] [<ffffffffa02f69e9>] br_handle_frame+0x1b9/0x290 [bridge]
[ 217.641048] [<ffffffffa02f6280>] ? br_handle_local_finish+0x80/0x80 [bridge]
[ 217.726446] [<ffffffff81540e82>] __netif_receive_skb_core+0x342/0xa20
[ 217.804566] [<ffffffff815a7916>] ? tcp4_gro_receive+0x126/0x1d0
[ 217.876445] [<ffffffff815b7446>] ? inet_gro_receive+0x1c6/0x250
[ 217.948322] [<ffffffff81541578>] __netif_receive_skb+0x18/0x60
[ 218.019161] [<ffffffff815415e3>] netif_receive_skb_internal+0x23/0x80
[ 218.097281] [<ffffffff81542213>] napi_gro_receive+0xc3/0x110
[ 218.166051] [<ffffffffa00a801f>] ixgbe_clean_rx_irq+0x52f/0xa70 [ixgbe]
[ 218.246255] [<ffffffffa00a9248>] ixgbe_poll+0x438/0x790 [ixgbe]
[ 218.318131] [<ffffffff81541a6e>] net_rx_action+0x1ee/0x320
[ 218.384813] [<ffffffff8109c837>] ? handle_irq_event_percpu+0x167/0x1d0
[ 218.463973] [<ffffffff8105c3c1>] __do_softirq+0x101/0x280
[ 218.529608] [<ffffffff8105c69e>] irq_exit+0x8e/0x90
[ 218.589007] [<ffffffff816dd504>] do_IRQ+0x54/0xd0
[ 218.646323] [<ffffffff816dba02>] common_interrupt+0x82/0x82

[Dan Streetman (ddstreet)]

I verified the Trusty kernel isn't affected, this only applies to Xenial and later.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.10.0-15.17

---------------
linux (4.10.0-15.17) zesty; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1675868

  * In ZZ-BML (POWER9):ubuntu17.04 installation Fails (LP: #1675771)
    - powerpc/64s: fix handling of non-synchronous machine checks
    - powerpc/64s: allow machine check handler to set severity and initiator
    - powerpc/64s: POWER9 machine check handler

  * [Feature] R3 mwait support for Knights Mill (LP: #1637550)
    - x86/cpufeature: Enable RING3MWAIT for Knights Landing
    - x86/cpufeature: Enable RING3MWAIT for Knights Mill
    - x86/msr: Add MSR_MISC_FEATURE_ENABLES and RING3MWAIT bit
    - x86/elf: Add HWCAP2 to expose ring 3 MONITOR/MWAIT
    - x86/cpufeature: Add RING3MWAIT to CPU features

  * [Feature] GLK:New device IDs (LP: #1645951)
    - mfd: intel-lpss: Add Intel Gemini Lake PCI IDs
    - pwm: lpss: Add Intel Gemini Lake PCI ID
    - i2c: i801: Add support for Intel Gemini Lake
    - spi: pxa2xx: Add support for Intel Gemini Lake
    - [Config] CONFIG_PINCTRL_GEMINILAKE=m
    - pinctrl: intel: Add Intel Gemini Lake pin controller support

  * Zesty update to v4.10.5 stable release (LP: #1675032)
    - net/mlx5e: Register/unregister vport representors on interface attach/detach
    - net/mlx5e: Do not reduce LRO WQE size when not using build_skb
    - net/mlx5e: Fix broken CQE compression initialization
    - net/mlx5e: Update MPWQE stride size when modifying CQE compress state
    - net/mlx5e: Fix wrong CQE decompression
    - vxlan: correctly validate VXLAN ID against VXLAN_N_VID
    - vti6: return GRE_KEY for vti6
    - vxlan: don't allow overwrite of config src addr
    - ipv4: add missing initialization for flowi4_uid
    - ipv4: mask tos for input route
    - sctp: set sin_port for addr param when checking duplicate address
    - net sched actions: decrement module reference count after table flush.
    - l2tp: avoid use-after-free caused by l2tp_ip_backlog_recv
    - vxlan: lock RCU on TX path
    - geneve: lock RCU on TX path
    - mlxsw: spectrum_router: Avoid potential packets loss
    - net: bridge: allow IPv6 when multicast flood is disabled
    - net: don't call strlen() on the user buffer in packet_bind_spkt()
    - net: net_enable_timestamp() can be called from irq contexts
    - ipv6: orphan skbs in reassembly unit
    - dccp: Unlock sock before calling sk_free()
    - amd-xgbe: Stop the PHY before releasing interrupts
    - amd-xgbe: Be sure to set MDIO modes on device (re)start
    - amd-xgbe: Don't overwrite SFP PHY mod_absent settings
    - bonding: use ETH_MAX_MTU as max mtu
    - strparser: destroy workqueue on module exit
    - tcp: fix various issues for sockets morphing to listen state
    - net: fix socket refcounting in skb_complete_wifi_ack()
    - net: fix socket refcounting in skb_complete_tx_timestamp()
    - net/sched: act_skbmod: remove unneeded rcu_read_unlock in tcf_skbmod_dump
    - dccp: fix use-after-free in dccp_feat_activate_values
    - team: use ETH_MAX_MTU as max mtu
    - vrf: Fix use-after-free in vrf_xmit
    - net/tunnel: set inner protocol in network gro hooks
    - uapi: fix linux/packet_diag.h use...

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'. If the problem still exists, change the tag 'verification-needed-yakkety' to 'verification-failed-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Dan Streetman (ddstreet)]

With a virsh guest installed with standard xenial, using the 4.4.0-72-generic kernel, I created a bridge and attached the virtio interface to it. I then started iperf3 -s on the guest, and started iperf3 -c GUESTIP -t 300 on the host, to create continuous traffic to the guest.

Then I added a simple iptables rule:
$ sudo iptables -A INPUT -s 192.168.122.0/24 -j MARK --set-mark 1234

that rule matches all incoming iperf3 packets. Then, simply modprobing the br_netfilter module reproduces this kernel oops.

I then added xenial-proposed to the apt sources, and upgraded the kernel to version 4.4.0-74-generic, and re-ran the above test, which allowed the br_netfilter module to be loaded with no kernel oops or other problem.

[Dan Streetman (ddstreet)]

Same test setup as above, but using yakkety with kernel 4.8.0-46-generic, reproduces the kernel oops. After upgrading to kernel 4.8.0-48-generic, the kernel oops no longer happens when modprobing or rmmoding br_netfilter.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-75.96

---------------
linux (4.4.0-75.96) xenial; urgency=low

  * linux: 4.4.0-75.96 -proposed tracker (LP: #1684441)

  * [Hyper-V] hv: util: move waiting for release to hv_utils_transport itself
    (LP: #1682561)
    - Drivers: hv: util: move waiting for release to hv_utils_transport itself

linux (4.4.0-74.95) xenial; urgency=low

  * linux: 4.4.0-74.95 -proposed tracker (LP: #1682041)

  * [Hyper-V] hv: vmbus: Raise retry/wait limits in vmbus_post_msg()
    (LP: #1681893)
    - Drivers: hv: vmbus: Raise retry/wait limits in vmbus_post_msg()

linux (4.4.0-73.94) xenial; urgency=low

  * linux: 4.4.0-73.94 -proposed tracker (LP: #1680416)

  * CVE-2017-6353
    - sctp: deny peeloff operation on asocs with threads sleeping on it

  * vfat: missing iso8859-1 charset (LP: #1677230)
    - [Config] NLS_ISO8859_1=y

  * Regression: KVM modules should be on main kernel package (LP: #1678099)
    - [Config] powerpc: Add kvm-hv and kvm-pr to the generic inclusion list

  * linux-lts-xenial 4.4.0-63.84~14.04.2 ADT test failure with linux-lts-xenial
    4.4.0-63.84~14.04.2 (LP: #1664912)
    - SAUCE: apparmor: fix link auditing failure due to, uninitialized var

  * regession tests failing after stackprofile test is run (LP: #1661030)
    - SAUCE: fix regression with domain change in complain mode

  * Permission denied and inconsistent behavior in complain mode with 'ip netns
    list' command (LP: #1648903)
    - SAUCE: fix regression with domain change in complain mode

  * unexpected errno=13 and disconnected path when trying to open /proc/1/ns/mnt
    from a unshared mount namespace (LP: #1656121)
    - SAUCE: apparmor: null profiles should inherit parent control flags

  * apparmor refcount leak of profile namespace when removing profiles
    (LP: #1660849)
    - SAUCE: apparmor: fix ns ref count link when removing profiles from policy

  * tor in lxd: apparmor="DENIED" operation="change_onexec"
    namespace="root//CONTAINERNAME_<var-lib-lxd>" profile="unconfined"
    name="system_tor" (LP: #1648143)
    - SAUCE: apparmor: Fix no_new_privs blocking change_onexec when using stacked
      namespaces

  * apparmor oops in bind_mnt when dev_path lookup fails (LP: #1660840)
    - SAUCE: apparmor: fix oops in bind_mnt when dev_path lookup fails

  * apparmor auditing denied access of special apparmor .null fi\ le
    (LP: #1660836)
    - SAUCE: apparmor: Don't audit denied access of special apparmor .null file

  * apparmor label leak when new label is unused (LP: #1660834)
    - SAUCE: apparmor: fix label leak when new label is unused

  * apparmor reference count bug in label_merge_insert() (LP: #1660833)
    - SAUCE: apparmor: fix reference count bug in label_merge_insert()

  * apparmor's raw_data file in securityfs is sometimes truncated (LP: #1638996)
    - SAUCE: apparmor: fix replacement race in reading rawdata

  * unix domain socket cross permission check failing with nested namespaces
    (LP: #1660832)
    - SAUCE: apparmor: fix cross ns perm of unix domain sockets

  * Xenial update to v4.4.59 stable release (LP: #1678960)
    - xfrm: policy: init locks early
    - virtio_balloon: init ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.8.0-49.52

---------------
linux (4.8.0-49.52) yakkety; urgency=low

  * linux: 4.8.0-49.52 -proposed tracker (LP: #1684427)

  * [Hyper-V] hv: util: move waiting for release to hv_utils_transport itself
    (LP: #1682561)
    - Drivers: hv: util: move waiting for release to hv_utils_transport itself

linux (4.8.0-48.51) yakkety; urgency=low

  * linux: 4.8.0-48.51 -proposed tracker (LP: #1682034)

  * [Hyper-V] hv: vmbus: Raise retry/wait limits in vmbus_post_msg()
    (LP: #1681893)
    - Drivers: hv: vmbus: Raise retry/wait limits in vmbus_post_msg()

linux (4.8.0-47.50) yakkety; urgency=low

  * linux: 4.8.0-47.50 -proposed tracker (LP: #1679678)

  * CVE-2017-6353
    - sctp: deny peeloff operation on asocs with threads sleeping on it

  * CVE-2017-5986
    - sctp: avoid BUG_ON on sctp_wait_for_sndbuf

  * vfat: missing iso8859-1 charset (LP: #1677230)
    - [Config] NLS_ISO8859_1=y

  * [Hyper-V] pci-hyperv: Use device serial number as PCI domain (LP: #1667527)
    - net/mlx4_core: Use cq quota in SRIOV when creating completion EQs

  * Regression: KVM modules should be on main kernel package (LP: #1678099)
    - [Config] powerpc: Add kvm-hv and kvm-pr to the generic inclusion list

  * linux-lts-xenial 4.4.0-63.84~14.04.2 ADT test failure with linux-lts-xenial
    4.4.0-63.84~14.04.2 (LP: #1664912)
    - SAUCE: apparmor: fix link auditing failure due to, uninitialized var

  * regession tests failing after stackprofile test is run (LP: #1661030)
    - SAUCE: fix regression with domain change in complain mode

  * Permission denied and inconsistent behavior in complain mode with 'ip netns
    list' command (LP: #1648903)
    - SAUCE: fix regression with domain change in complain mode

  * unexpected errno=13 and disconnected path when trying to open /proc/1/ns/mnt
    from a unshared mount namespace (LP: #1656121)
    - SAUCE: apparmor: null profiles should inherit parent control flags

  * apparmor refcount leak of profile namespace when removing profiles
    (LP: #1660849)
    - SAUCE: apparmor: fix ns ref count link when removing profiles from policy

  * tor in lxd: apparmor="DENIED" operation="change_onexec"
    namespace="root//CONTAINERNAME_<var-lib-lxd>" profile="unconfined"
    name="system_tor" (LP: #1648143)
    - SAUCE: apparmor: Fix no_new_privs blocking change_onexec when using stacked
      namespaces

  * apparmor oops in bind_mnt when dev_path lookup fails (LP: #1660840)
    - SAUCE: apparmor: fix oops in bind_mnt when dev_path lookup fails

  * apparmor auditing denied access of special apparmor .null fi\ le
    (LP: #1660836)
    - SAUCE: apparmor: Don't audit denied access of special apparmor .null file

  * apparmor label leak when new label is unused (LP: #1660834)
    - SAUCE: apparmor: fix label leak when new label is unused

  * apparmor reference count bug in label_merge_insert() (LP: #1660833)
    - SAUCE: apparmor: fix reference count bug in label_merge_insert()

  * apparmor's raw_data file in securityfs is sometimes truncated (LP: #1638996)
    - SAUCE: apparmor: fix replacement race in reading rawdata

  * unix domain socket cross permission check failing with n...