iptables connlimit allows more connections than the limit when using multiple CPUs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| linux (Ubuntu) |
Medium
|
Mauricio Faria de Oliveira | ||
| Trusty |
Medium
|
Unassigned | ||
| Xenial |
Medium
|
Unassigned | ||
| Bionic |
Medium
|
Unassigned | ||
| Cosmic |
Medium
|
Unassigned |
Bug Description
[Impact]
* The iptables connection count/limit rules can be breached
with multithreaded network driver/
due to a race in the conncount/connlimit code.
* For example:
# iptables -A INPUT -p tcp -m tcp --syn --dport 7777 \
-m connlimit --connlimit-above 2000 --connlimit-mask 0 \
-j DROP
* The fix is a backport from an upstream commit that resolves
the problem (plus dependencies for a cleaner backport) that
address the race condition:
commit b36e4523d4d5 ("netfilter: nf_conncount: fix garbage
collection confirm race").
[Test Case]
* Server-side: (relevant kernel side)
(limit TCP port 7777 to only 2000 connections)
# iptables -A INPUT -p tcp -m tcp --syn --dport 7777 \
-m connlimit --connlimit-above 2000 --connlimit-mask 0 \
-j DROP
# ulimit -SHn 65000 # increase number of open files
# ruby server.rb # multi-threaded server
* Client-side:
# ulimit -SHn 65000
# ruby client.rb <server ip> <port> <target # connections> <# threads>
<test output>
* Results with Original kernel:
(client achieves target of 6000 connections > limit of 2000 connections)
# ruby client.rb 10.230.56.100 7777 6000 3
1
2
3
<...>
6000
Target reached. Thread finishing
6001
Target reached. Thread finishing
6002
Target reached. Thread finishing
Threads done. 6002 connections
press enter to exit
* Results with Modified kernel:
(client is limited to 2000 connections, and times out afterward)
# ruby client.rb 10.230.56.100 7777 6000 3
1
2
3
<...>
2000
<... blocks for a few minutes ...>
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
Threads done. 2000 connections
press enter to exit
* Test cases possibly available upon request,
depending on original author's permission.
[Regression Potential]
* The patchset has been reviewed by a netfilter maintainer [1] in
stable mailing list, and was considered OK for 4.14, and that's
essentially the same backport for 4.15 and 4.4.
* The changes are limited to netfilter connlimit/conncount (names
change between older/newer kernel versions).
[Other Info]
* The backport for 4.14 [2] is applied as of 4.14.92.
[1] https:/
[2] https:/
Changed in linux (Ubuntu): | |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
status: | New → Confirmed |
description: | updated |
description: | updated |
Changed in linux (Ubuntu): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Cosmic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Xenial): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Bionic): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu Cosmic): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu Xenial): | |
status: | New → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Confirmed → Fix Committed |
Mauricio Faria de Oliveira (mfo) wrote : | #1 |
Brad Figg (brad-figg) wrote : | #2 |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-cosmic |
tags: | added: verification-needed-bionic |
Brad Figg (brad-figg) wrote : | #3 |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
Mauricio Faria de Oliveira (mfo) wrote : | #4 |
Verification done for Cosmic.
cosmic-proposed:
---
- server:
root@shuckle:~# uname -a
Linux shuckle 4.18.0-14-generic #15-Ubuntu SMP Mon Jan 14 09:01:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
- client:
root@dixie:~# ruby client.rb 10.230.56.116 7777 6000 3
Connecting to ["10.230.
1
2
3
...
1998
1999
2000
<blocks then times out>
Mauricio Faria de Oliveira (mfo) wrote : | #5 |
Verification done for Bionic.
bionic-proposed:
---
- server:
root@shuckle:~# uname -a
Linux shuckle 4.15.0-44-generic #47-Ubuntu SMP Mon Jan 14 11:26:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
- client:
root@dixie:~# ruby client.rb 10.230.56.116 7777 6000 3
Connecting to ["10.230.
1
2
3
...
1998
1999
2000
<blocks then times out>
tags: |
added: verification-done-bionic verification-done-cosmic removed: verification-needed-bionic verification-needed-cosmic |
Brad Figg (brad-figg) wrote : | #6 |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-xenial |
Mauricio Faria de Oliveira (mfo) wrote : | #7 |
Verification done on Xenial.
- server:
root@shuckle:~# uname -a
Linux shuckle 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@shuckle:~# iptables -F
root@shuckle:~# iptables -A INPUT -p tcp -m tcp --syn --dport 7777 -m connlimit --connlimit-above 2000 --connlimit-mask 0 -j DROP
root@shuckle:~# ulimit -SHn 65000
root@shuckle:~# ruby server.rb
- client:
root@dixie:~# ruby client.rb 10.230.56.116 7777 6000 3
Connecting to ["10.230.
1
2
3
...
2000
<blocks>
tags: |
added: verification-done-xenial removed: verification-needed-xenial |
Launchpad Janitor (janitor) wrote : | #8 |
This bug was fixed in the package linux - 4.15.0-44.47
---------------
linux (4.15.0-44.47) bionic; urgency=medium
* linux: 4.15.0-44.47 -proposed tracker (LP: #1811419)
* Packaging resync (LP: #1786013)
- [Packaging] update helper scripts
* CPU hard lockup with rigorous writes to NVMe drive (LP: #1810998)
- blk-wbt: pass in enum wbt_flags to get_rq_wait()
- blk-wbt: Avoid lock contention and thundering herd issue in wbt_wait
- blk-wbt: move disable check into get_limit()
- blk-wbt: use wq_has_sleeper() for wq active check
- blk-wbt: fix has-sleeper queueing check
- blk-wbt: abstract out end IO completion handler
- blk-wbt: improve waking of tasks
* To reduce the Realtek USB cardreader power consumption (LP: #1811337)
- mmc: sdhci: Disable 1.8v modes (HS200/HS400/UHS) if controller can't support
1.8v
- mmc: core: Introduce MMC_CAP_
- mmc: rtsx_usb_sdmmc: Don't runtime resume the device while changing led
- mmc: rtsx_usb: Use MMC_CAP2_NO_SDIO
- mmc: rtsx_usb: Enable MMC_CAP_ERASE to allow erase/discard/trim requests
- mmc: rtsx_usb_sdmmc: Re-work runtime PM support
- mmc: rtsx_usb_sdmmc: Re-work card detection/removal support
- memstick: rtsx_usb_ms: Add missing pm_runtime_
- misc: rtsx_usb: Use USB remote wakeup signaling for card insertion detection
- memstick: Prevent memstick host from getting runtime suspended during card
detection
- memstick: rtsx_usb_ms: Use ms_dev() helper
- memstick: rtsx_usb_ms: Support runtime power management
* Support non-strict iommu mode on arm64 (LP: #1806488)
- iommu/io-
- iommu/arm-smmu-v3: Implement flush_iotlb_all hook
- iommu/dma: Add support for non-strict mode
- iommu: Add "iommu.strict" command line option
- iommu/io-
- iommu/arm-smmu-v3: Add support for non-strict mode
- iommu/io-
- iommu/arm-smmu: Support non-strict mode
* ELAN900C:00 04F3:2844 touchscreen doesn't work (LP: #1811335)
- pinctrl: cannonlake: Fix community ordering for H variant
- pinctrl: cannonlake: Fix HOSTSW_OWN register offset of H variant
* Add Cavium ThunderX2 SoC UNCORE PMU driver (LP: #1811200)
- perf: Export perf_event_
- Documentation: perf: Add documentation for ThunderX2 PMU uncore driver
- drivers/perf: Add Cavium ThunderX2 SoC UNCORE PMU driver
- [Config] New config CONFIG_
* Update hisilicon SoC-specific drivers (LP: #1810457)
- SAUCE: Revert "net: hns3: Updates RX packet info fetch in case of multi BD"
- Revert "UBUNTU: SAUCE: {topost} net: hns3: separate roce from nic when
resetting"
- Revert "UBUNTU: SAUCE: {topost} net: hns3: Use roce handle when calling roce
callback function"
- Revert "UBUNTU: SAUCE: {topost} net: hns3: Add calling roce callback
function when link status change"
- Revert "UBUNTU: SAUCE: {topost} net: hns3: optimize the process of notifying
roce client"
- Revert "UBUNTU: S...
Changed in linux (Ubuntu Bionic): | |
status: | Fix Committed → Fix Released |
Changed in linux (Ubuntu Trusty): | |
importance: | Undecided → Medium |
status: | New → Fix Committed |
Launchpad Janitor (janitor) wrote : | #9 |
This bug was fixed in the package linux - 4.4.0-142.168
---------------
linux (4.4.0-142.168) xenial; urgency=medium
* linux: 4.4.0-142.168 -proposed tracker (LP: #1811846)
* Packaging resync (LP: #1786013)
- [Packaging] update helper scripts
* iptables connlimit allows more connections than the limit when using
multiple CPUs (LP: #1811094)
- netfilter: xt_connlimit: don't store address in the conn nodes
- SAUCE: netfilter: xt_connlimit: remove the 'addr' parameter in add_hlist()
- netfilter: nf_conncount: expose connection list interface
- netfilter: nf_conncount: Fix garbage collection with zones
- netfilter: nf_conncount: fix garbage collection confirm race
- netfilter: nf_conncount: don't skip eviction when age is negative
* CVE-2017-5715
- SAUCE: x86/speculation: Cleanup IBPB runtime control handling
- SAUCE: x86/speculation: Cleanup IBRS runtime control handling
- SAUCE: x86/speculation: Use x86_spec_ctrl_base in entry/exit code
- SAUCE: x86/speculation: Move RSB_CTXSW hunk
* Xenial update: 4.4.167 upstream stable release (LP: #1811077)
- media: em28xx: Fix use-after-free when disconnecting
- Revert "wlcore: Add missing PM call for
wlcore_
- rapidio/rionet: do not free skb before reading its length
- s390/qeth: fix length check in SNMP processing
- usbnet: ipheth: fix potential recvmsg bug and recvmsg bug 2
- kvm: mmu: Fix race in emulated page table writes
- xtensa: enable coprocessors that are being flushed
- xtensa: fix coprocessor context offset definitions
- Btrfs: ensure path name is null terminated at btrfs_control_ioctl
- ALSA: wss: Fix invalid snd_free_pages() at error path
- ALSA: ac97: Fix incorrect bit shift at AC97-SPSA control write
- ALSA: control: Fix race between adding and removing a user element
- ALSA: sparc: Fix invalid snd_free_pages() at error path
- ext2: fix potential use after free
- dmaengine: at_hdmac: fix memory leak in at_dma_xlate()
- dmaengine: at_hdmac: fix module unloading
- btrfs: release metadata before running delayed refs
- USB: usb-storage: Add new IDs to ums-realtek
- usb: core: quirks: add RESET_RESUME quirk for Cherry G230 Stream series
- misc: mic/scif: fix copy-paste error in scif_create_
- Kbuild: suppress packed-not-aligned warning for default setting only
- exec: avoid gcc-8 warning for get_task_comm
- disable stringop truncation warnings for now
- kobject: Replace strncpy with memcpy
- unifdef: use memcpy instead of strncpy
- kernfs: Replace strncpy with memcpy
- ip_tunnel: Fix name string concatenate in __ip_tunnel_
- drm: gma500: fix logic error
- scsi: bfa: convert to strlcpy/strlcat
- staging: rts5208: fix gcc-8 logic error warning
- kdb: use memmove instead of overlapping memcpy
- iser: set sector for ambiguous mr status errors
- uprobes: Fix handle_swbp() vs. unregister() + register() race once more
- MIPS: ralink: Fix mt7620 nd_sd pinmux
- mips: fix mips_get_
- drm/ast: Fix incorrect free on ioregs
...
Changed in linux (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #10 |
This bug was fixed in the package linux - 4.18.0-14.15
---------------
linux (4.18.0-14.15) cosmic; urgency=medium
* linux: 4.18.0-14.15 -proposed tracker (LP: #1811406)
* CPU hard lockup with rigorous writes to NVMe drive (LP: #1810998)
- blk-wbt: Avoid lock contention and thundering herd issue in wbt_wait
- blk-wbt: move disable check into get_limit()
- blk-wbt: use wq_has_sleeper() for wq active check
- blk-wbt: fix has-sleeper queueing check
- blk-wbt: abstract out end IO completion handler
- blk-wbt: improve waking of tasks
* To reduce the Realtek USB cardreader power consumption (LP: #1811337)
- mmc: core: Introduce MMC_CAP_
- mmc: rtsx_usb_sdmmc: Don't runtime resume the device while changing led
- mmc: rtsx_usb_sdmmc: Re-work runtime PM support
- mmc: rtsx_usb_sdmmc: Re-work card detection/removal support
- memstick: rtsx_usb_ms: Add missing pm_runtime_
- misc: rtsx_usb: Use USB remote wakeup signaling for card insertion detection
- memstick: Prevent memstick host from getting runtime suspended during card
detection
- memstick: rtsx_usb_ms: Use ms_dev() helper
- memstick: rtsx_usb_ms: Support runtime power management
* Support non-strict iommu mode on arm64 (LP: #1806488)
- iommu/io-
- iommu/arm-smmu-v3: Implement flush_iotlb_all hook
- iommu/dma: Add support for non-strict mode
- iommu: Add "iommu.strict" command line option
- iommu/io-
- iommu/arm-smmu-v3: Add support for non-strict mode
- iommu/io-
- iommu/arm-smmu: Support non-strict mode
* [Regression] crashkernel fails on HiSilicon D05 (LP: #1806766)
- efi: honour memory reservations passed via a linux specific config table
- efi/arm: libstub: add a root memreserve config table
- efi: add API to reserve memory persistently across kexec reboot
- irqchip/gic-v3-its: Change initialization ordering for LPIs
- irqchip/gic-v3-its: Simplify LPI_PENDBASE_SZ usage
- irqchip/gic-v3-its: Split property table clearing from allocation
- irqchip/gic-v3-its: Move pending table allocation to init time
- irqchip/gic-v3-its: Keep track of property table's PA and VA
- irqchip/gic-v3-its: Allow use of pre-programmed LPI tables
- irqchip/gic-v3-its: Use pre-programmed redistributor tables with kdump
kernels
- irqchip/gic-v3-its: Check that all RDs have the same property table
- irqchip/gic-v3-its: Register LPI tables with EFI config table
- irqchip/gic-v3-its: Allow use of LPI tables in reserved memory
- arm64: memblock: don't permit memblock resizing until linear mapping is up
- efi/arm: Defer persistent reservations until after paging_init()
- efi: Permit calling efi_mem_
- efi: Prevent GICv3 WARN() by mapping the memreserve table before first use
* ELAN900C:00 04F3:2844 touchscreen doesn't work (LP: #1811335)
- pinctrl: cannonlake: Fix community ordering for H variant
- pinctrl: c...
Changed in linux (Ubuntu Cosmic): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #11 |
This bug was fixed in the package linux - 4.19.0-12.13
---------------
linux (4.19.0-12.13) disco; urgency=medium
* linux: 4.19.0-12.13 -proposed tracker (LP: #1813664)
* kernel oops in bcache module (LP: #1793901)
- SAUCE: bcache: never writeback a discard operation
* Disco update: 4.19.18 upstream stable release (LP: #1813611)
- ipv6: Consider sk_bound_dev_if when binding a socket to a v4 mapped address
- mlxsw: spectrum: Disable lag port TX before removing it
- mlxsw: spectrum_switchdev: Set PVID correctly during VLAN deletion
- net: dsa: mv88x6xxx: mv88e6390 errata
- net, skbuff: do not prefer skb allocation fails early
- qmi_wwan: add MTU default to qmap network interface
- ipv6: Take rcu_read_lock in __inet6_bind for mapped addresses
- net: clear skb->tstamp in bridge forwarding path
- netfilter: ipset: Allow matching on destination MAC address for mac and
ipmac sets
- gpio: pl061: Move irq_chip definition inside struct pl061
- drm/amd/display: Guard against null stream_state in set_crc_source
- drm/amdkfd: fix interrupt spin lock
- ixgbe: allow IPsec Tx offload in VEPA mode
- platform/x86: asus-wmi: Tell the EC the OS will handle the display off
hotkey
- e1000e: allow non-monotonic SYSTIM readings
- usb: typec: tcpm: Do not disconnect link for self powered devices
- selftests/bpf: enable (uncomment) all tests in test_libbpf.sh
- of: overlay: add missing of_node_put() after add new node to changeset
- writeback: don't decrement wb->refcnt if !wb->bdi
- serial: set suppress_bind_attrs flag only if builtin
- bpf: Allow narrow loads with offset > 0
- ALSA: oxfw: add support for APOGEE duet FireWire
- x86/mce: Fix -Wmissing-
- MIPS: SiByte: Enable swiotlb for SWARM, LittleSur and BigSur
- crypto: ecc - regularize scalar for scalar multiplication
- arm64: perf: set suppress_bind_attrs flag to true
- drm/atomic-helper: Complete fake_commit-
- clk: meson: meson8b: fix incorrect divider mapping in cpu_scale_table
- samples: bpf: fix: error handling regarding kprobe_events
- usb: gadget: udc: renesas_usb3: add a safety connection way for
forced_
- fpga: altera-cvp: fix probing for multiple FPGAs on the bus
- selinux: always allow mounting submounts
- ASoC: pcm3168a: Don't disable pcm3168a when CONFIG_PM defined
- scsi: qedi: Check for session online before getting iSCSI TLV data.
- drm/amdgpu: Reorder uvd ring init before uvd resume
- rxe: IB_WR_REG_MR does not capture MR's iova field
- efi/libstub: Disable some warnings for x86{,_64}
- jffs2: Fix use of uninitialized delayed_work, lockdep breakage
- clk: imx: make mux parent strings const
- pstore/ram: Do not treat empty buffers as valid
- media: uvcvideo: Refactor teardown of uvc on USB disconnect
- powerpc/xmon: Fix invocation inside lock region
- powerpc/
- media: firewire: Fix app_info parameter type in avc_ca{,_app}_info
- ASoC: use dma_ops of parent device for acp_audio_dma
- media: ve...
Changed in linux (Ubuntu): | |
status: | Fix Committed → Fix Released |
Brad Figg (brad-figg) wrote : | #12 |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-trusty |
Mauricio Faria de Oliveira (mfo) wrote : | #13 |
Verification successful on trusty-proposed.
Updates kernel (goes above 2000 connections)
---
root@petilil:~# uname -a
Linux petilil 3.13.0-165-generic #215-Ubuntu SMP Wed Jan 16 11:46:47 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@petilil:~# iptables -F
root@petilil:~# iptables -A INPUT -p tcp -m tcp --syn --dport 7777 -m connlimit --connlimit-above 2000 --connlimit-mask 0 -j DROP
root@petilil:~# ulimit -SHn 65000
root@petilil:~# ruby ~ubuntu/server.rb
root@rotom:~# ulimit -SHn 65000
root@rotom:~# ruby client.rb 10.230.56.100 7777 6000 3
1
2
3
...
6000
Target reached. Thread finishing
6001
Target reached. Thread finishing
6002
Target reached. Thread finishing
Threads done. 6002 connections
press enter to exit
Proposed kernel (stops at 2000 connections)
---
root@petilil:~# uname -a
Linux petilil 3.13.0-166-generic #216-Ubuntu SMP Thu Feb 7 14:07:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@petilil:~# iptables -F
root@petilil:~# iptables -A INPUT -p tcp -m tcp --syn --dport 7777 -m connlimit --connlimit-above 2000 --connlimit-mask 0 -j DROP
root@petilil:~# ulimit -SHn 65000
root@petilil:~# ruby ~ubuntu/server.rb
root@rotom:~# ulimit -SHn 65000
root@rotom:~# ruby client.rb 10.230.56.100 7777 6000 3
ruby: No such file or directory -- client.rb (LoadError)
root@rotom:~# cd /home/mfo/sf192750/
root@rotom:
Connecting to ["10.230.
1
2
3
...
2000
<blocks for a while>
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
failed to create connection: Connection timed out - connect(2) for "10.230.56.100" port 7777
Threads done. 2000 connections
press enter to exit
tags: |
added: verification-done-trusty removed: verification-needed-trusty |
[SRU T][PATCH 0/3] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097878. html
https:/
[SRU X][PATCH 0/6] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097698. html
https:/
[SRU B][PATCH 0/5] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097705. html
https:/
[SRU C, D/Unstable][PATCH 0/1] netfilter: nf_conncount: fix for LP#1811094 /lists. ubuntu. com/archives/ kernel- team/2019- January/ 097711. html
https:/