TCP memory leak, slow network (arm64)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Philip Cox | ||
Mantic |
Fix Released
|
Medium
|
Philip Cox | ||
Noble |
Fix Released
|
Medium
|
Philip Cox |
Bug Description
Hello! 👋
We have Ubuntu OS-based servers running in both AWS and Azure clouds. These servers are handling thousands of connections, and we've been experiencing issues with TCP memory usage since upgrading to Ubuntu 22.04.3 from 22.04.2.
$ cat /proc/net/sockstat
sockets: used 6642
TCP: inuse 5962 orphan 0 tw 292 alloc 6008 mem 128989
UDP: inuse 5 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
As shown in the output below, even after stopping all possible services and closing all open connections, the TCP memory usage remains high and only decreases very slowly.
$ cat /proc/net/sockstat
sockets: used 138
TCP: inuse 2 orphan 0 tw 0 alloc 3 mem 128320
UDP: inuse 3 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
I have attached a screenshot of linear TCP memory usage growth, which we believe may indicate a TCP memory leak
When net.ipv4.tcp_mem limit is reached, it causes network slowdown
We've never had these issues before, and the only solution we've found so far is to reboot the node. Do you have any suggestions on how to troubleshoot further?
Thank you for any help or guidance you can provide!
ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-
ProcVersionSign
Uname: Linux 6.2.0-1015-aws aarch64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: arm64
CasperMD5CheckR
CloudArchitecture: aarch64
CloudID: aws
CloudName: aws
CloudPlatform: ec2
CloudRegion: us-west-2
CloudSubPlatform: metadata (http://
Date: Mon Dec 4 13:13:08 2023
Ec2AMI: ami-095a68e28e7
Ec2AMIManifest: (unknown)
Ec2Architecture: arm64
Ec2Availability
Ec2Imageid: ami-095a68e28e7
Ec2InstanceType: m7g.large
Ec2Instancetype: m7g.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
Ec2Region: us-west-2
ProcEnviron:
TERM=xterm-
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
RebootRequiredPkgs: Error: path contained symlinks.
SourcePackage: linux-signed-
UpgradeStatus: No upgrade log present (probably fresh install)
CVE References
- 2023-52880
- 2023-6270
- 2024-23307
- 2024-24861
- 2024-26593
- 2024-26642
- 2024-26643
- 2024-26694
- 2024-26710
- 2024-26712
- 2024-26733
- 2024-26734
- 2024-26735
- 2024-26736
- 2024-26748
- 2024-26782
- 2024-26789
- 2024-26790
- 2024-26792
- 2024-26798
- 2024-26802
- 2024-26803
- 2024-26809
- 2024-26838
- 2024-26889
- 2024-26890
- 2024-26922
- 2024-26923
- 2024-26924
- 2024-26925
- 2024-26926
description: | updated |
Changed in linux-signed-aws-6.2 (Ubuntu): | |
assignee: | nobody → Philip Cox (philcox) |
Changed in linux (Ubuntu Noble): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Mantic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Noble): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu Mantic): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Fix Committed → Fix Released |
Status changed to 'Confirmed' because the bug affects multiple users.