kernel panic hit by kube-proxy iptables-save/restore caused by aufs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Medium
|
Mauricio Faria de Oliveira | ||
Xenial |
Fix Released
|
Medium
|
Mauricio Faria de Oliveira | ||
Bionic |
Fix Released
|
Medium
|
Mauricio Faria de Oliveira | ||
Eoan |
Won't Fix
|
Medium
|
Mauricio Faria de Oliveira | ||
Focal |
Fix Released
|
Medium
|
Mauricio Faria de Oliveira | ||
Groovy |
Won't Fix
|
Medium
|
Mauricio Faria de Oliveira |
Bug Description
[Impact]
* Systems with aufs mounts are vulnerable to a kernel BUG(),
which can turn into a panic/crash if panic_on_oops is set.
* It is exploitable by unprivileged local users; and also
remote access operations (e.g., web server) potentially.
* This issue has also manifested in Kubernetes deployments
with a kernel panic in iptables-save or iptables-restore
after a few weeks of uptime, without user interaction.
* Usually all Kubernetes worker nodes hit the issue around
the same time.
[Fix]
* The issue is fixed with 2 patches in aufs4-linux.git:
- 515a586eeef3 aufs: do not call i_readcount_inc()
- f10aea57d39d aufs: bugfix, IMA i_readcount
* The first addresses the issue, and the second addresses a
regression in the aufs feature to change RW branches to RO.
* The kernel v5.3 aufs patches had an equivalent fix to the
second patch, which is present in the Focal aufs patchset
(and on ubuntu-
- 1d26f910c53f aufs: for v5.3-rc1, maintain i_readcount
(in aufs5-linux.git)
[Test Case]
* Repeatedly open/close the same file in read-only mode in
aufs (UINT_MAX times, to overflow a signed int back to 0.)
* Alternatively, monitor the underlying filesystems's file
inode.
(should not monotonically increase; rather, return to 0.)
[Regression Potential]
* This changes the core path that aufs opens files, so there
is a risk of regression; however, the fix changes aufs for
how other filesystems work, so this generally is OK to do.
In any case, most regressions would manifest in open() or
close() (where the VFS handles/checks inode.i_readcount.)
* The aufs maintainer has access to an internal test-suite
used to validate aufs changes, used to identify the first
regression (in the branch RW/RO mode change), and then to
validate/publish the patches upstream; should be good now.
* This has also been tested with 'stress-ng --class filesystem'
and with 'xfstests -overlay' (patch to use aufs vs overlayfs)
on Xenial/Bionic/Focal (-proposed vs. -proposed + patches).
No regressions observed in stress-ng/xfstests log or dmesg.
[Other Info]
* Applied on Unstable (branches master and master-5.8)
* Not required on Groovy (still 5.4; should sync from Unstable)
* Required on LTS releases: Bionic and Focal and Xenial.
* Required on other releases: Disco and Eoan (for custom kernels)
[Original Bug Description]
Problem Report:
--------------
An user reported several nodes in their Kubernetes clusters
hit a kernel panic at about the same time, and periodically
(usually 35 days of uptime, and in same order nodes booted.)
The kernel panics message/stack trace are consistent across
nodes, in __fput() by iptables-
Example:
"""
[3016161.866702] kernel BUG at .../include/
[3016161.866704] invalid opcode: 0000 [#1] SMP
...
[3016161.866780] CPU: 40 PID: 33068 Comm: iptables-restor Tainted: P OE 4.4.0-133-generic #159-Ubuntu
...
[3016161.866786] RIP: 0010:[...] [...] __fput+0x223/0x230
...
[3016161.866818] Call Trace:
[3016161.866823] [...] ____fput+0xe/0x10
[3016161.866827] [...] task_work_
[3016161.866831] [...] exit_to_
[3016161.866833] [...] syscall_
[3016161.866839] [...] int_ret_
"""
(uptime: 3016161 seconds / (24*60*60) = 34.90 days)
They have provided a crashdump (privately available) used
for analysis later in this bug report.
Note: the root cause turns out to be independent of K8s,
as explained in the Root Cause section.
Related Report:
--------------
This behavior matches this public bug of another user:
https:/
"""
I have several machines happen kernel panic,and these
machine have same dump trace like below:
KERNEL: /usr/lib/
...
PANIC: "kernel BUG at .../include/
...
COMMAND: "iptables-restor"
...
crash> bt
...
[exception RIP: __fput+541]
...
#8 [ffff880199f33e60] __fput at ffffffff812125ac
#9 [ffff880199f33ea8] ____fput at ffffffff812126ee
#10 [ffff880199f33eb8] task_work_run at ffffffff8109f101
#11 [ffff880199f33ef8] exit_to_
#12 [ffff880199f33f30] syscall_
#13 [ffff880199f33f50] int_ret_
...
The above showed command "iptables-restor" cause the kernel
panic and its pid is 16884,its parent process is kube-proxy.
Sometimes the process of kernel panic is "iptables-save" and
the dump trace are same.
The kernel panic always happens every 26 days(machine uptime)
"""
<< Adding further sections as comments to keep page short. >>
CVE References
tags: | added: sts |
Changed in linux (Ubuntu): | |
status: | New → In Progress |
Changed in linux (Ubuntu Bionic): | |
status: | New → In Progress |
importance: | Undecided → Medium |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
Changed in linux (Ubuntu Eoan): | |
status: | New → Won't Fix |
importance: | Undecided → Medium |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
Changed in linux (Ubuntu Focal): | |
status: | New → In Progress |
importance: | Undecided → Medium |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
Changed in linux (Ubuntu Groovy): | |
status: | In Progress → Won't Fix |
importance: | Undecided → Medium |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
Changed in linux (Ubuntu): | |
importance: | Undecided → Medium |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
description: | updated |
description: | updated |
description: | updated |
Changed in linux (Ubuntu Eoan): | |
status: | Won't Fix → In Progress |
description: | updated |
description: | updated |
description: | updated |
tags: | added: patch |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Fix Released |
Changed in linux (Ubuntu Focal): | |
status: | In Progress → Fix Released |
Changed in linux (Ubuntu Xenial): | |
status: | New → Fix Released |
importance: | Undecided → Medium |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
Changed in linux (Ubuntu Eoan): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu): | |
status: | In Progress → Fix Released |
Security Impact:
---
The root cause of this problem can be easily exploited
by unprivileged users, both local and remote attackers.
It only needs access to an aufs mount point with read
permissions to any file; opening it in read-only mode,
repeatedly.
For that reason, probably sending the patch for this,
even if keeping it low profile and boring on wording,
may reveal enough information to exploit the problem,
and probably needs some care taking and coordination.
Details in 'Exploit / Local' (and Remote) sections.