A deadlock exists in the XFS filesystem that occurs when the XFS log buffer becomes completely exhausted.
XFS maintains a circular ring buffer for the log, and it is a fixed size. To be able to create a transaction, you need to be able to reserve space on the log buffer.
Certain ioend transactions, such as file append, can be preallocated for a negligible performance gain. This takes up space in the log buffer, and these preallocated ioends are placed on a workqueue, behind other ioends that are not preallocated.
The deadlock occurs when the XFS log buffer is running low on space, and an ioend append transaction comes in. It is preallocated, consuming nearly all of the remaining XFS log buffer space, and is placed at the very end of the ioend workqueue. The kernel then takes a ioend from the top of the ioend workqeueue, creates a transaction, xfs_trans_alloc(), attempts to allocate space for it, xfs_trans_reserve(), xfs_log_reserve(), and then waits in a while loop checking free space in the log, xlog_grant_head_check(), xlog_grant_head_wait().
Since there is no space, the thread sleeps with schedule(). This happens with all ioend transactions, since the log is exhausted and I/O is not moving.
Since I/O never moves, the thread is never woken up, and we get hung task timeouts, with system failure shortly afterward.
An example hung task timeout is:
INFO: task kworker/60:0:4002982 blocked for more than 360 seconds.
Tainted: G OE 5.4.0-137-generic #154-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/60:0 D 0 4002982 2 0x90004080
Workqueue: xfs-conv/dm-3 xfs_end_io [xfs]
Call Trace:
__schedule+0x2e3/0x740
schedule+0x42/0xb0
xlog_grant_head_wait+0xb9/0x1e0 [xfs]
xlog_grant_head_check+0xde/0x100 [xfs]
xfs_log_reserve+0xc9/0x1e0 [xfs]
xfs_trans_reserve+0x17a/0x1e0 [xfs]
xfs_trans_alloc+0xda/0x170 [xfs]
xfs_iomap_write_unwritten+0x128/0x2f0 [xfs]
xfs_end_ioend+0x15b/0x1b0 [xfs]
xfs_end_io+0xb1/0xe0 [xfs]
process_one_work+0x1eb/0x3b0
worker_thread+0x4d/0x400
kthread+0x104/0x140
? process_one_work+0x3b0/0x3b0
? kthread_park+0x90/0x90
ret_from_fork+0x1f/0x40
There is no known workaround, other than to have a larger log buffer at filesystem creation, but even then, it only buys you time until you get high enough load to exhaust the log buffer.
[Fix]
This was fixed in 5.13-rc1 by the following patch:
The patch more or less removes all preallocated ioend transactions, and instead, when ioend appends are needed, they go to the standard workqueue like any other ioend, where transactions are allocated when they reach the top of the workqueue.
The patch required some backporting for Focal. The changes to the patch itself is minimal and should be straightforward to read, however, the changes to the XFS ioend subsystem between 5.4 and 5.13-rc1 were quite extreme, with a lot of refactoring taking place over very many commits.
Additionally, the patch was part of a five part series, the first, fixes the deadlock, and the rest remove all the code to do with transaction preallocation.
It is safe to leave the rest of the code in place. It will become dead code, but it will not be reachable, and not cause any risk of regression, due to ioend->io_append_trans always being NULL, and not entering into any of the if statements.
[Testcase]
There is currently no known testcase for this issue. The issue has only been seen in a customer production environment, running under heavy load. The issue has not been seen in a customer test environment, only production.
The production workload is a busy Kubernetes cluster running containers and VMs, with the hosts's filesystem being broken into separate mountpoints over several partitions, all XFS.
The kubernetes containers are all backed from a large 4TB disk which is XFS.
This test kernel has been deployed to several production hosts, and the deadlock no longer occurs.
[Where problems can occur]
We are changing how certain ioend transactions take place in the XFS filesystem. If a regression were to occur, it would impact all XFS users. Users would have to downgrade their kernel, as there are no workarounds for enabling or disabling the behaviour change.
The overall risk of the change should be low. ioend append transactions would still be processed in nearly the same way, still being placed at the end of the ioend workqueue like any other transaction, with the only change being it is allocated at the time the transaction is created, at the top of the workqueue when it is processed, and not preallocated when the ioend is first submitted.
There will be a very minor performance penalty, but it wouldn't be measurable in any tangible workload.
I have run xfstests for the xfs/* subset against the released 5.4.0-137-generic and the test 5.4.0-137-generic test kernel with the patch included. The test kernel had identical results, it will likely not cause any regressions.
There is some additional risk leaving the dead code in place, but I have read the code and the commits to remove the dead code, and I came to the conclusion it is less risky to leave it in place, than to backport the refactor commits.
[Other Info]
The XFS ioend subsystem refactor took place in the following commits:
commit 6e552494fb90acae005d74ce6a2ee102d965184b
Author: Brian Foster <email address hidden>
Date: Tue May 4 08:54:29 2021 -0700
Subject: iomap: remove unused private field from ioend
Link: ttps://github.com/torvalds/linux/commit/6e552494fb90acae005d74ce6a2ee102d965184b
As you can see, the four latter commits are not necessary. They simply remove dead code, which has no harm being left in place. They also do not backport at all, not without the ALL of the significant refactor commits.
Hence, we only take "xfs: drop submit side trans alloc for append ioends" only, as it is the only needed commit to solve the problem.
BugLink: https:/ /bugs.launchpad .net/bugs/
[Impact]
A deadlock exists in the XFS filesystem that occurs when the XFS log buffer becomes completely exhausted.
XFS maintains a circular ring buffer for the log, and it is a fixed size. To be able to create a transaction, you need to be able to reserve space on the log buffer.
Certain ioend transactions, such as file append, can be preallocated for a negligible performance gain. This takes up space in the log buffer, and these preallocated ioends are placed on a workqueue, behind other ioends that are not preallocated.
The deadlock occurs when the XFS log buffer is running low on space, and an ioend append transaction comes in. It is preallocated, consuming nearly all of the remaining XFS log buffer space, and is placed at the very end of the ioend workqueue. The kernel then takes a ioend from the top of the ioend workqeueue, creates a transaction, xfs_trans_alloc(), attempts to allocate space for it, xfs_trans_ reserve( ), xfs_log_reserve(), and then waits in a while loop checking free space in the log, xlog_grant_ head_check( ), xlog_grant_ head_wait( ).
Since there is no space, the thread sleeps with schedule(). This happens with all ioend transactions, since the log is exhausted and I/O is not moving.
Since I/O never moves, the thread is never woken up, and we get hung task timeouts, with system failure shortly afterward.
An example hung task timeout is:
INFO: task kworker/ 60:0:4002982 blocked for more than 360 seconds. kernel/ hung_task_ timeout_ secs" disables this message. 0x2e3/0x740 head_wait+ 0xb9/0x1e0 [xfs] head_check+ 0xde/0x100 [xfs] reserve+ 0xc9/0x1e0 [xfs] reserve+ 0x17a/0x1e0 [xfs] alloc+0xda/ 0x170 [xfs] write_unwritten +0x128/ 0x2f0 [xfs] ioend+0x15b/ 0x1b0 [xfs] io+0xb1/ 0xe0 [xfs] one_work+ 0x1eb/0x3b0 thread+ 0x4d/0x400 0x104/0x140 one_work+ 0x3b0/0x3b0 park+0x90/ 0x90 fork+0x1f/ 0x40
Tainted: G OE 5.4.0-137-generic #154-Ubuntu
"echo 0 > /proc/sys/
kworker/60:0 D 0 4002982 2 0x90004080
Workqueue: xfs-conv/dm-3 xfs_end_io [xfs]
Call Trace:
__schedule+
schedule+0x42/0xb0
xlog_grant_
xlog_grant_
xfs_log_
xfs_trans_
xfs_trans_
xfs_iomap_
xfs_end_
xfs_end_
process_
worker_
kthread+
? process_
? kthread_
ret_from_
There is no known workaround, other than to have a larger log buffer at filesystem creation, but even then, it only buys you time until you get high enough load to exhaust the log buffer.
[Fix]
This was fixed in 5.13-rc1 by the following patch:
commit 7cd3099f4925d7c 15887d1940ebd65 acd66100f5 /github. com/torvalds/ linux/commit/ 7cd3099f4925d7c 15887d1940ebd65 acd66100f5
Author: Brian Foster <email address hidden>
Date: Fri Apr 9 10:27:43 2021 -0700
Subject: xfs: drop submit side trans alloc for append ioends
Link: https:/
The patch more or less removes all preallocated ioend transactions, and instead, when ioend appends are needed, they go to the standard workqueue like any other ioend, where transactions are allocated when they reach the top of the workqueue.
The patch required some backporting for Focal. The changes to the patch itself is minimal and should be straightforward to read, however, the changes to the XFS ioend subsystem between 5.4 and 5.13-rc1 were quite extreme, with a lot of refactoring taking place over very many commits.
Additionally, the patch was part of a five part series, the first, fixes the deadlock, and the rest remove all the code to do with transaction preallocation.
It is safe to leave the rest of the code in place. It will become dead code, but it will not be reachable, and not cause any risk of regression, due to ioend-> io_append_ trans always being NULL, and not entering into any of the if statements.
[Testcase]
There is currently no known testcase for this issue. The issue has only been seen in a customer production environment, running under heavy load. The issue has not been seen in a customer test environment, only production.
The production workload is a busy Kubernetes cluster running containers and VMs, with the hosts's filesystem being broken into separate mountpoints over several partitions, all XFS.
The kubernetes containers are all backed from a large 4TB disk which is XFS.
A test kernel is available in the following ppa:
https:/ /launchpad. net/~mruffell/ +archive/ ubuntu/ sf353709- test
This test kernel has been deployed to several production hosts, and the deadlock no longer occurs.
[Where problems can occur]
We are changing how certain ioend transactions take place in the XFS filesystem. If a regression were to occur, it would impact all XFS users. Users would have to downgrade their kernel, as there are no workarounds for enabling or disabling the behaviour change.
The overall risk of the change should be low. ioend append transactions would still be processed in nearly the same way, still being placed at the end of the ioend workqueue like any other transaction, with the only change being it is allocated at the time the transaction is created, at the top of the workqueue when it is processed, and not preallocated when the ioend is first submitted.
There will be a very minor performance penalty, but it wouldn't be measurable in any tangible workload.
I have run xfstests for the xfs/* subset against the released 5.4.0-137-generic and the test 5.4.0-137-generic test kernel with the patch included. The test kernel had identical results, it will likely not cause any regressions.
There is some additional risk leaving the dead code in place, but I have read the code and the commits to remove the dead code, and I came to the conclusion it is less risky to leave it in place, than to backport the refactor commits.
[Other Info]
The XFS ioend subsystem refactor took place in the following commits:
commit 598ecfbaa742aca 0dcdbbea2568140 6f95cc0b63 /github. com/torvalds/ linux/commit/ 598ecfbaa742aca 0dcdbbea2568140 6f95cc0b63
From: Christoph Hellwig <email address hidden>
Date: Thu, 17 Oct 2019 13:12:15 -0700
Subject: iomap: lift the xfs writeback code to iomap
Link: https:/
commit 9e91c5728cab3d0 aa3197d009c3d63 e147914e77 /github. com/torvalds/ linux/commit/ 9e91c5728cab3d0 aa3197d009c3d63 e147914e77
From: Christoph Hellwig <email address hidden>
Date: Thu, 17 Oct 2019 13:12:13 -0700
Subject: iomap: lift common tracing code from xfs to iomap
Link: https:/
commit 433dad94ec5d6b9 0385b56a8bc8718 dd9542b289 /github. com/torvalds/ linux/commit/ 433dad94ec5d6b9 0385b56a8bc8718 dd9542b289
From: Christoph Hellwig <email address hidden>
Date: Thu, 17 Oct 2019 13:12:07 -0700
Subject: xfs: refactor the ioend merging code
Link: https:/
ioend-> io_append_ trans was renamed to ioend->io_private in the following commit:
commit 5653017bc44e54b aa299f3523f160c 23ac0628fd /github. com/torvalds/ linux/commit/ 5653017bc44e54b aa299f3523f160c 23ac0628fd
From: Christoph Hellwig <email address hidden>
Date: Thu, 17 Oct 2019 13:12:09 -0700
Subject: xfs: turn io_append_trans into an io_private void pointer
Link: https:/
The full five part preallocated ioend deadlock patch series is:
commit 7cd3099f4925d7c 15887d1940ebd65 acd66100f5 /github. com/torvalds/ linux/commit/ 7cd3099f4925d7c 15887d1940ebd65 acd66100f5
Author: Brian Foster <email address hidden>
Date: Fri Apr 9 10:27:43 2021 -0700
Subject: xfs: drop submit side trans alloc for append ioends
Link: https:/
commit 7adb8f14e134d5f 885d47c4ccd6208 36235f0b7f /github. com/torvalds/ linux/commit/ 7adb8f14e134d5f 885d47c4ccd6208 36235f0b7f
Author: Brian Foster <email address hidden>
Date: Fri Apr 9 10:27:55 2021 -0700
Subject: xfs: open code ioend needs workqueue helper
Link: https:/
commit 044c6449f18f174 ba8d86640936add 3fc7582e49 github. com/torvalds/ linux/commit/ 044c6449f18f174 ba8d86640936add 3fc7582e49
Author: Brian Foster <email address hidden>
Date: Fri Apr 9 10:27:55 2021 -0700
Subject: xfs: drop unused ioend private merge and setfilesize code
Link: ttps://
commit e7a3d7e792a5ad5 0583a2e6c35e72b d2ca6096f4 github. com/torvalds/ linux/commit/ e7a3d7e792a5ad5 0583a2e6c35e72b d2ca6096f4
Author: Brian Foster <email address hidden>
Date: Fri Apr 9 10:27:56 2021 -0700
Subject: xfs: drop unnecessary setfilesize helper
Link: ttps://
commit 6e552494fb90aca e005d74ce6a2ee1 02d965184b github. com/torvalds/ linux/commit/ 6e552494fb90aca e005d74ce6a2ee1 02d965184b
Author: Brian Foster <email address hidden>
Date: Tue May 4 08:54:29 2021 -0700
Subject: iomap: remove unused private field from ioend
Link: ttps://
As you can see, the four latter commits are not necessary. They simply remove dead code, which has no harm being left in place. They also do not backport at all, not without the ALL of the significant refactor commits.
Hence, we only take "xfs: drop submit side trans alloc for append ioends" only, as it is the only needed commit to solve the problem.