Recalled NFSv4 files delegations overwhelm server
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
dann frazier | ||
Hirsute |
Fix Released
|
Undecided
|
Unassigned | ||
Impish |
Fix Released
|
Undecided
|
Unassigned | ||
Jammy |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Impact]
An NFSv4 client that does a lot of opens/closes can overwhelm and NFSv4 server, causing a significant drop in performance. In my testing, I've seen performance drop from ~700MiB/s down to < 10MiB/s. The same workload using NFSv3 does not have this problem.
[Test Case]
This can be demonstrated using the elbencho benchmark from https:/
$ elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu
You'll notice the nfsd threads (I stuck w/ the default of 4) start to consume 100% CPU, and the performance of the elbencho benchmark will begin to trickle.
[Fix]
The following fix solves the problem, but there are a number of patches dependencies required before it will apply to focal:
commit 10717f45639f6c1
Author: Trond Myklebust <email address hidden>
Date: Mon Jan 27 09:58:19 2020 -0500
NFSv4: Limit the total number of cached delegations
Delegations can be expensive to return, and can cause scalability issues
for the server. Let's therefore try to limit the number of inactive
delegations we hold.
Once the number of delegations is above a certain threshold, start
to return them on close.
Signed-off-by: Trond Myklebust <email address hidden>
Signed-off-by: Anna Schumaker <email address hidden>
[What could go wrong]
The fixes are restricted to NFS code, so problems should be limited to NFS users. They could include performance issues, crashes, etc. Because these changes are mostly related to NFS delegations, I use the `nfstest_
./nfstest_
Both before and after applying the fixes, I see the same 146 tests pass and 23 failures. The 23 failures are expected because I was using a Linux-based NFSv4 server which does not support all of the delegation modes that the test checks for.
[*] git://git.
Changed in linux (Ubuntu Jammy): | |
status: | New → Fix Released |
Changed in linux (Ubuntu Impish): | |
status: | New → Fix Released |
Changed in linux (Ubuntu Hirsute): | |
status: | New → Fix Released |
Changed in linux (Ubuntu Focal): | |
status: | New → In Progress |
assignee: | nobody → dann frazier (dannf) |
Changed in linux (Ubuntu Focal): | |
importance: | Undecided → Medium |
status: | In Progress → Fix Committed |
This bug is awaiting verification that the linux/5.4.0-102.115 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification- needed- focal' to 'verification- done-focal' . If the problem still exists, change the tag 'verification- needed- focal' to 'verification- failed- focal'.
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/ /wiki.ubuntu. com/Testing/ EnableProposed for documentation how to enable and use -proposed. Thank you!