Recalled NFSv4 files delegations overwhelm server

Bug #1957986 reported by dann frazier
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Medium
dann frazier
Hirsute
Fix Released
Undecided
Unassigned
Impish
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
An NFSv4 client that does a lot of opens/closes can overwhelm and NFSv4 server, causing a significant drop in performance. In my testing, I've seen performance drop from ~700MiB/s down to < 10MiB/s. The same workload using NFSv3 does not have this problem.

[Test Case]
This can be demonstrated using the elbencho benchmark from https://github.com/breuner/elbencho:
 $ elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu

You'll notice the nfsd threads (I stuck w/ the default of 4) start to consume 100% CPU, and the performance of the elbencho benchmark will begin to trickle.

[Fix]
The following fix solves the problem, but there are a number of patches dependencies required before it will apply to focal:

commit 10717f45639f6c1bc27b56405252c3a027406d92 (refs/bisect/bad)
Author: Trond Myklebust <email address hidden>
Date: Mon Jan 27 09:58:19 2020 -0500

    NFSv4: Limit the total number of cached delegations

    Delegations can be expensive to return, and can cause scalability issues
    for the server. Let's therefore try to limit the number of inactive
    delegations we hold.
    Once the number of delegations is above a certain threshold, start
    to return them on close.

    Signed-off-by: Trond Myklebust <email address hidden>
    Signed-off-by: Anna Schumaker <email address hidden>

[What could go wrong]
The fixes are restricted to NFS code, so problems should be limited to NFS users. They could include performance issues, crashes, etc. Because these changes are mostly related to NFS delegations, I use the `nfstest_delegation` test suite from nfstest[*] to try and identify any regressions:

./nfstest_delegation --client 192.168.42.1 --server 192.168.42.2 -e /srv/nfstest --trcdelay 4

Both before and after applying the fixes, I see the same 146 tests pass and 23 failures. The 23 failures are expected because I was using a Linux-based NFSv4 server which does not support all of the delegation modes that the test checks for.

[*] git://git.linux-nfs.org/projects/mora/nfstest.git

dann frazier (dannf)
Changed in linux (Ubuntu Jammy):
status: New → Fix Released
Changed in linux (Ubuntu Impish):
status: New → Fix Released
Changed in linux (Ubuntu Hirsute):
status: New → Fix Released
Changed in linux (Ubuntu Focal):
status: New → In Progress
assignee: nobody → dann frazier (dannf)
Revision history for this message
dann frazier (dannf) wrote :
description: updated
Revision history for this message
dann frazier (dannf) wrote :
Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-102.115 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
dann frazier (dannf) wrote (last edit ):
Download full text (11.2 KiB)

= focal verification =
ubuntu@client:~$ cat /proc/version_signature
Ubuntu 5.4.0-102.115-generic 5.4.174
ubuntu@client:~$ ./elbencho/bin/elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu
OPERATION RESULT TYPE FIRST DONE LAST DONE
========= ================ ========== =========
READ Elapsed ms : 470642 471169
          Files/s : 4245 4244
          Throughput MiB/s : 530 530
          Total files : 1998154 2000000
          Total MiB : 249770 250000
---
ubuntu@client:~$

Regression:
ubuntu@client:~/nfstest$ PYTHONPATH=$(pwd) ./test/nfstest_delegation --client 192.168.42.1 --server 192.168.42.2 -e /srv/nfstest --trcdelay 4
UID is: 1000
Creating: /mnt/t/nfstest_delegation_20220225_223841_f_001
UID is: 1000
Creating: /mnt/t/nfstest_delegation_20220225_223841_f_002

*** Basic READ delegation test
    TEST: Running test 'basic01'
    PASS: Open file for READ should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: READ delegation should be granted
    TIME: 5.113318s

*** Basic WRITE delegation test
    TEST: Running test 'basic02'
    PASS: Open file for WRITE should succeed
    PASS: Write file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for WRITE on same process should succeed
    PASS: Write file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: Open file for WRITE from a different process should succeed
    PASS: Write file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: WRITE delegation should be granted
    TIME: 4.940012s

*** Basic READ delegation test with file stat
    TEST: Running test 'basic03'
    PASS: Stat file to cache file metadata should succeed
    PASS: Open file for READ should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: LOOKUP operation should be sent
    PASS: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_FH
    PASS: OPEN should be sent with the filehandle of the file to be opened
    PASS: READ delegation should be granted
    PASS: OPEN should not be sent for the same file
    ...

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (37.4 KiB)

This bug was fixed in the package linux - 5.4.0-105.119

---------------
linux (5.4.0-105.119) focal; urgency=medium

  * CVE-2022-0847
    - lib/iov_iter: initialize "flags" in new pipe_buffer

  * Broken network on some AWS instances with focal/impish kernels
    (LP: #1961968)
    - SAUCE: Revert "PCI/MSI: Mask MSI-X vectors only on success"

  * [UBUNTU 20.04] kernel: Add support for CPU-MF counter second version 7
    (LP: #1960182)
    - s390/cpumf: Support for CPU Measurement Facility CSVN 7
    - s390/cpumf: Support for CPU Measurement Sampling Facility LS bit

  * Hipersocket page allocation failure on Ubuntu 20.04 based SSC environments
    (LP: #1959529)
    - s390/qeth: use memory reserves to back RX buffers

  * CVE-2022-0516
    - KVM: s390: Return error on SIDA memop on normal guest

  * CVE-2022-0435
    - tipc: improve size validations for received domain records

  * CVE-2022-0492
    - cgroup-v1: Require capabilities to set release_agent

  * Recalled NFSv4 files delegations overwhelm server (LP: #1957986)
    - NFSv4: Fix delegation handling in update_open_stateid()
    - NFSv4: nfs4_callback_getattr() should ignore revoked delegations
    - NFSv4: Delegation recalls should not find revoked delegations
    - NFSv4: fail nfs4_refresh_delegation_stateid() when the delegation was
      revoked
    - NFS: Rename nfs_inode_return_delegation_noreclaim()
    - NFSv4: Don't remove the delegation from the super_list more than once
    - NFSv4: Hold the delegation spinlock when updating the seqid
    - NFSv4: Clear the NFS_DELEGATION_REVOKED flag in
      nfs_update_inplace_delegation()
    - NFSv4: Update the stateid seqid in nfs_revoke_delegation()
    - NFSv4: Revoke the delegation on success in nfs4_delegreturn_done()
    - NFSv4: Ignore requests to return the delegation if it was revoked
    - NFSv4: Don't reclaim delegations that have been returned or revoked
    - NFSv4: nfs4_return_incompatible_delegation() should check delegation
      validity
    - NFSv4: Fix nfs4_inode_make_writeable()
    - NFS: nfs_inode_find_state_and_recover() fix stateid matching
    - NFSv4: Fix races between open and delegreturn
    - NFSv4: Handle NFS4ERR_OLD_STATEID in delegreturn
    - NFSv4: Don't retry the GETATTR on old stateid in nfs4_delegreturn_done()
    - NFSv4: nfs_inode_evict_delegation() should set NFS_DELEGATION_RETURNING
    - NFS: Clear NFS_DELEGATION_RETURN_IF_CLOSED when the delegation is returned
    - NFSv4: Try to return the delegation immediately when marked for return on
      close
    - NFSv4: Add accounting for the number of active delegations held
    - NFSv4: Limit the total number of cached delegations
    - NFSv4: Ensure the delegation is pinned in nfs_do_return_delegation()
    - NFSv4: Ensure the delegation cred is pinned when we call delegreturn

  * Focal update: v5.4.174 upstream stable release (LP: #1960566)
    - HID: uhid: Fix worker destroying device without any protection
    - HID: wacom: Reset expected and received contact counts at the same time
    - HID: wacom: Ignore the confidence flag when a touch is removed
    - HID: wacom: Avoid using stale array indicies to read contact count
    - f2fs: fix to ...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.