NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server

[Frank Burkhardt]

BugLink: https://bugs.launchpad.net/bugs/1828978

[Impact]

There is a bug in NFS v4.1 that causes a large amount of RPC calls between a client and server when a previous RPC call is interrupted. This uses a large amount of bandwidth and can saturate the network.

The symptoms are so:

* On NFS clients:
Attempts to access mounted NFS shares associated with the affected server block indefinitely.

* On the network:
A storm of repeated RPCs between NFS client and server uses a lot of bandwidth. Each RPC is acknoledged by the server with an NFS4ERR_SEQ_MISORDERED error.

* Other NFS clients connected to the same NFS server:
Performance drops dramatically.

This occurs during a "false retry", when a client attempts to make a new RPC call using a slot+sequence number that references an older, cached call. This happens when a user process interrupts an RPC call that is in progress.

[Fix]

This was fixed in 5.1 upstream with the below commit:

commit 3453d5708b33efe76f40eca1c0ed60923094b971
Author: Trond Myklebust <email address hidden>
Date: Wed Jun 20 17:53:34 2018 -0400
Subject: NFSv4.1: Avoid false retries when RPC calls are interrupted

The fix is to pre-emptively increment the sequence number if an RPC call is interrupted, and to address corner cases we interpret the NFS4ERR_SEQ_MISORDERED error as a sign we need to locate an approperiate sequence number between the value we sent, and the last successfully acked SEQUENCE call.

Commit 3453d5708b33efe76f40eca1c0ed60923094b971 is a clean cherry-pick to disco.

[Testcase]

This is difficult to reproduce on test systems, and has instead been verified on a production NFS v4.1 system in a customer environment. This server is heavily trafficked and has a large number of different NFS clients connected to it.

I have built a test kernel that contains the above patch, and also patches for Bug 1842037. It is available here:

https://launchpad.net/~mruffell/+archive/ubuntu/sf241068-test

Note that the above kernel is for bionic HWE, and not explicitly disco.

Discussion about the patch validation can be found at the bottom of Bug 1842037.

On unpatched kernels, expect to see the symptoms mentioned in Impact, and on patched systems, everything working as intended.

[Regression Potential]

The changes are localised to NFS v4.1 only, and other versions of NFS are not affected. If a regression occurs, users can downgrade NFS versions to v4.0 or v3.x until a fix is made.

The changes only impact when connections are interrupted, and under typical blue sky scenarios would not be invoked.

There have been no fixup commits or commits near the requested commit in newer kernels, which points to this commit fixing the issue, and adopted by the community.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-disco' to 'verification-done-disco'. If the problem still exists, change the tag 'verification-needed-disco' to 'verification-failed-disco'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Frank Burkhardt (fbo)]

I installed the patched kernel on 6 machines with several of my useds hitting them hard. The problem can no longer be reproduced.

tag verification-done-disco

[Frank Burkhardt (fbo)]

hi,

----- Ursprüngliche Mail -----
> Von: "Bug 1842037" <email address hidden>
> An: "burk" <email address hidden>
> Gesendet: Freitag, 8. November 2019 00:37:27
> Betreff: [Bug 1842037] Re: SUNRPC: Use after free when GSSD credentials are invalid causes oops

> Hi Frank,
>
> Just giving you an update on the status of these bugs. The kernel team
> has reviewed the patches I submitted, and each set of patches have
> received two acks each, meaning they will be built into the next kernel
> update.

thank you very much. I managed to install the proposed kernel on several
machines and had a lot of users testing them successfully. However, I can't
find a way to add a tag to a bug. Same goes for bug 1828978 which is fixed
in the same kernel.

Best,

Frank

--
Frank Burkhardt <email address hidden>
IT Dept., Max Planck Institute for Human Cognitive
and Brain Sciences, Leipzig, Germany

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.0.0-37.40

---------------
linux (5.0.0-37.40) disco; urgency=medium

  * disco/linux: 5.0.0-37.40 -proposed tracker (LP: #1852253)

  * System hangs at early boot (LP: #1851216)
    - x86/timer: Skip PIT initialization on modern chipsets

  * drm/i915: Add support for another CMP-H PCH (LP: #1848491)
    - drm/i915/cml: Add second PCH ID for CMP

  * Some EFI systems fail to boot in efi_init() when booted via maas
    (LP: #1851810)
    - efi: efi_get_memory_map -- increase map headroom

  * seccomp: fix SECCOMP_USER_NOTIF_FLAG_CONTINUE test (LP: #1849281)
    - SAUCE: seccomp: avoid overflow in implicit constant conversion
    - SAUCE: seccomp: rework define for SECCOMP_USER_NOTIF_FLAG_CONTINUE
    - SAUCE: seccomp: fix SECCOMP_USER_NOTIF_FLAG_CONTINUE test

  * dkms artifacts may expire from the pool (LP: #1850958)
    - [Packaging] dkms -- try launchpad librarian for pool downloads
    - [Packaging] dkms -- dkms-build quieten wget verbiage

  * update ENA driver to version 2.1.0 (LP: #1850175)
    - net: ena: fix swapped parameters when calling
      ena_com_indirect_table_fill_entry
    - net: ena: fix: Free napi resources when ena_up() fails
    - net: ena: fix incorrect test of supported hash function
    - net: ena: fix return value of ena_com_config_llq_info()
    - net: ena: improve latency by disabling adaptive interrupt moderation by
      default
    - net: ena: fix ena_com_fill_hash_function() implementation
    - net: ena: add handling of llq max tx burst size
    - net: ena: ethtool: add extra properties retrieval via get_priv_flags
    - net: ena: replace free_tx/rx_ids union with single free_ids field in
      ena_ring
    - net: ena: arrange ena_probe() function variables in reverse christmas tree
    - net: ena: add newline at the end of pr_err prints
    - net: ena: documentation: update ena.txt
    - net: ena: allow automatic fallback to polling mode
    - net: ena: add support for changing max_header_size in LLQ mode
    - net: ena: optimise calculations for CQ doorbell
    - net: ena: add good checksum counter
    - net: ena: use dev_info_once instead of static variable
    - net: ena: add MAX_QUEUES_EXT get feature admin command
    - net: ena: enable negotiating larger Rx ring size
    - net: ena: make ethtool show correct current and max queue sizes
    - net: ena: allow queue allocation backoff when low on memory
    - net: ena: add ethtool function for changing io queue sizes
    - net: ena: remove inline keyword from functions in *.c
    - net: ena: update driver version from 2.0.3 to 2.1.0
    - net: ena: Fix bug where ring allocation backoff stopped too late
    - Revert "net: ena: ethtool: add extra properties retrieval via
      get_priv_flags"
    - net: ena: don't wake up tx queue when down
    - net: ena: clean up indentation issue

  * Add Intel Comet Lake ethernet support (LP: #1848555)
    - SAUCE: e1000e: Add support for Comet Lake

  * Intel Wireless AC 3168 on Eoan complaints FW error in SYNC CMD
    GEO_TX_POWER_LIMIT (LP: #1846016)
    - iwlwifi: exclude GEO SAR support for 3168

  * tsc marked unstable after entered PC10 on Intel CoffeeLake (LP: #1840239...