nfsd hangs and never recovers after NFS4ERR_DELAY and a connection loss

[Matthew Ruffell]

BugLink: https://bugs.launchpad.net/bugs/2103564

[Impact]

nfsd loops forever in nfsd4_cb_sequence_done() after it receives a NFS4ERR_DELAY
and the connection is subsequently lost.

What happens is that NFS4ERR_DELAY sets cb->cb_seq_status to -10008, but it is
never set back to 1, so it just keeps sending NFS4ERR_DELAY.

The stack trace looks like:

watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [kworker/u120:29:1520679]
 Kernel panic - not syncing: softlockup: hung tasks
CPU: 33 PID: 1520679 Comm: kworker/u120:29 Tainted: G L 5.15.0-1069-gke #75-Ubuntu
Workqueue: rpciod rpc_async_schedule [sunrpc]
Call Trace:
  RIP: 0010:__rpc_sleep_on_priority_timeout+0x7b/0x110 [sunrpc]
  Code: 0f b6 f9 66 90 44 89 fa 48 89 de 4d 8d 7e 50 4c 89 f7 e8 c8 fb ff ff 4c 89 6b 28 49 8b 46 50 49 39 c7 74 5a 4d 3b 6e 60 78 54 <49> 8b 56 50 48 8d 43 60 48 89 42 08 48 89 53 60 4c 89 7b 68 49 89
...
  rpc_sleep_on_timeout+0x56/0xa0 [sunrpc]
  rpc_delay+0x29/0x30 [sunrpc]
  nfsd4_cb_sequence_done+0x1b9/0x250 [nfsd]
  nfsd4_cb_done+0x1d/0xf0 [nfsd]
  pc_exit_task+0x5c/0x110 [sunrpc]
  ? __rpc_sleep_on_priority+0x80/0x80 [sunrpc]
  __rpc_execute+0x68/0x270 [sunrpc]
  rpc_async_schedule+0x30/0x50 [sunrpc]
  process_one_work+0x22b/0x3d0
  worker_thread+0x53/0x420
  ? process_one_work+0x3d0/0x3d0
  kthread+0x12a/0x150
  ? set_kthread_struct+0x50/0x50
  ret_from_fork+0x22/0x30
  </TASK>

There is no workaround.

[Fix]

This was fixed in 6.9-rc1 by:

commit 961b4b5e86bf56a2e4b567f81682defa5cba957e
From: Chuck Lever <email address hidden>
Date: Fri, 26 Jan 2024 12:45:17 -0500
Subject: NFSD: Reset cb_seq_status after NFS4ERR_DELAY
Link: https://github.com/torvalds/linux/commit/961b4b5e86bf56a2e4b567f81682defa5cba957e

This is present in 5.15.179 and 6.6.76 upstream stable.

[Testcase]

There is no known synthetic reproducer available.

Currently we see it in production workloads on Google Kubernetes Engine, and
we have successfully deployed and ran a test kernel in production with no
further incidents occurring. Before it would lock up once a day.

The test kernel is available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf407307-test

If you install the kernel from the ppa, the issue no longer occurs.

[Where problems can occur]

We are resetting the value of cb->cb_seq_status back to 1 to let it get out of
its state machine, and to actually make some progress, instead of being
trapped at NFS4ERR_DELAY.

If a regression were to occur, it would affect NFS v4.x systems, and it wouldn't
likely cause any real issues, likely some flapping between NFS4ERR_DELAY and
sending callbacks.

[Matthew Ruffell (mruffell)]

Patches submitted to the Kernel Team Mailing List

Cover Letter:
https://lists.ubuntu.com/archives/kernel-team/2025-March/158128.html
Patch:
https://lists.ubuntu.com/archives/kernel-team/2025-March/158129.html

[Tim Whisonant (tswhison)]

This is also being applied early to jammy:linux-gke at Google's request.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gke/5.15.0-1080.86 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-gke' to 'verification-done-jammy-linux-gke'. If the problem still exists, change the tag 'verification-needed-jammy-linux-gke' to 'verification-failed-jammy-linux-gke'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.8.0-60.63 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.15.0-140.150 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux' to 'verification-done-jammy-linux'. If the problem still exists, change the tag 'verification-needed-jammy-linux' to 'verification-failed-jammy-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Matthew Ruffell (mruffell)]

This is currently in -proposed2 for 5.15.0-1080-gke.

[Matthew Ruffell (mruffell)]

Performing verification for Jammy and Noble.

Since there is no known synthetic reproducer for this, and we have had success with the test kernel running in production, I just ran NFStest (https://wiki.linux-nfs.org/wiki/index.php/NFStest) on both jammy with 5.15.0-138-generic and then 5.15.0-140-generic and noted no differences, and then redid the same with 6.8.0-58-generic and then 6.8.0-60-generic, also with no differences.

This should be good to go. I will keep a keen out for regressions, but the test kernel has been running in production for a few months now, and should be okay. Marking verified for Jammy and Noble.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-gke - 5.15.0-1080.86

---------------
linux-gke (5.15.0-1080.86) jammy; urgency=medium

  * jammy/linux-gke: 5.15.0-1080.86 -proposed tracker (LP: #2107001)

  * nfsd hangs and never recovers after NFS4ERR_DELAY and a connection loss
    (LP: #2103564)
    - NFSD: Reset cb_seq_status after NFS4ERR_DELAY

  [ Ubuntu: 5.15.0-139.149 ]

  * jammy/linux: 5.15.0-139.149 -proposed tracker (LP: #2107038)
  * Packaging resync (LP: #1786013)
    - [Packaging] update annotations scripts
  * CVE-2023-52664
    - net: atlantic: eliminate double free in error handling logic
  * CVE-2023-52927
    - netfilter: allow exp not to be removed in nf_ct_find_expectation

 -- Benjamin Wheeler <email address hidden> Wed, 16 Apr 2025 15:12:27 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.8.0-60.63

---------------
linux (6.8.0-60.63) noble; urgency=medium

  * noble/linux: 6.8.0-60.63 -proposed tracker (LP: #2107138)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.04.14)

  * Missing upstream commits for LP: #2102181 (LP: #2107336)
    - libperf cpumap: Add any, empty and min helpers
    - libperf cpumap: Ensure empty cpumap is NULL from alloc

  * Noble update: upstream stable patchset 2025-04-10 (LP: #2106770)
    - memblock: use numa_valid_node() helper to check for invalid node ID
    - jbd2: increase IO priority for writing revoke records
    - jbd2: flush filesystem device before updating tail sequence
    - dm array: fix unreleased btree blocks on closing a faulty array cursor
    - dm array: fix cursor index when skipping across block boundaries
    - exfat: fix the infinite loop in __exfat_free_cluster()
    - erofs: fix PSI memstall accounting
    - ASoC: rt722: add delay time to wait for the calibration procedure
    - ASoC: mediatek: disable buffer pre-allocation
    - selftests/alsa: Fix circular dependency involving global-timer
    - ieee802154: ca8210: Add missing check for kfifo_alloc() in ca8210_probe()
    - net: 802: LLC+SNAP OID:PID lookup on start of skb data
    - tcp/dccp: complete lockless accesses to sk->sk_max_ack_backlog
    - tcp/dccp: allow a connection when sk_max_ack_backlog is zero
    - net: libwx: fix firmware mailbox abnormal return
    - pds_core: limit loop over fw name list
    - bnxt_en: Fix possible memory leak when hwrm_req_replace fails
    - cxgb4: Avoid removal of uninserted tid
    - ice: fix incorrect PHY settings for 100 GB/s
    - igc: return early when failing to read EECD register
    - tls: Fix tls_sw_sendmsg error handling
    - eth: gve: use appropriate helper to set xdp_features
    - Bluetooth: hci_sync: Fix not setting Random Address when required
    - Bluetooth: MGMT: Fix Add Device to responding before completing
    - Bluetooth: btnxpuart: Fix driver sending truncated data
    - tcp: Annotate data-race around sk->sk_mark in tcp_v4_send_reset
    - riscv: Fix early ftrace nop patching
    - memblock tests: fix implicit declaration of function 'numa_valid_node'
    - iio: imu: inv_icm42600: fix timestamps after suspend if sensor is on
    - netfilter: nf_tables: imbalance in flowtable binding
    - drm/mediatek: stop selecting foreign drivers
    - [Config] updateconfigs for MTK_SMI
    - drm/mediatek: Fix YCbCr422 color format issue for DP
    - drm/mediatek: Fix mode valid issue for dp
    - drm/mediatek: Add return value check when reading DPCD
    - cpuidle: riscv-sbi: fix device node release in early exit of
      for_each_possible_cpu
    - scsi: ufs: qcom: Power off the PHY if it was already powered on in
      ufs_qcom_power_up_sequence()
    - dm-ebs: don't set the flag DM_TARGET_PASSES_INTEGRITY
    - ksmbd: Implement new SMB3 POSIX type
    - thermal: of: fix OF node leak in of_thermal_zone_find()
    - smb: client: sync the root session and superblock context passwords before
      automounting
    - ACPI: resource: Add TongFang GM...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.15.0-140.150

---------------
linux (5.15.0-140.150) jammy; urgency=medium

  * jammy/linux: 5.15.0-140.150 -proposed tracker (LP: #2106996)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.04.14)

  * NFS, overlay, fstab issue after update to kernel 5.15.0-133-generic and -134
    (LP: #2103598)
    - udf: Fix directory iteration for longer tail extents

  * Remove floppy kernel module causes null pointer deference (LP: #2104326)
    - floppy: fix add_disk() assumption on exit due to new developments

  * CVE-2025-21971
    - net_sched: Prevent creation of classes with TC_H_ROOT

  * CVE-2024-56599
    - wifi: ath10k: avoid NULL pointer error during sdio remove

  * CVE-2024-56721
    - x86/CPU/AMD: Terminate the erratum_1386_microcode array

  * Jammy update: v5.15.179 upstream stable release (LP: #2106026)
    - afs: Fix EEXIST error returned from afs_rmdir() to be ENOTEMPTY
    - afs: Fix directory format encoding struct
    - hung_task: move hung_task sysctl interface to hung_task.c
    - sysctl: use const for typically used max/min proc sysctls
    - sysctl: share unsigned long const values
    - fs: move inode sysctls to its own file
    - fs: move fs stat sysctls to file_table.c
    - fs: fix proc_handler for sysctl_nr_open
    - block: deprecate autoloading based on dev_t
    - block: retry call probe after request_module in blk_request_module
    - pstore/blk: trivial typo fixes
    - nvme: Add error check for xa_store in nvme_get_effects_log
    - partitions: ldm: remove the initial kernel-doc notation
    - select: Fix unbalanced user_access_end()
    - afs: Fix the fallback handling for the YFS.RemoveFile2 RPC call
    - sched/psi: Use task->psi_flags to clear in CPU migration
    - sched/fair: Fix value reported by hot tasks pulled in /proc/schedstat
    - drm/etnaviv: Fix page property being used for non writecombine buffers
    - genirq: Make handle_enforce_irqctx() unconditionally available
    - wifi: rtlwifi: do not complete firmware loading needlessly
    - wifi: rtlwifi: rtl8192se: rise completion of firmware loading as last step
    - wifi: rtlwifi: wait for firmware loading before releasing memory
    - wifi: rtlwifi: fix init_sw_vars leak when probe fails
    - wifi: rtlwifi: usb: fix workqueue leak when probe fails
    - spi: zynq-qspi: Add check for clk_enable()
    - dt-bindings: mmc: controller: clarify the address-cells description
    - spi: dt-bindings: add schema listing peripheral-specific properties
    - dt-bindings: Another pass removing cases of 'allOf' containing a '$ref'
    - dt-bindings: leds: Add Qualcomm Light Pulse Generator binding
    - dt-bindings: leds: Optional multi-led unit address
    - dt-bindings: leds: Add multicolor PWM LED bindings
    - dt-bindings: leds: class-multicolor: reference class directly in multi-led
      node
    - dt-bindings: leds: class-multicolor: Fix path to color definitions
    - rtlwifi: replace usage of found with dedicated list iterator variable
    - wifi: rtlwifi: remove unused timer and related code
    - wifi: rtlwifi: remove unused dua...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-gke - 6.8.0-1025.29

---------------
linux-gke (6.8.0-1025.29) noble; urgency=medium

  * noble/linux-gke: 6.8.0-1025.29 -proposed tracker (LP: #2107112)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.gke/dkms-versions -- update from kernel-versions
      (main/2025.04.14)

  * Regression in NFS ro mount behavior (LP: #2108862)
    - SAUCE: nfs: clear SB_RDONLY before getting superblock

  * IDPF: TX timeout and crash (LP: #2093622)
    - idpf: make virtchnl2.h self-contained
    - idpf: split &idpf_queue into 4 strictly-typed queue structures
    - idpf: fix memory leaks and crashes while performing a soft reset
    - idpf: enable WB_ON_ITR
    - idpf: fix idpf_vc_core_init error path
    - idpf: add support for SW triggered interrupts
    - idpf: trigger SW interrupt when exiting wb_on_itr mode

  * kdump failures with instances using IDPF (LP: #2104324)
    - idpf: call set_real_num_queues in idpf_open

  [ Ubuntu: 6.8.0-60.63 ]

  * noble/linux: 6.8.0-60.63 -proposed tracker (LP: #2107138)
  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.04.14)
  * Missing upstream commits for LP: #2102181 (LP: #2107336)
    - libperf cpumap: Add any, empty and min helpers
    - libperf cpumap: Ensure empty cpumap is NULL from alloc
  * Noble update: upstream stable patchset 2025-04-10 (LP: #2106770)
    - memblock: use numa_valid_node() helper to check for invalid node ID
    - jbd2: increase IO priority for writing revoke records
    - jbd2: flush filesystem device before updating tail sequence
    - dm array: fix unreleased btree blocks on closing a faulty array cursor
    - dm array: fix cursor index when skipping across block boundaries
    - exfat: fix the infinite loop in __exfat_free_cluster()
    - erofs: fix PSI memstall accounting
    - ASoC: rt722: add delay time to wait for the calibration procedure
    - ASoC: mediatek: disable buffer pre-allocation
    - selftests/alsa: Fix circular dependency involving global-timer
    - ieee802154: ca8210: Add missing check for kfifo_alloc() in ca8210_probe()
    - net: 802: LLC+SNAP OID:PID lookup on start of skb data
    - tcp/dccp: complete lockless accesses to sk->sk_max_ack_backlog
    - tcp/dccp: allow a connection when sk_max_ack_backlog is zero
    - net: libwx: fix firmware mailbox abnormal return
    - pds_core: limit loop over fw name list
    - bnxt_en: Fix possible memory leak when hwrm_req_replace fails
    - cxgb4: Avoid removal of uninserted tid
    - ice: fix incorrect PHY settings for 100 GB/s
    - igc: return early when failing to read EECD register
    - tls: Fix tls_sw_sendmsg error handling
    - eth: gve: use appropriate helper to set xdp_features
    - Bluetooth: hci_sync: Fix not setting Random Address when required
    - Bluetooth: MGMT: Fix Add Device to responding before completing
    - Bluetooth: btnxpuart: Fix driver sending truncated data
    - tcp: Annotate data-race around sk->sk_mark in tcp_v4_send_reset
    - riscv: Fix early ftrace nop patching
    - memblock tests: fix implicit declaration of function 'numa_valid_node'
    - iio: imu...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-5.15/5.15.0-1039.39~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-done-focal-linux-nvidia-tegra-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-failed-focal-linux-nvidia-tegra-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm-gt-tdx/6.8.0-1027.28+tdx1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-ibm-gt-tdx' to 'verification-done-noble-linux-ibm-gt-tdx'. If the problem still exists, change the tag 'verification-needed-noble-linux-ibm-gt-tdx' to 'verification-failed-noble-linux-ibm-gt-tdx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-riscv-6.8/6.8.0-62.65~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-riscv-6.8' to 'verification-done-jammy-linux-riscv-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-riscv-6.8' to 'verification-failed-jammy-linux-riscv-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/6.8.0-1007.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-done-noble-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-failed-noble-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-fips/6.8.0-72.72+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-fips' to 'verification-done-noble-linux-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-fips' to 'verification-failed-noble-linux-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-fips/6.8.0-1035.37+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-done-noble-linux-gcp-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-failed-noble-linux-gcp-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-fips/6.8.0-1034.36+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-done-noble-linux-aws-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-failed-noble-linux-aws-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-xilinx/6.8.0-1017.18 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-done-noble-linux-xilinx'. If the problem still exists, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-failed-noble-linux-xilinx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-fips/6.8.0-1034.39+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-done-noble-linux-azure-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-failed-noble-linux-azure-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!