cifs: DFS Caching feature causing problems traversing multi-tier DFS setups
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Medium
|
Matthew Ruffell | ||
Disco |
Fix Released
|
Medium
|
Matthew Ruffell | ||
Eoan |
Fix Released
|
Medium
|
Matthew Ruffell | ||
Focal |
Fix Released
|
Medium
|
Matthew Ruffell |
Bug Description
BugLink: https:/
[Impact]
There is a problem where kernels 5.0-rc1 and onwards cannot mount a multi tier cifs DFS setup, while kernels 4.20 and below can mount the share fine.
The DFS tiering structure looks like this:
Domain virtual DFS (i.e. \\company.
|-- Domain controller DFS (i.e. \\regional-
|-- Regional DFS Server (i.e. \\regional-
|-- Actual file server (i.e. \\regional-
On the 5.x series kernels, after getting the DFS referrals list through to the Regional DFS Server, which responds with the correct server/share, instead of going to the Actual file server, the kernel backtracks from the Regional DFS Server back to the Domain controller and requests the share there. Of course, this share does not exist on the Domain controller, as it only exists on the Actual file server, and the connection dies.
We have collected a packet capture, and the flow looks like this:
Legend:
-------
DC = Domain Controller / Domain DFS Root
RDC = Regional Domain Controller / Domain DFS Root
RDS = Regional DFS Server
AFS = Actual File Server
4.18.0-21-generic Ubuntu kernel - Good
Host: request/response
-------
DC: company.com\folders
DC: Referral List
RDC: start convo
RDC: <Regional Domain Controller>
RDC: <Regional Domain Controller>
RDS: start convo
RDS: <Regional DFS Server>
RDS: STATUS_
RDS: request referrals
RDS: Referral List
AFS: convo started
AFS: <Actual File Server>\<Share>
AFS: Good response
5.0.0-26-generic Ubuntu kernel - Bad
Host: request/response
-------
DC: company.com\folders
RDC: start convo
RDC: <Regional Domain Controller>
RDC: STATUS_
RDS: start convo
RDS: <Regional DFS Server>
RDS: STATUS_
RDC: <Regional DFS Server>
RDC: STATUS_
From there the debugging output was more or less the same between the two kernel versions, until the problematic area:
Linux 4.18:
Full log: https:/
Status code returned 0xc0000257 STATUS_
fs/cifs/
fs/cifs/connect.c: build_unc_
fs/cifs/smb2ops.c: smb2_get_dfs_refer path <\<Regional DFS Server>
fs/cifs/misc.c: num_referrals: 1 dfs flags: 0x2 ...
fs/cifs/
fs/cifs/connect.c: Username: XXX
// mounts the share successfully
Linux 5.0:
Full log: https:/
Status code returned 0xc0000257 STATUS_
fs/cifs/
fs/cifs/connect.c: build_unc_
fs/cifs/connect.c: build_unc_
fs/cifs/
fs/cifs/
fs/cifs/
fs/cifs/smb2ops.c: smb2_get_dfs_refer path <\<Regional DFS Server>
fs/cifs/smb2pdu.c: SMB2 IOCTL
Status code returned 0xc0000225 STATUS_NOT_FOUND
fs/cifs/
// mounting the share fails shortly after
This has quite a big impact to customers who need to mount their multi-tier DFS mounts, as they have to remain on the 4.15 bionic kernel and cannot use the HWE kernel for their machines.
[Fix]
After some debugging, I narrowed the cause down to a new DFS caching feature introduced in 5.0-rc1. I started a discussion with the upstream maintainer of cifs, which you can read here:
https://<email address hidden>/T/#u
This discussion resulted in the below upstream commit, which was merged in the 5.5 development window:
commit 5bb30a4dd60e2a1
Author: Paulo Alcantara (SUSE) <email address hidden>
Date: Fri Nov 22 12:30:56 2019 -0300
Subject: cifs: Fix retrieval of DFS referrals in cifs_mount()
You can read it here:
https:/
This commit sets referrals to be passed to the newest resolved root server, instead of older ones up the order. This ensures that we keep descending down the tree instead of backtracking, which what was happening.
This commit has been submitted for upstream -stable, and is still being processed. The commit is needed on kernels 5.0 and up. I will update this section if it is accepted for -stable.
[Testcase]
To test this commit you need a multi-tier cifs DFS with a similar structure as the tree mentioned in the Impact section. From there, you simply try and mount a cifs share.
On patched kernels, the mount will succeed. On broken kernels, the mount will fail.
I have prepared a test kernel for Bionic HWE, based on 5.0.0-37.40~18.04 which you can find here:
https:/
This test kernel has been tested by the customer and mounts the cifs DFS correctly.
[Regression Potential]
I believe the risk of regression for this commit is low. All changes are limited to DFS within cifs, and only change the behaviour of what server is the root server referrals are sent to.
The commit is a clean cherry pick for disco, eoan and focal. The maintainer has submitted the commit for upstream -stable, and we have tested the commit with the customer, and things are now working as intended.
CVE References
Changed in linux (Ubuntu Disco): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Eoan): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Focal): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Disco): | |
status: | New → In Progress |
Changed in linux (Ubuntu Eoan): | |
status: | New → In Progress |
Changed in linux (Ubuntu Focal): | |
status: | New → In Progress |
Changed in linux (Ubuntu Disco): | |
assignee: | nobody → Matthew Ruffell (mruffell) |
Changed in linux (Ubuntu Eoan): | |
assignee: | nobody → Matthew Ruffell (mruffell) |
Changed in linux (Ubuntu Focal): | |
assignee: | nobody → Matthew Ruffell (mruffell) |
tags: | added: sts |
description: | updated |
description: | updated |
Changed in linux (Ubuntu Eoan): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Disco): | |
status: | In Progress → Fix Committed |
This bug was fixed in the package linux - 5.4.0-9.12
---------------
linux (5.4.0-9.12) focal; urgency=medium
* alsa/hda/realtek: the line-out jack doens't work on a dell AIO
(LP: #1855999)
- SAUCE: ALSA: hda/realtek - Line-out jack doesn't work on a Dell AIO
* scsi: hisi_sas: Check sas_port before using it (LP: #1855952)
- scsi: hisi_sas: Check sas_port before using it
* CVE-2019-19078
- ath10k: fix memory leak
* cifs: DFS Caching feature causing problems traversing multi-tier DFS setups
(LP: #1854887)
- cifs: Fix retrieval of DFS referrals in cifs_mount()
* Support DPCD aux brightness control (LP: #1856134) aux_enable_ backlight( )
- SAUCE: drm/i915: Fix eDP DPCD aux max backlight calculations
- SAUCE: drm/i915: Assume 100% brightness when not in DPCD control mode
- SAUCE: drm/i915: Fix DPCD register order in intel_dp_
- SAUCE: drm/i915: Auto detect DPCD backlight support by default
- SAUCE: drm/i915: Force DPCD backlight mode on X1 Extreme 2nd Gen 4K AMOLED
panel
- USUNTU: SAUCE: drm/i915: Force DPCD backlight mode on Dell Precision 4K sku
* The system cannot resume from S3 if user unplugs the TB16 during suspend
state (LP: #1849269)
- PCI: pciehp: Do not disable interrupt twice on suspend
- PCI: pciehp: Prevent deadlock on disconnect
* change kconfig of the soundwire bus driver from y to m (LP: #1855685)
- [Config]: SOUNDWIRE=m
* alsa/sof: change to use hda hdmi codec driver to make hdmi audio on the dsp-generic: use snd-hda-codec-hdmi dsp-generic: fix include guard name max98357a: common hdmi codec support max98357a: common hdmi codec support SOF_HDA_ COMMON_ HDMI_CODEC= y
docking station work (LP: #1855666)
- ALSA: hda/hdmi - implement mst_no_extra_pcms flag
- ASoC: hdac_hda: add support for HDMI/DP as a HDA codec
- ASoC: Intel: skl-hda-
- ASoC: Intel: skl-hda-
- ASoC: SOF: Intel: add support for snd-hda-codec-hdmi
- ASoC: Intel: bxt-da7219-
- ASoC: Intel: glk_rt5682_
- ASoC: intel: sof_rt5682: common hdmi codec support
- ASoC: Intel: bxt_rt298: common hdmi codec support
- ASoC: SOF: enable sync_write in hdac_bus
- [config]: SND_SOC_
* Fix unusable USB hub on Dell TB16 after S3 (LP: #1855312)
- SAUCE: USB: core: Make port power cycle a seperate helper function
- SAUCE: USB: core: Attempt power cycle port when it's in eSS.Disabled state
* Focal update: v5.4.3 upstream stable release (LP: #1856583) sql-viewer. py: Fix use of TRUE with SQLite get_irq_ optional( ) for optional irq
- rsi: release skb if rsi_prepare_beacon fails
- arm64: tegra: Fix 'active-low' warning for Jetson TX1 regulator
- arm64: tegra: Fix 'active-low' warning for Jetson Xavier regulator
- perf scripts python: exported-
- sparc64: implement ioremap_uc
- lp: fix sparc64 LPSETTIMEOUT ioctl
- time: Zero the upper 32-bits in __kernel_timespec on 32-bit
- mailbox: tegra: Fix superfluous IRQ error message
- staging/octeon: Use stubs for MIPS && !CAVIUM_OCTEON_SOC
- usb: gadget: u_serial: add missing port entry locking
- serial: 8250-mtk: Use platform_
- tty: serial: fsl_lpuart: use the sg ...