Regression CIFS mounted DFS

[Heiko]

On Ubuntu 22.04.3 LTS (jammy jellyfish) we mount a DFS file service using CIFS with kerberos authentication. After the kernel upgrade from 5.15.0-86-generic to 5.15.0-88-generic (exact version: 5.15.0-88.98) this stopped working. The mount seems to have worked, but files and directories are inaccessible. A non-DFS CIFS share mounted on the same client machine is not affected.

In our case the filesystem is mounted using autofs with the following mapping:
/dfs-share -fstype=cifs,sec=krb5,vers=3,multiuser,noserverino ://example.com/share

Kernel log messages:
CIFS: Attempting to mount \\example.com\share
CIFS: VFS: BAD_NETWORK_NAME: \\example.com\share
CIFS: VFS: Verify user has a krb5 ticket and keyutils is installed
CIFS: VFS: \\CLx-Ny.EXAMPLE.COM Send error in SessSetup = -126

We confirmed that kernel version 5.15.0-88 causes this by booting different kernel versions repeatedly. When booting 5.15.0-86 everything worked properly as before.

Though not familiar with kernel coding, we found changes in the source code that seem to confirm that some code dealing with DFS has changed, moved, or removed.

After:
git clone git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy

This:
git diff Ubuntu-5.15.0-86.96 Ubuntu-5.15.0-88.98 -- fs/cifs/cifs_dfs_ref.c

This suggests the function cifs_dfs_do_mount() has been (re)moved.

[Timo Aaltonen (tjaalton)]

pulled in via upstream stable updates from 5.15.124

commit 015222ee0044b075c205010f3a6add2ed79e1159
Author: Paulo Alcantara <email address hidden>
Date: Tue Oct 4 18:41:20 2022 -0300

    cifs: use fs_context for automounts

    BugLink: https://bugs.launchpad.net/bugs/2035400

I don't see a fix for this in upstream v5.15.x (currently at .138), so I'm not sure if a newer mainline would work, but you could give them a go. Fetch from https://kernel.ubuntu.com/mainline/ and install two debs for a version, linux-image-unsigned and linux-modules (copy locally, then just 'sudo dpkg -i <deb>').

[Matthew Ruffell (mruffell)]

Hi Heiko,

I had a look at the changes between 5.15.0-86-generic and 5.15.0-88-generic, and there are the following cifs commits:

ubuntu-jammy $ git log --grep "cifs" --oneline Ub
untu-5.15.0-86.96..Ubuntu-5.15.0-88.98
c92787866aa8 cifs: if deferred close is disabled then close files immediately
5c684782dfb6 ksmbd: remove internal.h include
015222ee0044 cifs: use fs_context for automounts
0ba530127927 cifs: missing directory in MAINTAINERS file
a6f0ea8bffe0 keys: Fix linking a duplicate key to a keyring's assoc_array

The interesting one is "cifs: use fs_context for automounts" in particular, which you have already noted removes the cifs_dfs_do_mount() function and spreads the logic into cifs_dfs_do_automount().

It seems "cifs: use fs_context for automounts" was introduced in 6.2-rc1 upstream, and was brought into our kernel due to it being a dependency of "cifs: if deferred close is disabled then close files immediately" which was recently included in upstream stable.

I have built you a test kernel based on 5.15.0-88-generic with the following two commits reverted:

commit df9d70c18616760c6504b97fec66b6379c172dbb
Author: Bharath SM <email address hidden>
Date: Fri Jul 7 15:29:01 2023 +0000
Subject: cifs: if deferred close is disabled then close files immediately
Link: https://github.com/torvalds/linux/commit/df9d70c18616760c6504b97fec66b6379c172dbb

commit 9fd29a5bae6e8f94b410374099a6fddb253d2d5f
Author: Paulo Alcantara <email address hidden>
Date: Tue Oct 4 18:41:20 2022 -0300
Subject: cifs: use fs_context for automounts
Link: https://github.com/torvalds/linux/commit/9fd29a5bae6e8f94b410374099a6fddb253d2d5f

Please note this package is NOT SUPPORTED by Canonical, and is for TESTING
PURPOSES ONLY. ONLY Install in a dedicated test environment.

Instructions to install (On a Jammy system):
1) sudo add-apt-repository ppa:mruffell/lp2043118-test
2) sudo apt update
3) sudo apt install linux-image-unsigned-5.15.0-88-generic linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic linux-headers-5.15.0-88-generic
4) sudo reboot
5) uname -rv
5.15.0-88-generic #98+TEST2043118v20231110b1-Ubuntu SMP Fri Nov 10 10:00:22 UTC 2023

If you are asked to abort removal of the currently running kernel, say no.

Can you let me know if the test kernel works? If it does, then we at least know the commits which caused the issue. From there we can decide to revert in the next Ubuntu kernel release, or see if the current upstream is broken and come up with a fix instead.

Could you also try 6.6-rc5, like Timo suggested?

https://kernel.ubuntu.com/mainline/v6.6-rc5/

$ wget https://kernel.ubuntu.com/mainline/v6.6-rc5/amd64/linux-image-unsigned-6.6.0-060600rc5-generic_6.6.0-060600rc5.202310081731_amd64.deb
$ wget https://kernel.ubuntu.com/mainline/v6.6-rc5/amd64/linux-modules-6.6.0-060600rc5-generic_6.6.0-060600rc5.202310081731_amd64.deb
$ sudo dpkg -i ./linux-*
$ sudo reboot

Thanks,
Matthew

[Heiko (h-e-noordhof)]

Hi Matthew and Timo,

I've tried both these kernels in our test environment (uname -rv outputs listed):

1) 5.15.0-88-generic #98+TEST2043118v20231110b1-Ubuntu SMP Fri Nov 10 10:00:22 UTC 20
Works for us, that is: does not have the issue this bug report is about.

2) 6.6.0-060600rc5-generic #202310081731 SMP PREEMPT_DYNAMIC Thu Nov 16 04:38:01 UTC 2023
Still has the failing CIFS-mounted DFS filesystem.

FWIW, running the 6.2.0-36-generic kernel from the jammy repositories, executing "ls -l" to access the (autofs mounted) file system hangs. Whereas doing the same while running the 6.6.0-060600rc5-generic it just fails after about 10 secs.

Best regards,
Heiko

[Heiko (h-e-noordhof)]

FYI:

Our issue could be resolved by means of an additional (auto) mount option for the cifs filesystem:

# /etc/auto.master.d/misc --> added: cruid=${UID}
#
/dfs-share -fstype=cifs,sec=krb5,vers=3,cruid=${UID},multiuser,noserverino ://example.com/share

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.