CIFS DFS entries not accessible with 5.4.0-71.74-generic

[Hmpf]

When booting 5.4.0-71-generic I can not access DFS entries anymore, with 5.4.0-70-generic this still works. Some details:

fstab entry:

//example.com/public /home/user/remote cifs noauto,vers=default,noserverino,users,_netdev,user=remote_user,domain=example.com 0 0

shell session
~$ mount /home/user/remote
Password for remote_user@//example.com/public:
~$ ls /home/user/remote/dfs-folder
dfs-subfolder dfs-entry
~$ ls /home/user/remote/dfs-folder/dfs-entry
ls: cannot access '/home/user/remote/dfs-folder/dfs-entry': No such file or directory

dmesg excerpt
[ 1219.568778] CIFS: Attempting to mount //DC01.example.com/Public/dfs-folder/dfs-entry
[ 1219.584975] FS-Cache: Duplicate cookie detected
[ 1219.584986] FS-Cache: O-cookie c=000000006e995395 [p=00000000f4725c61 fl=222 nc=0 na=1]
[ 1219.584990] FS-Cache: O-cookie d=00000000a11c1428 n=0000000071ab21ba
[ 1219.584993] FS-Cache: O-key=[6] '5075626c6963'
[ 1219.585001] FS-Cache: N-cookie c=00000000a0608786 [p=00000000f4725c61 fl=2 nc=0 na=1]
[ 1219.585004] FS-Cache: N-cookie d=00000000a11c1428 n=00000000901f6a53
[ 1219.585007] FS-Cache: N-key=[6] '5075626c6963'
[ 1219.725834] FS-Cache: Duplicate cookie detected
[ 1219.725847] FS-Cache: O-cookie c=000000006e995395 [p=00000000f4725c61 fl=222 nc=0 na=1]
[ 1219.725851] FS-Cache: O-cookie d=00000000a11c1428 n=0000000071ab21ba
[ 1219.725854] FS-Cache: O-key=[6] '5075626c6963'
[ 1219.725864] FS-Cache: N-cookie c=000000000509709f [p=00000000f4725c61 fl=2 nc=0 na=1]
[ 1219.725868] FS-Cache: N-cookie d=00000000a11c1428 n=00000000556f158b
[ 1219.725870] FS-Cache: N-key=[6] '5075626c6963'
[ 1220.216524] CIFS VFS: cifs_read_super: get root inode failed

The dmesg from 5.4.0-70-generic does not have the line
CIFS VFS: cifs_read_super: get root inode failed

~$ lsb_release -rd
Description: KDE neon User Edition 5.21
Release: ~$ lsb_release -rd
Description: KDE neon User Edition 5.21
Release: 20.04
     20.04
~$ apt-cache policy linux-image-generic
linux-image-generic:
  Installed: 5.4.0.71.74
  Candidate: 5.4.0.71.74
  Version table:
 *** 5.4.0.71.74 500
        500 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     5.4.0.26.32 500
        500 http://de.archive.ubuntu.com/ubuntu focal/main amd64 Packages
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC2: dirk 2946 F.... pulseaudio
 /dev/snd/controlC1: dirk 2946 F.... pulseaudio
 /dev/snd/controlC0: dirk 2946 F.... pulseaudio
CasperMD5CheckResult: skip
CurrentDesktop: KDE
DistroRelease: KDE neon 20.04
InstallationDate: Installed on 2017-07-27 (1356 days ago)
InstallationMedia: neon userlts "Xenial" - Build amd64 LIVE Binary 20170614-00:52
MachineType: LENOVO 20HMS00T00
Package: linux (not installed)
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-71-generic root=/dev/mapper/vg00-lv_root ro intel_iommu=off
ProcVersionSignature: Ubuntu 5.4.0-71.79-generic 5.4.101
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-71-generic N/A
 linux-backports-modules-5.4.0-71-generic N/A
 linux-firmware 1.187.10
Tags: focal
Uname: Linux 5.4.0-71-generic x86_64
UnreportableReason: In diesem Bericht geht es um ein Paket, welches nicht installiert ist.
UpgradeStatus: Upgraded to focal on 2020-11-22 (142 days ago)
UserGroups: adm cdrom davfs2 dialout dip kvm libvirt lpadmin plugdev roccat sambashare scanner sudo vboxusers
_MarkForUpload: False
dmi.bios.date: 04/20/2020
dmi.bios.vendor: LENOVO
dmi.bios.version: R0IET61W (1.39 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20HMS00T00
dmi.board.vendor: LENOVO
dmi.board.version: Not Defined
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.modalias: dmi:bvnLENOVO:bvrR0IET61W(1.39):bd04/20/2020:svnLENOVO:pn20HMS00T00:pvrThinkPadX270:rvnLENOVO:rn20HMS00T00:rvrNotDefined:cvnLENOVO:ct10:cvrNone:
dmi.product.family: ThinkPad X270
dmi.product.name: 20HMS00T00
dmi.product.sku: LENOVO_MT_20HM_BU_Think_FM_ThinkPad X270
dmi.product.version: ThinkPad X270
dmi.sys.vendor: LENOVO

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1923670

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Hmpf (dns-hmpf)]

apport information

[Matthew Ruffell (mruffell)]

Hi @Hmpf,

Thanks for filing a bug report.

I had a look at the changes between 5.4.0-70-generic and 5.4.0-71-generic, and I found three patches for CIFS:

$ git log --grep "cifs" Ubuntu-5.4.0-70.78..Ubuntu-5.4.0-71.79
https://paste.ubuntu.com/p/YgrZcN8B6j/

I had a read of the patches, and I think that this one is causing problems for you.

commit 21b200d091826a83aafc95d847139b2b0582f6d1
Author: Aurelien Aptel <email address hidden>
Date: Fri Feb 5 15:42:48 2021 +0100
Subject: cifs: report error instead of invalid when revalidating a dentry fails
Link: https://github.com/torvalds/linux/commit/21b200d091826a83aafc95d847139b2b0582f6d1

This was backported to 5.4.0-71-generic through upstream stable in the below patch:

https://paste.ubuntu.com/p/8FWhRNr8KW/

Now, I think it is that commit due to the error message you are seeing:

CIFS VFS: cifs_read_super: get root inode failed

The commit changes how errors are passed back to the VFS. What it did before was just return 0 (invalid) on any error at all, when cifs_revalidate_dentry() failed.

What it does now is send 0 on ENOENT and ESTALE, and the actual error number on any other error. I think you are getting an "other" error, and cifs isn't sure what to do with it.

Now, looking at cifs_read_super(), where the error message came from:

 150 static int
 151 cifs_read_super(struct super_block *sb)
 152 {
 ...
 205 rc = super_setup_bdi(sb);
 206 if (rc)
 207 goto out_no_root;
 ...
 213 inode = cifs_root_iget(sb);
 214
 215 if (IS_ERR(inode)) {
 216 rc = PTR_ERR(inode);
 217 goto out_no_root;
 218 }
 ...
 225 sb->s_root = d_make_root(inode);
 226 if (!sb->s_root) {
 227 rc = -ENOMEM;
 228 goto out_no_root;
 229 }
 ...
 240 out_no_root:
 241 cifs_dbg(VFS, "%s: get root inode failed\n", __func__);
 242 return rc;
 243 }

We see the out_no_root label is really just a catch-all error handler, and doesn't really tell us what happened, or try to handle the error.

Looking at the code added by the commit, it added a new debug string:

737 static int
738 cifs_d_revalidate(struct dentry *direntry, unsigned int flags)
739 {
...
751 rc = cifs_revalidate_dentry(direntry);
752 if (rc) {
753 cifs_dbg(FYI, "cifs_revalidate_dentry failed with rc=%d", rc);

We can use this string to see what the actual error is, but it is turned off by default, which is why it doesn't appear in dmesg.

Would you be able to turn on this extra error message so we can keep helping you to figure out what is happening?

You can turn on extra debugging with instruction from:
https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting

1) sudo -s
2) echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
3) echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
4) echo 7 > /proc/fs/cifs/cifsFYI
5) exit

Then, try your mount command and then try ls on that dentry file.

If you check dmesg, you will get much more logging. What we are interested in will be a line with "cifs_revalidate_dentry failed with rc" and then an error number.

We can then look up what the number is, and try go from there.

The easiest way to ...

[Hmpf (dns-hmpf)]

Here are the dmesg entries the shell session

~$ mount /home/user/remote
Password for remote_user@//example.com/public:
~$ ls /home/user/remote/dfs-folder
dfs-subfolder dfs-entry
~$ ls /home/user/remote/dfs-folder/dfs-entry
ls: cannot access '/home/user/remote/dfs-folder/dfs-entry': No such file or directory
~$ umount /home/user/remote

It has no line "cifs_revalidate_dentry failed with rc". I will try again later today to ensure that I did not mess anything up.

[Hmpf (dns-hmpf)]

After figuring out the correct upstream repo for the ubuntu kernel repo (not https://git.launchpad.net/ubuntu/+source/linux but https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal) I replicated your command

$ git log --grep 'cifs' Ubuntu-5.4.0-70.78..Ubuntu-5.4.0-71.79

and the most recent commit (f28b34becc655b6139216a72d12378001faa5784)
caught my attention. Indeed it was reverted upstream 2 days ago [1]. I think those are the related discussions on linux-kernel [2] and debian-user [3]. The last one describes the same scenario.

1: https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.4.112

2: https://www.mail-archive.com/search?<email address hidden>&q=subject:%22Re%5C%3A+%5C%5BEXTERNAL%5C%5D+Re%5C%3A+%5C%5BPATCH+4.19+013%5C%2F247%5C%5D+cifs%5C%3A+Set+CIFS_MOUNT_USE_PREFIX_PATH+flag+on+setting+cifs_sb%5C-%3Eprepath.%22&o=newest

3: https://lists.debian.org/debian-user/2021/04/msg00062.html

[Tim Gardner (timg-tpi)]

Hmpf - Here is a test kernel with just commit ea42fd91d3041b81d704624b87f35b56f3a00841 from v5.4.112 applied.

https://kernel.ubuntu.com/~rtg/LP1923670/

[Hmpf (dns-hmpf)]

Yes, I can access the DFS entries with the provided test kernel. Please let me know if you need more info from my side.

I do not think that it matters here, but I did not install the header files as linux-headers-5.4.0-72-generic depends on linux-headers-5.4.0-72 which is not in the linked web directory.

Cheers and thanks for the fast work. :)

[Matthew Ruffell (mruffell)]

Thanks Tim for building the test kernel, and Hmpf, thanks for testing it and confirming it fixes the problem.

Okay, so the problem is:

commit a738c93fb1c17e386a09304b517b1c6b2a6a5a8b
Author: Shyam Prasad N <email address hidden>
Date: Thu Feb 11 03:26:54 2021 -0800
Subject: cifs: Set CIFS_MOUNT_USE_PREFIX_PATH flag on setting cifs_sb->prepath.
Link: https://github.com/torvalds/linux/commit/a738c93fb1c17e386a09304b517b1c6b2a6a5a8b

Looking at Focal, it seems Tim has submitted the revert and it has been applied for the next kernel update:

https://lists.ubuntu.com/archives/kernel-team/2021-April/119241.html
https://lists.ubuntu.com/archives/kernel-team/2021-April/119242.html
https://lists.ubuntu.com/archives/kernel-team/2021-April/119243.html
https://lists.ubuntu.com/archives/kernel-team/2021-April/119244.html
https://lists.ubuntu.com/archives/kernel-team/2021-April/119246.html

Thanks Tim!

As for Bionic, it seems to still be applied to master-next, and hasn't been tagged into a release yet:

https://paste.ubuntu.com/p/k9SffBrBMb/

Groovy seems to be in the same situation:

https://paste.ubuntu.com/p/wRX79WKpy3/

Tim, are there any plans to get the revert landed in Bionic and Groovy for this cycle? I really don't want cause any regressions if we can avoid it.

[Tim Gardner (timg-tpi)]

That is my goal.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Hmpf (dns-hmpf)]

DFS entries are accessible with kernel 5.10.0-1023-oem from proposed (package: linux-image-5.10.0-1023-oem, version: 5.10.0-1023.24).

I could not find the docu for managing tags, so lets see if this works

** Tags added: verification-done-focal

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem-5.10 - 5.10.0-1023.24

---------------
linux-oem-5.10 (5.10.0-1023.24) focal; urgency=medium

  * focal/linux-oem-5.10: 5.10.0-1023.24 -proposed tracker
    (LP: #1924513)

  * Can't adjust brightness on HP ZBook Fury 17 G7 (LP: #1923000)
    - SAUCE: drm/i915/dp add a quirk for backlight issue

  * CIFS DFS entries not accessible with 5.4.0-71.74-generic (LP: #1923670)
    - Revert "cifs: Set CIFS_MOUNT_USE_PREFIX_PATH flag on setting
      cifs_sb->prepath."

 -- Stefan Bader <email address hidden> Thu, 15 Apr 2021 15:12:17 +0200

[Kjell Braden (afflux)]

Will this be fixed for non-OEM kernels in focal? It's listed as Fix Released, but I don't see a fixed version.

[Tim Gardner (timg-tpi)]

It will be released in Ubuntu-5.4.0-73.82 which is proposed.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matthew Ruffell (mruffell)]

Just adding a note for Bionic users, that the commit "cifs: Set CIFS_MOUNT_USE_PREFIX_PATH flag on setting cifs_sb->prepath." landed in 4.15.0-143-generic, causing the regression, and it has been reverted in 4.15.0-144-generic.

If you are experiencing any issues with 4.15.0-143-generic, please upgrade to the fixed 4.15.0-144-generic.