multiuser mount with sec=krb5: cifs_mount failed w/return code = -2

[Alexander Fieroch]

I want to mount a cifs-share with kerberos and multiuser option. On Ubuntu it fails. Same command and system configuration is working on a RedHat linux. Maybe there's a regression in cifs-utils or another library that differs from RedHat?

Ubuntu 20.04.1
cifs-utils 6.9-1ubuntu0.1

RedHat 7.9
cifs-utils 6.2-10.el7

We have joined our clients to AD with realm --membership-software=adcli and use sssd for authentication.

What I did:
root@kubuntu-lts:# kinit -k KUBUNTU-LTS$
root@kubuntu-lts:# klist
Ticket cache: FILE:/tmp/krb5cc_10011_r0AC1F
Default principal: KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE

Valid starting Expires Service principal
21.10.2020 16:16:20 22.10.2020 02:16:20 <email address hidden>
        renew until 22.10.2020 16:16:20
root@kubuntu-lts:# mount //FILESERVER/SHARE /mnt/test -o sec=krb5,multiuser,file_mode=0660,dir_mode=0770,nounix,noserverino
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

Share can be mounted as a user without multiuser option but not with system UPN and multiuser option. This is working on a system with RedHat 7.9.

I also enabled debug information for cifs:

echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
echo 7 > /proc/fs/cifs/cifsFYI
echo 1 > /sys/module/dns_resolver/parameters/debug

now I can see additional information in dmesg:

[350004.228812] fs/cifs/cifsfs.c: Devname: //SERVER/SHARE flags: 0
[350004.228856] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[350004.228858] fs/cifs/connect.c: Username: root
[350004.228862] fs/cifs/connect.c: file mode: 0660 dir mode: 0770
[350004.228866] fs/cifs/connect.c: CIFS VFS: in mount_get_conns as Xid: 109 with uid: 0
[350004.228867] fs/cifs/connect.c: UNC: \\SERVER\SHARE
[350004.228881] fs/cifs/connect.c: Socket created
[350004.228883] fs/cifs/connect.c: sndbuf 16384 rcvbuf 131072 rcvtimeo 0x6d6
[350004.229238] fs/cifs/connect.c: Demultiplex PID: 94569
[350004.229278] fs/cifs/fscache.c: cifs_fscache_get_client_cookie: (0x0000000035f51052/0x00000000f0122aa2)
[350004.229297] fs/cifs/connect.c: CIFS VFS: in cifs_get_smb_ses as Xid: 110 with uid: 0
[350004.229315] fs/cifs/connect.c: Existing smb sess not found
[350004.229321] fs/cifs/smb2pdu.c: Negotiate protocol
[350004.229350] fs/cifs/transport.c: Sending smb: smb_len=284
[350004.230011] fs/cifs/connect.c: RFC1002 header 0x114
[350004.230018] fs/cifs/smb2misc.c: SMB2 data length 85 offset 128
[350004.230019] fs/cifs/smb2misc.c: SMB2 len 213
[350004.230020] fs/cifs/smb2misc.c: length of negcontexts 60 pad 3
[350004.230046] fs/cifs/transport.c: cifs_sync_mid_result: cmd=0 mid=0 state=4
[350004.230052] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[350004.230054] fs/cifs/smb2pdu.c: mode 0x1
[350004.230055] fs/cifs/smb2pdu.c: negotiated smb3.1.1 dialect
[350004.230058] fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2 0x348 0x1bb92
[350004.230059] fs/cifs/asn1.c: OID len = 10 oid = 0x1 0x3 0x6 0x1
[350004.230060] fs/cifs/smb2pdu.c: decoding 2 negotiate contexts
[350004.230061] fs/cifs/smb2pdu.c: decode SMB3.11 encryption neg context of len 4
[350004.230061] fs/cifs/smb2pdu.c: SMB311 cipher type:2
[350004.230063] fs/cifs/connect.c: Security Mode: 0x1 Capabilities: 0x300056 TimeAdjust: 0
[350004.230064] fs/cifs/smb2pdu.c: Session Setup
[350004.230065] fs/cifs/smb2pdu.c: sess setup type 5
[350004.230069] fs/cifs/cifs_spnego.c: key description = ver=0x2;host=SERVER;ip4=XXX.XXX.XXX.XXX;sec=krb5;uid=0x0;creduid=0x0;user=root;pid=0x17167
[350004.235985] CIFS VFS: Verify user has a krb5 ticket and keyutils is installed
[350004.235994] CIFS VFS: \\SERVER Send error in SessSetup = -126
[350004.236000] fs/cifs/connect.c: CIFS VFS: leaving cifs_get_smb_ses (xid = 110) rc = -126
[350004.236004] fs/cifs/connect.c: build_unc_path_to_root: full_path=\\SERVER\SHARE
[350004.236006] fs/cifs/connect.c: build_unc_path_to_root: full_path=\\SERVER\SHARE
[350004.236008] fs/cifs/connect.c: build_unc_path_to_root: full_path=\\SERVER\SHARE
[350004.236011] fs/cifs/dfs_cache.c: do_dfs_cache_find: search path: \SERVER\SHARE
[350004.236013] fs/cifs/dfs_cache.c: do_dfs_cache_find: cache miss
[350004.236017] fs/cifs/dfs_cache.c: do_dfs_cache_find: search path: \SERVER\SHARE
[350004.236018] fs/cifs/dfs_cache.c: do_dfs_cache_find: cache miss
[350004.236029] fs/cifs/fscache.c: cifs_fscache_release_client_cookie: (0x0000000035f51052/0x00000000f0122aa2)
[350004.236037] fs/cifs/connect.c: CIFS VFS: leaving mount_put_conns (xid = 109) rc = 0
[350004.236039] CIFS VFS: cifs_mount failed w/return code = -2

The error is:
[350004.235985] CIFS VFS: Verify user has a krb5 ticket and keyutils is installed
[350004.235994] CIFS VFS: \\SERVER Send error in SessSetup = -126
[350004.236039] CIFS VFS: cifs_mount failed w/return code = -2

Of course keyutils is installed:

root@kubuntu-lts:# dpkg -l keyutils
ii keyutils 1.6-6ubuntu1 amd64 Linux Key Management Utilities

It looks like the kerberos ticket is not found to mount the share with UPN. But I have a valid ticket:

root@kubuntu-lts:# klist
Ticket cache: FILE:/tmp/krb5cc_10011_r0AC1F
Default principal: KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE

Valid starting Expires Service principal
21.10.2020 16:16:20 22.10.2020 02:16:20 <email address hidden>
        renew until 22.10.2020 16:16:20

My keytab:

root@kubuntu-lts:# klist -ket /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
   3 21.10.2020 15:43:11 kubuntu-lts$@MPI-DORTMUND.MPG.DE (arcfour-hmac)
   3 21.10.2020 15:43:11 kubuntu-lts$@MPI-DORTMUND.MPG.DE (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:11 kubuntu-lts$@MPI-DORTMUND.MPG.DE (aes256-cts-hmac-sha1-96)
   3 21.10.2020 15:43:12 KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE (arcfour-hmac)
   3 21.10.2020 15:43:12 KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:12 KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE (aes256-cts-hmac-sha1-96)
   3 21.10.2020 15:43:12 <email address hidden> (arcfour-hmac)
   3 21.10.2020 15:43:12 <email address hidden> (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:12 <email address hidden> (aes256-cts-hmac-sha1-96)
   3 21.10.2020 15:43:12 <email address hidden> (arcfour-hmac)
   3 21.10.2020 15:43:13 <email address hidden> (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:13 <email address hidden> (aes256-cts-hmac-sha1-96)
   3 21.10.2020 15:43:13 <email address hidden> (arcfour-hmac)
   3 21.10.2020 15:43:13 <email address hidden> (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:13 <email address hidden> (aes256-cts-hmac-sha1-96)
   3 21.10.2020 15:43:13 <email address hidden> (arcfour-hmac)
   3 21.10.2020 15:43:13 <email address hidden> (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:14 <email address hidden> (aes256-cts-hmac-sha1-96)
   3 21.10.2020 15:43:14 <email address hidden> (arcfour-hmac)
   3 21.10.2020 15:43:14 <email address hidden> (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:14 <email address hidden> (aes256-cts-hmac-sha1-96)
   3 21.10.2020 15:43:14 <email address hidden> (arcfour-hmac)
   3 21.10.2020 15:43:14 <email address hidden> (aes128-cts-hmac-sha1-96)
   3 21.10.2020 15:43:14 <email address hidden> (aes256-cts-hmac-sha1-96)

This looks like a bug or regression for my because it's working on RedHat 7.9 with a previous release of cifs-utils.

[Sergio Durigan Junior (sergiodj)]

Thanks for the bug report, Alexander!

I have a local setup here with an Active Directory running on Windows Server 2019, and I fired up a Focal VM and tried to reproduce the steps you mentioned above. In a nutshell, here's what I did:

- realm join mydomain --membership-software=adcli
- Installed krb5-user and made sure everything was working correctly
- Installed smbclient et al and made sure everything was also working correctly
- Installed keyutils

Then, I acquired a krb5 ticket (using "kinit user", but without resorting to a separate keytab, as you did above):

# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: <email address hidden>

Valid starting Expires Service principal
10/27/2020 22:23:04 10/28/2020 08:23:04 <email address hidden>
 renew until 10/28/2020 22:23:01

Finally, I tried mounting a SMB share from the Windows Server machine:

# mount //ad1.ad1.example.com/windows /mnt/ -o sec=krb5,multiuser,file_mode=0660,dir_mode=0770,nounix,noserverino

And everything worked correctly. I'm able to list the contents of the share, and if I switch to another user I see that the multiuser option kicks in and I see the files' owner/group is changed accordingly.

Here's the version of everything I'm using:

cifs-utils:
  Installed: 2:6.9-1ubuntu0.1
sssd:
  Installed: 2.2.3-3
smbclient:
  Installed: 2:4.11.6+dfsg-0ubuntu1.5

Unless I'm missing some step from your configuration, it seems I can't reproduce the bug. The only way I can reproduce the same error you had is when I kdestroy my credentials and try to mount the share again.

I will try setting up a samba share on another machine in the realm and then try to reproduce the issue, but initially I don't see how this could make a difference. I'll get back when I have something.

[Alexander Fieroch (fieroch)]

Hm, if I add an AD username I can mount the share with an valid kerberos ticket for the user:

root@kubuntu-lts:# mount -vvv -o sec=krb5,multiuser,vers=3.0,cruid=ntfieroch //FILESERVER/share /mnt/test/
mount.cifs kernel mount options: ip=X.X.X.X,unc=\\FILESERVER/share,sec=krb5,multiuser,vers=3.0,cruid=10011,user=root,pass=********

I want to mount the samba share with multiuser option with the machine accounts UPN in AD. Is that working for you?

If I specify UPN I get:

root@kubuntu-lts:# kinit -k KUBUNTU-LTS$
root@kubuntu-lts:# klist -ket /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
   2 22.10.2020 10:54:16 KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE (arcfour-hmac)
   2 22.10.2020 10:54:16 KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE (aes128-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE (aes256-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 <email address hidden> (arcfour-hmac)
   2 22.10.2020 10:54:16 <email address hidden> (aes128-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 <email address hidden> (aes256-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 <email address hidden> (arcfour-hmac)
   2 22.10.2020 10:54:16 <email address hidden> (aes128-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 <email address hidden> (aes256-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 <email address hidden> (arcfour-hmac)
   2 22.10.2020 10:54:16 <email address hidden> (aes128-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 <email address hidden> (aes256-cts-hmac-sha1-96)
   2 22.10.2020 10:54:16 <email address hidden> (arcfour-hmac)
   2 22.10.2020 10:54:17 <email address hidden> (aes128-cts-hmac-sha1-96)
   2 22.10.2020 10:54:17 <email address hidden> (aes256-cts-hmac-sha1-96)

root@kubuntu-lts:# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: KUBUNTU-LTS$@MPI-DORTMUND.MPG.DE

Valid starting Expires Service principal
29.10.2020 13:49:42 29.10.2020 23:49:42 <email address hidden>
        renew until 30.10.2020 13:49:42

root@kubuntu-lts:# mount -vvv -o sec=krb5,multiuser,vers=3.0,username='KUBUNTU-LTS$' //FILESERVER/share /mnt/test/
mount.cifs kernel mount options: ip=X.X.X.X,unc=\\FILESERVER\share,sec=krb5,multiuser,vers=3.0,user=KUBUNTU-LTS$,pass=********
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

The samba configuration for the smb share on FILESERVER has the UPN as valid user:
[share]
  path = /mnt/share
  valid users = +"domain users", "KUBUNTU-LTS$"
  force group = "domain users"

Hm, now I get a different return code =-13 in dmesg:

[87872.570848] fs/cifs/cifsfs.c: Devname: //FILESERVE...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Karl O. Pinc (kop)]

I believe this is the same bug as Debian bug#986168:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986168

A work-around can be found there.