Problems with SSSD + pam mount

[Paul-Georg Majev]

In our IT-Department we are preparing ourselves for Ubuntu 18.04.
While doing so, we discovered, that Domain Users loging into Ubuntu with their home Directories mounted through pam-mount can not modify the icons in the Sidebar and are also unable to activate certain settings. What we discovered so far is:

1. They can not change their Keyboard Layout.
2. They can not apply "show hidden files" in nautilus.
3. They can not change the icons in the sidebar (as mentioned)

What could be the Problem here and how should we go about solving it?

How to reproduce it:
1. Join Client to domain via net ads join -U administrator
2. Restart SSSD Daemon
3. Configure pam_mount.conf.xml as such:
<pam_mount>
<debug enable = "1" />
<mntoptions allow = "nosuid,nodev,loop,encryption,fsck,nonempty,allow_root,allow_other" />
<mntoptions require = "nosuid,nodev" />
<logout wait = "5" hup = "0" term="no" kill="no" />
<mkmountpoint enable = "1" remove = "true" />
<volume fstype = "cifs"
       server = "server.domain"
       path = "home/%(DOMAIN_USER)"
       mountpoint = "/home/%(DOMAIN_USER)"
       options = "dir_mode=0700,workgroup=DOMAIN,iocharset=utf8,username=%(DOMAIN_USER),cruid=%(USERUID),uid=%(USERUID)"
       user = "*"/>
</pam_mount>
4. Restart
5. Log in with a domain user
Everything seems to work, all files appear in the home directory, but the problems above occur.
Doing the exact same thing in Ubuntu 16.04 works.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1764778/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Andreas Hasenack (ahasenack)]

Why do you think it's related to pam_mount? Are the mount points mounted read-only?

Can you show the output of "sudo mount -t cifs"?

[Paul-Georg Majev (peajee)]

As we are not understanding the problem ourselves, we are not in the position to pinpoint which package is responsible for the problem. SSSD and pam mount are just the main two packages installed by us for our domain setup. Thats why I included both of them in the bug report. There are no problems with reading or writing files such as text files in folders that are mounted by pam_mount. But if I am not mistaken, the settings described as not working above are also saved in the users home directory?

The output of sudo mount -t cifs

//server.domain/home/testuser on /home/testuser type cifs (rw,relatime,vers=default,cache=strict,username=testuser,domain=DOMAIN,uid=826404655,forceuid,gid=826400513,forcegid,addr=XX.XX.XX.X,file_mode=0755,dir_mode=0700,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)

[Paul-Georg Majev (peajee)]

Just to be clear: We are not 100% sure, the problem is not in our configuration. We just dont see why the same configuration would yield such different results between 16.04 and 18.04.

This is our sssd.conf

[sssd]
config_file_version = 2
services = nss,pam
domains = domain.domain

[nss]

[pam]

[domain/domain.domain]
id_provider = ad
access_provider = ad
auth_provider = ad
enumerate = true
cache_credentials = false
case_sensitive = false
override_homedir = /home/%u
default_shell = /bin/bash
create_homedir = true
remove_homedir = true
ad_gpo_access_control = permissive

[Andreas Hasenack (ahasenack)]

Thanks for the feedback.

I haven't tried the sssd gpo controls myself yet, and I see you have them set at "permissive", which tells me it would not be preventing users from changing things. But let's say there is a bug in that area, would the GPO controls be able to cause what you are seeing?

Can you also verify that all files under the user's home directory are writable and owned by the actual user?

Bionic is shipping with sssd 1.16.1, some default might have changed there, or a new feature. If you change "ad_gpo_access_control" to "disabled", does that change anything?

Do you see anything related in the system logs? Check:
/var/log/syslog <-- gnome-shell logs go here as well
/var/log/sssd/*

[Paul-Georg Majev (peajee)]

Setting ad_gpo_access_control to disabled did not solve the problem.

I can confirm, that all files in the domain users mounted homefolder are owned by the user with the rights set to 700.

In the timeframe i tried to trigger the error, the only entrances in /var/log/syslog were:

Apr 23 09:50:10 CLIENT gnome-shell[3975]: Some code accessed the property 'discreteGpuAvailable' on the module 'appDisplay'. That property was defined with 'let' or 'const' insidethe module. This was previously supported, but is not correct according to the ES6 standard. Any symbols to be exported from a module must be defined with 'var'. The property access will work as previously for the time being, but please fix your code anyway.

Apr 23 09:50:12 CLIENT gnome-shell[3975]: failed to commit changes to dconf: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code2: Failed to rename file “/home/USER/.config/dconf/user.ZHTTHZ” to “/home/USER/.config/dconf/user”: g_rename() failed: Permission denied

Apr 23 09:50:19 CLIENT deja-dup-monito[4430]: failed to commit changes to dconf: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code2: Failed to rename file “/home/USER/.config/dconf/user.RH6NHZ” to “/home/USER/.config/dconf/user”: g_rename() failed: Permission denied

Apr 23 09:50:19 CLIENT deja-dup-monito[4430]: failed to commit changes to dconf: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code2: Failed to rename file “/home/USER/.config/dconf/user.S09MHZ” to “/home/USER/.config/dconf/user”: g_rename() failed: Permission denied

Apr 23 09:50:19 CLIENT deja-dup-monito[4430]: failed to commit changes to dconf: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code2: Failed to rename file “/home/USER/.config/dconf/user.VSCMHZ” to “/home/USER/.config/dconf/user”: g_rename() failed: Permission denied

So we actually get a Permission denied error even though the user has full rights (700) in the dconf folder. When executing ls -lh in the dconf folder it shows a file named user. However, when I try to do ls -lh or more on the file, the error is "file not found".

The /var/log/sssd/ files had no content.

[Andreas Hasenack (ahasenack)]

This permission denied error looks relevant:

Apr 23 09:50:12 CLIENT gnome-shell[3975]: failed to commit changes to dconf: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code2: Failed to rename file “/home/USER/.config/dconf/user.ZHTTHZ” to “/home/USER/.config/dconf/user”: g_rename() failed: Permission denied

Did you sanitize this log and replace the real user with "USER"?

Can you verify if you can do simple file operations in that directory? Create a file, mv it to another name in the same path, *as* the user (not root or somebody else)?

Also confirm that the user processes are owned by the same uid/gid then the files in /home/$user. Maybe you have a uid/gid mismatch between the userid who owns the files in the fileserver, and what sssd gives that user in the remote machine. You can use "ps fauxw", "getent passwd <user>",
"ls -ln" to check the numeric uids/gids.

Are there symlinks involved in this networked /home setup?

[Paul-Georg Majev (peajee)]

Did you sanitize this log and replace the real user with "USER"? Yes I did.

Can you verify if you can do simple file operations in that directory? Create a file, mv it to another name in the same path, *as* the user (not root or somebody else)? Indeed I am able to do all those things without problem.

The owner uid/gids of the dconf folder match with the ids output by getent passwd. The users processes are also executed with that uid.

Are there symlinks involved in this networked /home setup? No, there are no symlinks involved.

[Andreas Hasenack (ahasenack)]

Do you have apparmor DENIED errors in your dmesg output? Or other maybe relevant messages there that match the timing of the "g_rename() failed" error?

[Paul-Georg Majev (peajee)]

Not that I can see. At the time of trying to change an icon in the sidebar, no errors are output to dmesg (tried with dmesg -Tw, dmesg -w and just dmesg without arguments). The last error before that is:

[Mo Apr 23 18:12:56 2018] CIFS VFS: Send error in SessSetup = -2
[Mo Apr 23 18:12:56 2018] CIFS VFS: cifs_mount failed w/return code = -2
[Mo Apr 23 18:12:57 2018] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). Toect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.

Apparently concerning the pam mount.

The only Apparmor DENIED error I could find was for cupsd.

[Andreas Hasenack (ahasenack)]

Can you please paste me the output of "mount -t cifs" on a system where this problem isn't happening?

Also, please paste the exact versions of cifs-utils in both cases and the running kernel.

As for a minimal reproducing case I'm trying to find, does this fail for you as one of these regular users (just a random dconf setting, harmless enough, to play with)

dconf write /org/gnome/rhythmbox/player/volume 1.0

[Andreas Hasenack (ahasenack)]

I setup a watch on the dconf directory, and can confirm the rename happens just as in your error log message when I use dconf:

andreas@nsnx:~$ inotifywait -m /home/andreas/.config/dconf/
Setting up watches.
Watches established.

(run in another terminal: dconf write /org/gnome/rhythmbox/player/volume 1.0)

/home/andreas/.config/dconf/ OPEN user
/home/andreas/.config/dconf/ CREATE user.3S6GHZ
/home/andreas/.config/dconf/ OPEN user.3S6GHZ
/home/andreas/.config/dconf/ MODIFY user.3S6GHZ
/home/andreas/.config/dconf/ CLOSE_WRITE,CLOSE user.3S6GHZ
/home/andreas/.config/dconf/ MOVED_FROM user.3S6GHZ
/home/andreas/.config/dconf/ MOVED_TO user
/home/andreas/.config/dconf/ CLOSE_NOWRITE,CLOSE user

I'm about to try that on a cifs mount. I also checked the documentation for rename(2) (the system call that g_rename() is a wrapper for, see https://developer.gnome.org/glib/stable/glib-File-Utilities.html#g-rename), and there is a scenario where it would fail due to the filesystem not allowing it:

       EPERM or EACCES
(...) ; or the filesystem containing pathname does not support renaming of the type requested.

I cooked up a quick C rename.c program but it worked just fine in my cifs mount, so the investigation continues.

[Andreas Hasenack (ahasenack)]

I was able to reproduce this with a bionic desktop and a home-mounted cifs share. Same setup but with xenial, and with the same user against the same windows server, worked just fine.

[Andreas Hasenack (ahasenack)]

Ok, the problem is caused by the change in minimum protocol version. In xenial, smb 1.0 is the default, but in bionic it's 3.0.

In the working xenial case, when I force protocol version 3.0 via "vers=3.0" in the pam_mount config, I get the same error as in bionic.

The troubleshooting continues!

[Andreas Hasenack (ahasenack)]

I'm wondering if this is not related to https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1572132

I see less files when I mount the share with smb3. In particular, ~/.config/dconf/user is missing when using smb3:
--- home-list-smb3.txt 2018-04-24 21:43:54.727520865 +0000
+++ home-list-smb1.txt 2018-04-24 21:47:30.003049993 +0000
@@ -255,6 +255,7 @@
 ./.config/compiz-1/compizconfig/config
 ./.config/compiz-1/compizconfig/done_upgrades
 ./.config/dconf
+./.config/dconf/user
 ./.config/evolution
 ./.config/evolution/sources
 ./.config/evolution/sources/birthdays.source

With smb3, ~/.config/dconf shows up as empty:
jsmith@xenial-desktop:~/.config/dconf$ l
total 8.0K
drwx------ 2 jsmith domain users 4.0K Apr 24 21:59 .
drwx------ 2 jsmith domain users 4.0K Apr 24 21:12 ..

But is it?
jsmith@xenial-desktop:~/.config/dconf$ touch user
touch: cannot touch 'user': No such file or directory

Other files can be created just fine:
jsmith@xenial-desktop:~/.config/dconf$ touch bla
jsmith@xenial-desktop:~/.config/dconf$ l
total 8.0K
drwx------ 2 jsmith domain users 4.0K Apr 24 21:59 .
drwx------ 2 jsmith domain users 4.0K Apr 24 21:12 ..
-rwxr-xr-x 1 jsmith domain users 0 Apr 24 21:59 bla
jsmith@xenial-desktop:~/.config/dconf$

Intriguing.

[Paul-Georg Majev (peajee)]

First of all: Thank you very much for investing so much time into getting to the bottom of this problem! We are very grateful for it and relieved, that the error could be reproduced. Is there anything else I can do to help solve this?

[Andreas Hasenack (ahasenack)]

Can you try setting vers=1.0 in the mount options?

On Wed, Apr 25, 2018, 04:41 Paul-Georg Majev <email address hidden>
wrote:

> First of all: Thank you very much for investing so much time into
> getting to the bottom of this problem! We are very grateful for it and
> relieved, that the error could be reproduced. Is there anything else I
> can do to help solve this?
>
> --
> You received this bug notification because you are subscribed to sssd in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1764778
>
> Title:
> Problems with SSSD + pam mount
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/libpam-mount/+bug/1764778/+subscriptions
>

[Paul-Georg Majev (peajee)]

I did. It resolves the error.

[Andreas Hasenack (ahasenack)]

Adding a cifs-utils task. It's closer to the problem. But the bug might be in the kernel bit.

[Walter Cheuk (wwycheuk)]

Same problem happened to Ubuntu 16.04.5. I have to specify "vers=1.0" to mount a old NAS drive successfully. The help said the default version is 1.0, so I spent a lot of time to check other options. Anyone please update the help text.

[Tristan Engel (tristan-engel)]

I found the Solution, it is not really a Bug, but missing Docs.
You have to add two Mount-Options so that it works and change a Gnome Setting.
(1) "nobrl"-Option
(2) "mfsymlinks"-Option
(3) service-db:keyfile/user in /etc/dconf/profile/user

https://help.gnome.org/admin/system-admin-guide/stable/dconf-nfs-home.html.en

Thanks to https://serverfault.com/questions/922926/pam-mount-home-directories-over-cifs-sssd-and-bionic-beaver for the answer.