pam_mount trying to unmount my auto-mounted encrypted partitions every hour via CRON

[Ari]

Binary package hint: libpam-mount

Since I upgraded to Karmic, I noticed the weird fact that my two auto-mounted LUKS-encrypted partitions randomly seemed to disappear from mtab - I had to manually mount them again to use them. I've been using them for several years with all ubuntus so far, never had a problem until now.

These partitions are automatically mounted on login via pam_mount. These are the entries in pam_mount.conf.xml:

<volume fstype="crypt" path="/dev/sda3" mountpoint="/media/sda3" options="cipher=aes" />
<volume fstype="crypt" path="/dev/sda5" mountpoint="/media/sda5" options="cipher=aes" />

After digging in the logs, I've found that there is some sort of a cron job trying to unmount my two partitions and running every 60 minutes. If the partitions are busy then they survive; if they are not busy, then they are umounted (very weird !!!!!)

From my auth.log I see this every hour:

Nov 15 11:17:01 nahuatl CRON[1230]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(pam_mount.c:516): warning: could not obtain password interactively either
Nov 15 11:17:01 nahuatl CRON[1230]: pam_unix(cron:session): session closed for user root
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:67): umount messages:
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): umount: /media/sda5: device is busy.
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): (In some cases useful info about processes that use
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): the device is found by lsof(8) or fuser(1))
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): umount /media/sda5 failed with run_sync status 1
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): Command failed: Device busy
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): umount.crypt(crypto-dmc.c:168): Could not unload dm-crypt device "/dev/mapper/_dev_sda5", cryptsetup returned HXproc status 240
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:698): unmount of /dev/sda5 failed
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:67): umount messages:
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): umount: /media/sda3: device is busy.
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): (In some cases useful info about processes that use
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): the device is found by lsof(8) or fuser(1))
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): umount /media/sda3 failed with run_sync status 1
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): Command failed: Device busy
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:71): umount.crypt(crypto-dmc.c:168): Could not unload dm-crypt device "/dev/mapper/_dev_sda3", cryptsetup returned HXproc status 240
Nov 15 11:17:01 nahuatl CRON[1230]: pam_mount(mount.c:698): unmount of /dev/sda3 failed

In my view this is a major bug.

A hint or workaround on how to stop this CRON thing from trying to unmount my drives would be greatly appreciated.

I've found only this similar reference:

http://groups.google.com/group/linux.debian.user.german/browse_thread/thread/1c632245a397f7c9?utoken=JRGmfjsAAAD-4m7bW_nhUf_RRJzb_a9w_ObWXSwL4IOKiU8782Hmp-EH8COufF2SlzXILlU-DjyKjuOEmBHIEfBNJAzzPaif

seems to hint that the problem was introduced recently in debian, in libpam_mount 1.27-4

[Ari (ari-reads)]

After further investigation this seems to be related to a recent upstream bug:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550892

[Steve McGrath (smcgrath23)]

Looks like everytime Cron runs, it opens a PAM session, which calls common_session_noninteractive, which contains pam_mount. That's what message #48 in the upstream report is alluding to.

[Ari (ari-reads)]

I've been looking into crontab and the hourly task cron configuration and can't find any reference to calls to common_session_noninteractive nor to pam_mount.

I really need to stop cron + pam from periodically dismounting the encrypted partitions, it is extremely annoying and causing trouble to all server users.

What about removing the pam_mount call from common_session_noninteractive? Can someone give me a hint as to where to look for the common_session_noninteractive calls to do the removal? <TIA>

[Ari (ari-reads)]

It appears the bad behavior was introduced by a patch to cron by steve langasek

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=543303

[Steve McGrath (smcgrath23)]

I probably should have elaborated on this a little more.

Cron itself, when running *any* jobs from the crontab, is creating a PAM session as part of the environment to run the job in.

Cron's pam configuration is in /etc/pam.d/cron, which has a line to include /etc/pam.d/common_session_noninteractive.

common_session_noninteractive includes the pam_mount module, which may be necessary for other non-interactive processes. I'm really not sure. My comments are mostly directed at whoever eventually tries to fix this bug.

I suppose if I were having this problem, I'd try commenting out the pam_mount line in /etc/pam.d/common_session_noninteractive.

If that does in fact break other things, I might instead try copying all the directives except pam_mount from common_session_noninteractive into /etc/pam.d/cron, and removing the "@include common_session_noninteractive" line from /etc/pam.d/cron.

[Steve Langasek (vorlon)]

The problem is not caused by anything in Debian bug #543303. pam_mount is *also* referenced in /etc/pam.d/common-session.

The only recent behavior change in how pam_mount is used by cron is that in karmic, libpam-mount now automatically configures itself by default to do what README.Debian previously recommended that users do by hand.

Ultimately, this bug appears to come down to a reference counting bug in libpam-mount. libpam-mount should keep track of the number of sessions open for the user, and only unmount when the count drops to zero.

[Micah Gersten (micahg)]

As there is an upstream bug and the information in this bug seems complete as well, I am marking this Triaged. I concur with Steve McGrath that this should be Medium as it's an annoyance, but no data loss seems apparent.

[Ari (ari-reads)]

Thanks everyone for the comments

As a temporary workaround I commented-out the pam_mount call in /etc/pam.d/common_session_noninteractive

No apparent breakage, nothing of interest in the logs other than that I see in that cron still runs hourly, but now it leaves my mounted partitions alone :)

[Steve McGrath (smcgrath23)]

Glad to hear that you were able to workaround this. Now that this bug is marked Traiged, the developers will be looking at it and hopefully this will be fixed in the package.

[Stefan (alter-depp)]

I think I had the same problem, but since upgrade to jaunty. The solution was to run fsck on /dev/mapper/_dev_sdb1 manually, because some inodes were corrupted. It was not easy to find this solution. I also don't know how to reproduce the problem. Would be nice, if the system could give a good hint what to do.

Stefan

[David Gelvin (dgel923)]

Any progress on this? I'm running 12.04 beta2 with all the updates applied and I am experiencing this. I have two volume definitions in pam_mount.conf.xml. One is fstype 'crypt' that mounts my luks encrypted home directory, the other is fuse encfs that mounts an encfs encrypted folder.

When I log in, both are mounted fine. However, if I run any command using sudo, at the completion of the command, pam attempts to unmount both my home directory and the encfs folder. It is unable to unmount my home directory (which produces a number of 'ummount failed' messages, but it does successfully unmount my encfs folder. Which means that I need to remount it after I execute any sudo command.

[Omar W. Llorente (omar-wllorente)]

I was experiencing problems in ubuntu 12.04 fully updated and seems like solved with this option in pam_mount.conf.xml:

<logout wait="3000" hup="no" term="no" kill="yes" />

Hope that helps.

Omar