fuse mounts hang on xattr retrieval with auditd

[Peter Moody]

Impact: If audit is enabled in the kernel, mounting fuse filesystems hangs.
Development branch (and maverick): Addressed by upgrade to upstream fuse >= 2.8.3.
Patch: https://code.launchpad.net/~cjwatson/ubuntu/lucid/fuse/audit-hang (backported from upstream - it's not particularly short, but there were three intertwined fixes and this was the best I could do)
TEST CASE: Install auditd and configure a rule, then try to mount a fuse filesystem.
Regression potential: I'd recommend generally playing around with fuse and making sure it still works. Running the udisks test suite might be a good idea.

Original report follows:

Binary package hint: libfuse2

I'm being struck by the symptoms reported in this bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=493565

Namely, when auditd has any rules configured, I can't mount any fuse file-systems. Comment 61 seems to have a pretty solid description of what's happening underneath and the solution appears to be upgrading to a libfuse > 2.8.3. Only 2.8.1 (http://packages.ubuntu.com/lucid/libfuse2) appears to be available currently for lucid.

Cheers,
peter

[Mamoru (mamoru-redhat-bugs)]

Created attachment 337759
tarball containing ps auwwx, .xsession-errors log

Description of problem:

With the following procedure:
[1] Reboot with init 3. On tty1 log in as root
[2] On tty1, type # init 5 as root, then GDM launches
[3] With GDM log in as non-root user to GNOME session
[4] log out from GNOME session
[5] Again log in to GNOME session as the same user on [3]

Then GNOME session hangs.

Version-Release number of selected component (if applicable):
gvfs-1.2.0-2.fc11.i586
fuse-2.7.4-3.fc11.i586

How reproducible:
Currently 100%

Steps to Reproduce:
1. See above
2.
3.

Additional info:
- The results of "ps auwwx" on each stage [1] - [5] and .xsession-errors
  are in the attached tarball
- On the stage [4], when I kill the process 3576 and 3578
  with SIGKILL, the second login to GNOME works.

[Tomáš (tom-redhat-bugs-1)]

Just tested according to the repro steps provided, works for me. Tested with newly created account as well as with my old personal account.

From your logs, I can see possible point of failure:

gnome-session[4402]: WARNING: Application 'gnome-settings-daemon.desktop' failed to register before timeout
gnome-session[4402]: WARNING: Application 'gnome-panel.desktop' failed to register before timeout

(imsettings-applet:4548): IMSettings-WARNING **: メソッド WhatsInputMethodRunning' の呼出に失敗しました : Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

(gnome-panel:4537): GVFS-RemoteVolumeMonitor-WARNING **: invoking IsSupported() failed for remote volume monitor with dbus name org.gtk.Private.GduVolumeMonitor: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Looks like dbus is not running in the session. Can you post `rpm -qa | grep -e dbus -e glib2` here?

[Mamoru (mamoru-redhat-bugs)]

Hi:

[tasaka1@localhost ~]$ rpm -qa | grep -e dbus -e glib2 | sort
dbus-1.2.12-1.fc11.i586
dbus-debuginfo-1.2.12-1.fc11.i586
dbus-devel-1.2.12-1.fc11.i586
dbus-glib-0.80-2.fc11.i586
dbus-glib-debuginfo-0.80-2.fc11.i586
dbus-glib-devel-0.80-2.fc11.i586
dbus-libs-1.2.12-1.fc11.i586
dbus-python-0.83.0-5.fc11.i586
dbus-x11-1.2.12-1.fc11.i586
glib2-2.19.10-2.fc11.i586
glib2-debuginfo-2.19.10-2.fc11.i586
glib2-devel-2.19.10-2.fc11.i586
pulseaudio-libs-glib2-0.9.15-8.test7.fc11.i586
python-slip-dbus-0.1.15-3.fc11.noarch
ruby-glib2-0.18.1-6.fc11.i586
ruby-glib2-devel-0.18.1-6.fc11.i586

[Tomáš (tom-redhat-bugs-1)]

Can you try upgrading to glib2-2.20.0-1.fc11? I doubt it will make any difference, but just to be sure.

[Tomáš (tom-redhat-bugs-1)]

(In reply to comment #3)
> Can you try upgrading to glib2-2.20.0-1.fc11? I doubt it will make any
> difference, but just to be sure.
Oh, that build is failed, disregard my comment, my mistake.

[Tomáš (tom-redhat-bugs-1)]

Can you please check selinux status? (getenforce)
Also please check dmesg for any segfaults and avc denials.

We had some troubles with gvfs-fuse-daemon not exiting after logout, but it shouldn't cause failures on second login. I don't see anything like 3576 (fusermount) or 3578 (mount) here, make sure you don't have anything like this in /etc/fstab.

[Mamoru (mamoru-redhat-bugs)]

[tasaka1@localhost ~]$ getenforce
Disabled

My fstab is:
-----------------------------------------------------
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
/dev/VolGroup00/LogVol02 /home ext3 defaults 1 2
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
LABEL=/tmp /tmp ext3 defaults 1 2
/dev/VolGroup00/LogVol01 /usr ext3 defaults 1 2
LABEL=/var /var ext3 defaults 1 2
/dev/sda6 swap swap defaults 0 0
#/dev/sda2 /mnt/Windows-VFAT vfat noauto,owner,user,codepage=932,iocharset=euc-jp,posix,umask=0022 1 2
#/dev/sda1 /mnt/Windows-NTFS-3g ntfs-3g noauto,ro,locale=ja_JP.UTF-8 0 0
------------------------------------------------------
( Now I commented out the last 2 lines but the result is the
  same )

[Alexander (alexander-redhat-bugs)]

Its kinda weird that you see the fusermount and mount processes after the first login. These are supposed to be shortlived things that run and exit, only gvfs-fuse-daemon should be running for any longer period.

So, it seems to me that the process of mounting the fuse filesystem on ~/.gvfs somehow hangs.

Maybe you could try attaching to the fusermount and/or mount processes and get a backtrace?

[Tomáš (tom-redhat-bugs-1)]

Can you try the following in your first running session? Mount some ftp or sftp filesystem using Nautilus, make sure you can browse it. Then go to ~/.gvfs, you should see those mounts there. Check again if you can browse the mounts. I was wondering if the gvfs-fuse-daemon works as expected for you.

[Mamoru (mamoru-redhat-bugs)]

Created attachment 338039
screenshot of nautilus

First for this:

(In reply to comment #8)
> Can you try the following in your first running session? Mount some ftp or sftp
> filesystem using Nautilus, make sure you can browse it. Then go to ~/.gvfs, you
> should see those mounts there. Check again if you can browse the mounts. I was
> wondering if the gvfs-fuse-daemon works as expected for you.

Well, the actual result for this is:
When I launch $ nautilus from uxterm:
- nautilus won't show the contents of my home directory.
  (please see the attached png file)
  When I move the mouse pointer on nautilus, the mouse pointer
  shows it is still under processing.
- By "File -> Connect to Server" (I guess, my locale is ja_JP),
  mounting ftp server succeeds.
- I don't know how to go to ~/.gvfs on nautilus, because of the issue I wrote
  first

[Mamoru (mamoru-redhat-bugs)]

Created attachment 338049
gdb log

And gdb log is attached.

Note:
For process 3707, when trying to attach this process,
gdb itself hangs, so I had to kill gdb.

[Mamoru (mamoru-redhat-bugs)]

Created attachment 338060
ps auwwx when logged in GNOME session with kernel 2.6.27.x

(In reply to comment #7)
> Its kinda weird that you see the fusermount and mount processes after the first
> login. These are supposed to be shortlived things that run and exit, only
> gvfs-fuse-daemon should be running for any longer period.

Well, I tried to reboot with kernel-2.6.27.21-170.2.56.fc10.i686
and actually
- 2nd login to GNOME succeeds
- There are no "fusermount" "mount" process

[GOOD] kernel-2.6.27.21-170.2.56.fc10.i686
[BAD ] kernel-2.6.29.1-35.rc1.fc11.i586
[BAD ] kernel-2.6.29.1-46.fc11.i586

[Mamoru (mamoru-redhat-bugs)]

Created attachment 338061
dmesg (2.6.29.1-46.fc11.i586)

[Tomáš (tom-redhat-bugs-1)]

(In reply to comment #11)
> Well, I tried to reboot with kernel-2.6.27.21-170.2.56.fc10.i686
> and actually
> - 2nd login to GNOME succeeds
> - There are no "fusermount" "mount" process
>
> [GOOD] kernel-2.6.27.21-170.2.56.fc10.i686
> [BAD ] kernel-2.6.29.1-35.rc1.fc11.i586
> [BAD ] kernel-2.6.29.1-46.fc11.i586
Good catch, 2.6.29-21.fc11.x86_64 here, no problems with fuse.

(In reply to comment #9)
> - nautilus won't show the contents of my home directory.
> (please see the attached png file)
> When I move the mouse pointer on nautilus, the mouse pointer
> shows it is still under processing.
This is weird. Were there any nautilus errors on the uxterm shell? Could be
broken thumbnailer, but that should not make Nautilus unusable.
> - I don't know how to go to ~/.gvfs on nautilus, because of the issue I wrote
> first
You can use other tools, like mc (Midnight Commander) in a terminal or bash
commands

[Mamoru (mamoru-redhat-bugs)]

Created attachment 338081
nautilus output on terminal

With 2.6.29.1-46.fc11.i586:

(In reply to comment #13)
> (In reply to comment #11)
> (In reply to comment #9)
> > - nautilus won't show the contents of my home directory.
> > (please see the attached png file)
> > When I move the mouse pointer on nautilus, the mouse pointer
> > shows it is still under processing.
> This is weird. Were there any nautilus errors on the uxterm shell? Could be
> broken thumbnailer, but that should not make Nautilus unusable.

nautilus output is attached here.

> > - I don't know how to go to ~/.gvfs on nautilus, because of the issue I wrote
> > first
> You can use other tools, like mc (Midnight Commander) in a terminal or bash
> commands

Well, actually I tried
$ ls -al ~/.gvfs
$ cd ~/.gvfs
but they all hang... It seems that nautilus is hanging just
because nautilus cannot access to ~/.gvfs

By the way with kernel 2.6.27.21-170.2.56.fc10.i686
--------------------------------------------------------
$ ls -al ~/.gvfs
合計 24
dr-x------ 3 tasaka1 tasaka1 0 2009-04-04 01:06 .
drwx------. 244 tasaka1 tasaka1 20480 2009-04-04 01:06 ..
drwx------ 1 tasaka1 tasaka1 0 1970-01-01 09:00 ftp %28ftp.riken.go.jp%29
--------------------------------------------------------

[Alexander (alexander-redhat-bugs)]

Hmm, it seems like there is something wrong in the kernel wrt fuse mounts. But I don't see anything weird in the dmesg output.

Any chance you could try various kernel versions to try to figure out the first broken one?

[Dmytro (dmytro-redhat-bugs)]

This bug should block F13Target rather than depend on F13Target (since this bug can be fixed without fixing F13Target). I do not have permissions to fix this myself.

[Christopher (christopher-redhat-bugs)]

Fixed dependency/block relationships.

[Josef (josef-redhat-bugs)]

This bz depends on 577947, since we need mount to have the --no-canonicalize option. Once util-linux-ng is updated, fuse can be updated to 2.8.3 and this problem will go away.

[Ville-Pekka (ville-pekka-redhat-bugs)]

Are the target bugs actually used for anything these days? I think I've seen some discussion about Fedora possibly not using target bugs at all in the future. Should this bug rather depend on the Fedora 13 Blocker bug?

[Adam (adam-redhat-bugs)]

It's not a release blocking issue.

--
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

[John (john-redhat-bugs)]

Re comment 86, I can see that this might not block the release, but in that case is there a status that says, if this bug isn't fixed by F13, remove the software from the release?

Just how defective does software have to be to be pulled? Something that verifiably prevents all logins every time it is used would seem to me to be a problem.

Re comment 70, this makes total sense to me but it should be backported to F11 too.

Re comment 81, thanks for the info!

[Adam (adam-redhat-bugs)]

"Re comment 86, I can see that this might not block the release, but in that
case is there a status that says, if this bug isn't fixed by F13, remove the
software from the release?"

No.

"Just how defective does software have to be to be pulled? Something that
verifiably prevents all logins every time it is used would seem to me to be a
problem."

I don't know if there's a definitive policy on this. You could perhaps bring it to the attention of FESCo or the packaging committee?

--
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

[Josef (josef-redhat-bugs)]

Ok now that

https://bugzilla.redhat.com/show_bug.cgi?id=577947

has made it into F12, all that needs to be done is fuse needs to be updated to 2.8.3. Once thats done this problem will go away.

[Tomáš (tom-redhat-bugs-1)]

*** Bug 582878 has been marked as a duplicate of this bug. ***

[Stas (stas-redhat-bugs)]

Very annoying bug, nice to see
a progress with it.
But I am having the different
trace, in my case the lstat causes
the problem, and not readlink().
Does this make any difference?

mc S ffff88007b1ba800 0 3628 2668 0x00000080
 ffff8800354a9cc8 0000000000000082 000000006d2c3a80 ffffffff81a01218
 00000010354a9c38 ffffffff8112eef5 ffff8800354a9fd8 ffff8800354a9fd8
 ffff88005291c9f8 000000000000f980 0000000000015740 ffff88005291c9f8
Call Trace:
 [<ffffffff8112eef5>] ? dput+0x45/0x131
 [<ffffffffa0443c77>] fuse_get_req+0xae/0x13f [fuse]
 [<ffffffff810748ab>] ? autoremove_wake_function+0x0/0x39
 [<ffffffff811ec9fc>] ? inode_has_perm+0x7a/0x90
 [<ffffffffa0444596>] fuse_do_getattr+0x3c/0x24b [fuse]
 [<ffffffff8110b643>] ? virt_to_head_page+0xe/0x2f
 [<ffffffff811ece51>] ? dentry_has_perm+0x5a/0x70
 [<ffffffffa044606b>] fuse_update_attributes+0x30/0x5f [fuse]
 [<ffffffffa04460e3>] fuse_getattr+0x49/0x4e [fuse]
 [<ffffffff8111f985>] vfs_getattr+0x48/0x66
 [<ffffffff8111f9ee>] vfs_fstatat+0x4b/0x62
 [<ffffffff8111fa60>] vfs_lstat+0x1e/0x20
 [<ffffffff8111fa81>] sys_newlstat+0x1f/0x3d
 [<ffffffff81125373>] ? path_put+0x22/0x26
 [<ffffffff81011d32>] system_call_fastpath+0x16/0x1b

[Krishna (krishna-redhat-bugs)]

Here is what I did to (temporarily?) fix the problem:

yum remove gvfs-fuse totem-nautilus totem gvfs-smb bluez nautilus gvfs-gphoto2 gvfs-archive gvfs gvfs-obexftp gnome-bluetooth pulseaudio-module-bluetooth
rm -rf ~/.gvfs
reboot
yum install gvfs-fuse totem-nautilus totem gvfs-smb bluez nautilus gvfs-gphoto2 gvfs-archive gvfs gvfs-obexftp gnome-bluetooth pulseaudio-module-bluetooth
reboot

Obviously, this is a quick fix until the problem is really solved -- hopefully in F13.

[Krishna (krishna-redhat-bugs)]

I forgot to add what I current have on my system--

totem-2.28.5-1.fc12.x86_64
pulseaudio-module-bluetooth-0.9.21-5.fc12.x86_64
nautilus-extensions-2.28.4-2.fc12.x86_64
nautilus-sendto-2.28.2-2.fc12.x86_64
gvfs-archive-1.4.3-7.fc12.x86_64
bluez-cups-4.58-1.fc12.x86_64
gnome-bluetooth-libs-2.28.6-2.fc12.x86_64
gvfs-smb-1.4.3-7.fc12.x86_64
totem-nautilus-2.28.5-1.fc12.x86_64
bluez-4.58-1.fc12.x86_64
brasero-nautilus-2.28.3-1.fc12.x86_64
bluez-libs-4.58-1.fc12.x86_64
gvfs-1.4.3-7.fc12.x86_64
gvfs-obexftp-1.4.3-7.fc12.x86_64
nautilus-2.28.4-2.fc12.x86_64
totem-pl-parser-2.28.2-1.fc12.x86_64
gvfs-gphoto2-1.4.3-7.fc12.x86_64
totem-mozplugin-2.28.5-1.fc12.x86_64
gvfs-fuse-1.4.3-7.fc12.x86_64
gnome-bluetooth-2.28.6-2.fc12.x86_64

[Tomáš (tom-redhat-bugs-1)]

(In reply to comment #92)
...
> rm -rf ~/.gvfs
...

This will reliably delete all data from your active mounts (FTP, SFTP, SMB etc.).

[Sébastien (sbastien-redhat-bugs)]

"ls -la" in the home directory also hangs on F13.

[Brian (brian-redhat-bugs)]

I can also confirm this happens on F13.

[Mehmet (mehmet-redhat-bugs)]

Confirming bug in F13 . The same after all updates .

[Harald (harald-redhat-bugs)]

ARGH - This are really the audit-rules
After comment out them fuse works again

So the following report from me is the same and has nothing to do with umask
https://bugzilla.redhat.com/show_bug.cgi?id=585302
______________________________________

[root@srv-rhsoft:~]$ cat /etc/audit/audit.rules
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320

# Feel free to add below this line. See auditctl man page
# -w /etc/passwd -p wa -k password-file
# -w /etc/group -p wa -k password-file
# -w /etc/shadow -p wa -k password-file
# -w /etc/gshadow -p wa -k password-file
# -w /root/.ssh/authorized_keys2 -p wa -k ssh-keyfile

[David (david-redhat-bugs)]

I'm not running auditd and I still get hangs on 'ls -al ~', so it's doubtful that audit's involved.

[Adam (adam-redhat-bugs)]

*** Bug 585302 has been marked as a duplicate of this bug. ***

[Mehmet (mehmet-redhat-bugs)]

(In reply to comment #99)
> I'm not running auditd and I still get hangs on 'ls -al ~', so it's doubtful
> that audit's involved.

I can confirm this too .

I disabled audit service, removed gvfs.fuse via yum but still having this problem.

After repeating these steps in F12 I wrote above, there was no problem .
But in F13 it does not help. I can view home folder a couple of times after reboot then it hangs after 5~6 minutes uptime.

[Eric (eric-redhat-bugs)]

In response to comment #101, this must be some new problem. Can you give the output of ps -ef | grep mount during the hang? Can you also run

echo t > /proc/sysrq-trigger

and then attach the resulting output in /var/log/messages?

[Josef (josef-redhat-bugs)]

Fuse still needs to be updated to take advantage of the new --no-canonicalize option in util-linux. Until fuse is updated to 2.8.3 or later this problem will continue to happen. I'm not the maintainer of fuse for Fedora so I can't do it, whoever owns the package is going to have to do it.

[Peter (peter-redhat-bugs)]

I'll update fuse shortly.

Folks, next time, please, file tickets against fuse and not against fuse-utils.

[Fedora (fedora-redhat-bugs)]

fuse-2.8.4-1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/fuse-2.8.4-1.fc12

[Fedora (fedora-redhat-bugs)]

fuse-2.8.4-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/fuse-2.8.4-1.fc11

[Fedora (fedora-redhat-bugs)]

fuse-2.8.4-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/fuse-2.8.4-1.fc13

[Peter (peter-redhat-bugs)]

Folks, please install fuse from updates-testing and leave a comment in Bodhi about whether this build fixes this issue or not.

I, personally, can't say anything about this issue because I don't use gfvs at all (although it does installed on my machines as a dependency) - at least this build doesn't break anything.

[Mehmet (mehmet-redhat-bugs)]

The update above works. Problem solved.

[Fedora (fedora-redhat-bugs)]

fuse-2.8.4-1.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.

[Fedora (fedora-redhat-bugs)]

fuse-2.8.4-1.fc11 has been pushed to the Fedora 11 stable repository. If problems still persist, please make note of it in this bug report.

[Fedora (fedora-redhat-bugs)]

fuse-2.8.4-1.fc12 has been pushed to the Fedora 12 stable repository. If problems still persist, please make note of it in this bug report.

[Dave (dave-redhat-bugs)]

Tested on fedora 12 with fuse-2.8.4-1.fc12. fusecompress still hangs the system on boot if auto mounted in fstab.

[John (john-redhat-bugs)]

(In reply to comment #111)
> fuse-2.8.4-1.fc11 has been pushed to the Fedora 11 stable repository. If
> problems still persist, please make note of it in this bug report.

Still hangs on Fedora 11, which is using util-linux-ng version 2.14.2-11.

If I understand comment #68 and comment #70 correctly, the solution requires a "--no-canonicalize" option for mount. I have no idea where in the guts of the system this would take place, but I also understand from the same comment that util-linux-ng needs to be at least version 2.17 for this to work.

Since util-linux-ng goes SO deep into the system, I am wary of any experimenting. I assume, therefore, that I will need to upgrade at least to FC12 for this to work, but I also see from comment #70 that it hasn't been ported there, either. In fact, the latest I can see for FC12 is 2.16.2-9, so perhaps this will work only with FC13?

In any case, this part of the solution seems to rest with the maintainer of util-linux-ng, who I think is Karel Zak, not with Peter Lemenkov.

I see that Peter Lemenkov asked for comments to be left in Bodhi, which will be my next effort :)

[John (john-redhat-bugs)]

(In reply to comment #108)
> Folks, please install fuse from updates-testing and leave a comment in Bodhi
> about whether this build fixes this issue or not.
>
> I, personally, can't say anything about this issue because I don't use gfvs at
> all (although it does installed on my machines as a dependency) - at least this
> build doesn't break anything.

Though this fix was pushed to FC11, it doesn't solve the problem, as util-linux-ng was never upgraded on FC11. (See comment #70)

From comment #89, I see that util-linux-ng was upgraded on FC12, so perhaps the solution is to move to FC12, but it's kinda pointless to upgrade fuse on FC11 when its companion util-linux-ng problem remains.

[Peter (peter-redhat-bugs)]

> (In reply to comment #111)

> Still hangs on Fedora 11, which is using util-linux-ng version 2.14.2-11.

I was not aware of the status of util-linux-ng in F-11. I'm afraid that it won't be upgraded at all considering that F-11 will be EOLed very soon.

Anyway, I believe that I did my part of job.

> Since util-linux-ng goes SO deep into the system, I am wary of any
> experimenting. I assume, therefore, that I will need to upgrade at least to
> FC12 for this to work, but I also see from comment #70 that it hasn't been
> ported there, either. In fact, the latest I can see for FC12 is 2.16.2-9, so
> perhaps this will work only with FC13?

Yes, I'm afraid so - in order to fix this issue util-linux-ng should be upgraded in F-12 too.

[Peter (peter-redhat-bugs)]

Ok, since now fuse-related part of work was finished, I'm switching this ticket to util-linux-ng, which should be updated in F-12 in order to completely fix this issue.

[Karel (karel-redhat-bugs)]

Hmm..., bug 577947, I see:

* Mon Apr 12 2010 Karel Zak <email address hidden> 2.16.2-9
- fix #577947 - need --no-canonicalize option for mount

bodhi - 2010-05-07 17:25:08
This update has been pushed to stable

https://admin.fedoraproject.org/updates/util-linux-ng-2.16.2-9.fc12?_csrf_token=76c49b8f4800907c7c5dc3f082bf3e5a4a9262d9

[Henrique (henrique-redhat-bugs)]

Not sure if this is related but it was the bug that came out on a google search and it seems to have similar origin, i.e. fuse.

Question, are fuse mounts supposed to survive reboots?

They did for me once. X hang on my x86_64 nightly updated F13 when trying to display a quite large image from wikipedia. Mouse worked, not keyboard. Ssh'ed from another machine, killed a couple of things to see if I could recover but eventually needed to shutdown -r.

I have pam_ssh installed and modify the pam scripts so I log in with my ssh passphrase. I log on with gdm but run don't run gnome, xinit/xsession/fvwm instead.

When I logged on again I noticed I couldn't ssh to any of the places I usually do, without entering my passphrase, like ssh-agent wasn't proper. Logged off and on a couple of times and no go. Eventually I found that I still had another system fuse mounted, just like it was before the machine was rebooted.

Unmounted that fs, logged off then on, and things are normal again. Strange and scary. Not sure I can replicate it again.

[Jon (jon-redhat-bugs)]

The underlying mount command that is hanging does not hang for users that belong to the 'video' system group.

I took users out of that group which they get put into by default, and the problem happened consistently. Once I put them back in and killed the hung mount processes, the problem no longer happens.

[Etienne Goyer (etienne-goyer-outlands)]

Peter,

Upgrading a package to a newer upstream version goes against the Stable Release Update (SRU) policy; see https://wiki.ubuntu.com/StableReleaseUpdates for all the details. Exceptions have been granted, but only in rare and very specific circumstances. Obviously, upstream software do get upgraded when a new Ubuntu release is coming out, though.

Talking about new release, for some reason FUSE has not been updated in Maverick. I guess that is because 2.8.4 has been uploaded to Debian unstable in July, after the DebianImportFreeze. In any case, maverick is past FinalFreeze, which would complicate a sync from Debian at this point.

[Etienne Goyer (etienne-goyer-outlands)]

Debian bug #585648 is apparently related to this problem, and was fixed by uploading fuse package 2.8.4-1.

[Colin Watson (cjwatson)]

I'd rather not do the full upstream update at this point for Maverick, but I'm sure we can find and backport the relevant patch.

[scm (scm)]

Here are steps to reliably reproduce this problem on lucid laptop (stock) running libfuse2 (2.8.1-1.1ubuntu2):

1. install auditd (e.g. 1.7.13-1ubuntu2)
2. add a custom rules file with any rules other than "delete all rules" (I've attached an example) to /etc/audit/audit.rules
3. reboot (it's probably possible to restart auditd + gdm + gvfsd + mounts, but rebooting is more reliable)
4. access a fuse mounted filesystem (e.g. run 'stat .gvfs' if logged in under gnome)

[Colin Watson (cjwatson)]

On second thoughts, I think the easiest fix for Maverick will in fact be a merge from Debian, despite the relatively late stage in the release. Here's the complete upstream changelog between 2.8.1 and 2.8.4:

2010-04-26 Miklos Szeredi <email address hidden>

       * Released 2.8.4

2010-04-26 Miklos Szeredi <email address hidden>

       * Fix checking for symlinks in umount from /tmp. Reported by Al
       Viro

       * Fix umounting if /tmp is a symlink. Reported by Franco Broi

2010-02-18 Miklos Szeredi <email address hidden>

       * Fix definition of FUSE_OPT_END for C++. Reported by Tim
       Bruylants

2010-02-03 Miklos Szeredi <email address hidden>

       * Fix stack alignment for clone()

2010-02-01 Miklos Szeredi <email address hidden>

       * Released 2.8.3

2010-02-01 Miklos Szeredi <email address hidden>

       * Using "--no-canonicalize" with umount(8) conflicts with the race
       fix, sinceit assumes the supplied path is absolute, while the race
       fix relies on the path being relative to the current directory.
       Reported by Tom Rindborg

2010-01-26 Miklos Szeredi <email address hidden>

       * Released 2.8.2

2010-01-21 Miklos Szeredi <email address hidden>

       * Fix race if two "fusermount -u" instances are run in parallel.
       Reported by Dan Rosenberg

       * Make sure that the path to be unmounted doesn't refer to a
       symlink

2010-01-14 Miklos Szeredi <email address hidden>

       * Fix compile error on FreeBSD. Patch by Jay Sullivan

2009-12-17 Miklos Szeredi <email address hidden>

       * Use '--no-canonicalize' option of mount(8) (available in
       util-linux-ng version 2.17 or greater) to avoid calling
       readling(2) on the newly mounted filesystem before the mount
       procedure is finished. This has caused a deadlock if "audit" was
       enabled in the kernel. Also use '--no-canonicalize' for umount to
       avoid touching the mounted filesystem.

I'm attaching a sanitised upstream diff, excluding changelogs and the autotools update. Note that we already had much of this as a security patch. It seems pretty straightforward to me, and the care taken to support legacy versions of mount should also ensure that it works properly in the initramfs.

(Lucid will be a different matter. A backport would be more appropriate there.)

[Colin Watson (cjwatson)]

 configure.in | 2
 include/fuse_common.h | 2
 include/fuse_lowlevel.h | 1
 include/fuse_opt.h | 2
 lib/Makefile.am | 2
 lib/fuse.c | 6 +-
 lib/fuse_lowlevel.c | 2
 lib/helper.c | 2
 lib/mount_util.c | 102 +++++++++++++++++++++++++++++++++++++++++-------
 util/fusermount.c | 55 ++++++++++++++++---------
 10 files changed, 134 insertions(+), 42 deletions(-)

[Launchpad Janitor (janitor)]

This bug was fixed in the package fuse - 2.8.4-1ubuntu1

---------------
fuse (2.8.4-1ubuntu1) maverick; urgency=low

  * Resynchronise with Debian (fixing hang with auditd, LP: #634554).
    Remaining changes:
    - debian/control: Add Breaks to ensure right version of udev is used.
    - Use udev rules instead of init script:
      + Add debian/45-fuse.rules: Put /dev/fuse into group fuse.
      + debian/fuse-utils.postinst: Try to load the fuse module only if it's
        still a module, remove it from /etc/modules/ anyway.
      + debian/rules, debian/fuse-utils.install: Don't install the init
        script; install the udev rule.
    - initramfs support, for booting from ntfs-3g in wubi:
      + debian/fuse-utils.initramfs-hook: Copy /sbin/mount.fuse and the fuse
        kernel module into the initramfs. Use manual_add_modules not
        force_load; fuse will be loaded automatically if necessary (it's a
        built-in in Ubuntu anyway)
      + debian/rules: Install above file into fuse-utils.
      + debian/fuse-utils.postinst: Call update-initramfs.
      + (Forwarded to Debian #505691)
    - Create libfuse2-udeb and fuse-utils-udeb. (Forwarded to Debian #505697)
    - debian/fuse-utils.install: Install ulockmgr_server.
    - debian/{rules,libfuse2.install,fuse-utils.lintian}: Move fusermount and
      ulockmgr_server to /bin and associated libraries to /lib. This allows
      mounting ntfs filesystems in /etc/fstab. (Debian #452412)
    - debian/{rules,fuse-utils.postinst}: Install fusermount with 4755
      permissions (remaining change from "Dynamic foreground user access").
    - debian/fuse-utils.postinst:
      + Don't fail if udev is running and /dev/fuse does not exist.
        (Forwarded to Debian #505685)
    - debian/fuse-utils.preinst:
      + Remove the module configuration file on upgrade if unmodified.
      + Remove old rules file if unchanged
  * Re-enable 000-Build_system_do_not_install_init_script patch.

fuse (2.8.4-1) unstable; urgency=low

  * New upstream version.
    - ACK previous non-maintainer upload (559478)
    - fixes problems with gvfs (585648)
  * Added sparc64 to supported archs (560987)

fuse (2.8.1-1.2) unstable; urgency=high

  * Non-maintainer upload by the Security Team.
  * Fixed CVE-2009-3297: race condition in fusermount (Closes: #567633)
 -- Colin Watson <email address hidden> Fri, 24 Sep 2010 12:09:38 +0100

[Peter Moody (ubuntu-hda3)]

Hey folks,

What has to happen to get this fix pushed to Lucid as well (or am I not reading the bug status correctly)?

Cheers,
peter

[Etienne Goyer (etienne-goyer-outlands)]

It's been targeted for lucid too (see the "Milestone" column). It is a different upload, so the maverick task is "Fix Released" while the lucid one is "Triaged".

[Colin Watson (cjwatson)]

I intend to backport this to Lucid, but at the moment I'm 100% occupied with Maverick release tasks so it will likely be a little over a week until I can do so.

[Peter Moody (ubuntu-hda3)]

Excellent. Thanks, Colin.

Cheers,
peter

[Colin Watson (cjwatson)]

Sorry for the delay. I've uploaded a candidate fix, awaiting review by another member of the stable team.

[Martin Pitt (pitti)]

Accepted fuse into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Peter Moody (ubuntu-hda3)]

Hey Colin,

I might be doing something wrong, but this doesn't seem to fix the problem. I've installed libfuse2 and fuse-utils from lucid-proposed per the documentation you provided and I still get hangs when mounting fuse file systems with audit rules applied.

$ dpkg -l | grep fuse
ii fuse-utils 2.8.1-1.1ubuntu2.1 Filesystem in USErspace (utilities)
ii gvfs-fuse 1.6.1-0ubuntu1build1 userspace virtual filesystem - fuse server
ii libfuse2 2.8.1-1.1ubuntu2.1 Filesystem in USErspace library

$ sudo auditctl -l
No rules

$ sshfs pmoody@localhost:/usr/local mnt

$ ls mnt/
[stuff]

$ sudo umount mnt

$ sudo auditctl -w /etc/shadow -p wa

$ sudo auditctl -l
LIST_RULES: exit,always watch=/etc/shadow perm=wa

$ sshfs pmoody@localhost:/usr/local mnt
[hang]

Cheers,
peter

[Colin Watson (cjwatson)]

OK, thanks. I'll look into this and see what I can dig up.

[Colin Watson (cjwatson)]

I appear to have screwed up. The patch isn't even applied, and when I try to apply it it fails hideously due to a conflict with a security patch. I'm not quite sure what I was thinking here.

I'll beat it into a sensible state and try again ...

[Peter Moody (ubuntu-hda3)]

Thanks. I look forward to testing the update.

Cheers,
peter

On Tue, Dec 7, 2010 at 9:29 AM, Colin Watson <email address hidden> wrote:

> I appear to have screwed up. The patch isn't even applied, and when I
> try to apply it it fails hideously due to a conflict with a security
> patch. I'm not quite sure what I was thinking here.
>
> I'll beat it into a sensible state and try again ...
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/634554
>
> Title:
> fuse mounts hang on xattr retrieval with auditd
>
> Status in “fuse” package in Ubuntu:
> Fix Released
> Status in “fuse” source package in Lucid:
> Fix Committed
> Status in “fuse” source package in Maverick:
> Fix Released
> Status in “fuse” package in Debian:
> Fix Released
> Status in “fuse” package in Fedora:
> Unknown
>
> Bug description:
> Impact: If audit is enabled in the kernel, mounting fuse filesystems
> hangs.
> Development branch (and maverick): Addressed by upgrade to upstream fuse >=
> 2.8.3.
> Patch: https://code.launchpad.net/~cjwatson/ubuntu/lucid/fuse/audit-hang(backported from upstream - it's not particularly short, but there were
> three intertwined fixes and this was the best I could do)
> TEST CASE: Install auditd and configure a rule, then try to mount a fuse
> filesystem.
> Regression potential: I'd recommend generally playing around with fuse and
> making sure it still works. Running the udisks test suite might be a good
> idea.
>
> Original report follows:
>
> Binary package hint: libfuse2
>
> I'm being struck by the symptoms reported in this bug:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=493565
>
> Namely, when auditd has any rules configured, I can't mount any fuse
> file-systems. Comment 61 seems to have a pretty solid description of what's
> happening underneath and the solution appears to be upgrading to a libfuse >
> 2.8.3. Only 2.8.1 (http://packages.ubuntu.com/lucid/libfuse2) appears to
> be available currently for lucid.
>
> Cheers,
> peter
>
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/fuse/+bug/634554/+subscribe
>

[Marc Deslauriers (mdeslaur)]

I'm actually preparing a fuse security update for lucid that probably will address this issue. I'll let you know when packages are available for testing.

[Etienne Goyer (etienne-goyer-outlands)]

Following an IRC discussion with Marc, I tested the fuse packages from https://launchpad.net/~ubuntu-security-proposed/+archive/ppa. They fixed the hang when using auditd for me. I also tested around with various GVFS backends (ssh, FTP, SMB), and they worked as expected. +1 from me, thanks Marc for the fix.

[Peter Moody (ubuntu-hda3)]

Confirmed. thanks for the fix!

OOC, how long does a package stay in proposed?

Cheers,
peter

On Tue, Dec 21, 2010 at 11:58 AM, Etienne Goyer <<email address hidden>
> wrote:

> Following an IRC discussion with Marc, I tested the fuse packages from
> https://launchpad.net/~ubuntu-security-proposed/+archive/ppa. They
> fixed the hang when using auditd for me. I also tested around with
> various GVFS backends (ssh, FTP, SMB), and they worked as expected. +1
> from me, thanks Marc for the fix.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/634554
>
> Title:
> fuse mounts hang on xattr retrieval with auditd
>
> Status in “fuse” package in Ubuntu:
> Fix Released
> Status in “fuse” source package in Lucid:
> Fix Committed
> Status in “fuse” source package in Maverick:
> Fix Released
> Status in “fuse” package in Debian:
> Fix Released
> Status in “fuse” package in Fedora:
> Unknown
>
> Bug description:
> Impact: If audit is enabled in the kernel, mounting fuse filesystems
> hangs.
> Development branch (and maverick): Addressed by upgrade to upstream fuse >=
> 2.8.3.
> Patch: https://code.launchpad.net/~cjwatson/ubuntu/lucid/fuse/audit-hang(backported from upstream - it's not particularly short, but there were
> three intertwined fixes and this was the best I could do)
> TEST CASE: Install auditd and configure a rule, then try to mount a fuse
> filesystem.
> Regression potential: I'd recommend generally playing around with fuse and
> making sure it still works. Running the udisks test suite might be a good
> idea.
>
> Original report follows:
>
> Binary package hint: libfuse2
>
> I'm being struck by the symptoms reported in this bug:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=493565
>
> Namely, when auditd has any rules configured, I can't mount any fuse
> file-systems. Comment 61 seems to have a pretty solid description of what's
> happening underneath and the solution appears to be upgrading to a libfuse >
> 2.8.3. Only 2.8.1 (http://packages.ubuntu.com/lucid/libfuse2) appears to
> be available currently for lucid.
>
> Cheers,
> peter
>
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/fuse/+bug/634554/+subscribe
>

[Marc Deslauriers (mdeslaur)]

They will be released as a security update after the holidays once they go through proper QA regression testing.

[Etienne Goyer (etienne-goyer-outlands)]

Unless things changed recently, package stay in -proposed for seven days before being promoted to -updates. That being said, this does not apply for packages coming out of that specific PPA. Marc can certainly clarify the lifecycle of this update.

[Launchpad Janitor (janitor)]

This bug was fixed in the package fuse - 2.8.1-1.1ubuntu2.1

---------------
fuse (2.8.1-1.1ubuntu2.1) lucid-proposed; urgency=low

  * Use 'mount --no-canonicalize' to avoid deadlocks if audit is enabled in
    the kernel; also fix race if two 'fusermount -u' processes are run in
    parallel (fixes by Miklos Szeredi, backported from upstream;
    LP: #634554).
 -- Colin Watson <email address hidden> Tue, 09 Nov 2010 20:32:24 +0000

[scm (scm)]

For what it's worth (now that the fix is already released) I can confirm this worked for me in proposed and now that it's released, seems to have fixed my issues.

[Richard (richard-redhat-bugs)]

For those following this, I think this bug has reappeared
in Fedora 17. The symptoms are extremely similar (even
the FUSE "hello" example hangs), and disabling selinux fixes
it (but *not* just setting selinux to permissive).

https://bugzilla.redhat.com/show_bug.cgi?id=812798