try to access a .Trash-$USER directory on autofs mounts

[tgelter]

Binary package hint: nautilus

root@guapuraT61:~# /etc/init.d/autofs start
Starting automounter: done.

==> syslog <==
Apr 1 14:50:15 guapuraT61 kernel: [18916.265351] SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
Apr 1 14:50:16 guapuraT61 automount[10582]: >> /sbin/showmount: can't get address for .Trash
Apr 1 14:50:16 guapuraT61 automount[10582]: lookup(program): lookup for .Trash failed
Apr 1 14:50:16 guapuraT61 automount[10582]: failed to mount /net/.Trash
Apr 1 14:50:16 guapuraT61 automount[10589]: >> /sbin/showmount: can't get address for .Trash-500
Apr 1 14:50:16 guapuraT61 automount[10589]: lookup(program): lookup for .Trash-500 failed
Apr 1 14:50:16 guapuraT61 automount[10589]: failed to mount /net/.Trash-500
Apr 1 14:50:16 guapuraT61 automount[10597]: >> /sbin/showmount: can't get address for .Trash
Apr 1 14:50:16 guapuraT61 automount[10597]: lookup(program): lookup for .Trash failed
Apr 1 14:50:16 guapuraT61 automount[10597]: failed to mount /net/.Trash
Apr 1 14:50:16 guapuraT61 automount[10603]: >> /sbin/showmount: can't get address for .Trash-500
Apr 1 14:50:16 guapuraT61 automount[10603]: lookup(program): lookup for .Trash-500 failed
Apr 1 14:50:16 guapuraT61 automount[10603]: failed to mount /net/.Trash-500

Nautilus attempts to mount and access a .Trash-$USER directory under the autofs mount "/net" and consequently fails and reports a bunch of errors to /var/log/syslog.
There should be an ignore statement somewhere so that nautilus will ignore/not create the .Trash-$USER directory on read-only directories such as nfs shares that have been exported read-only and consequently *not* attempt to use the directory.

Additional information:

root@guapuraT61:~# lsb_release -rd
Description: Ubuntu hardy (development branch)
Release: 8.04

root@guapuraT61:~# apt-cache policy nautilus
nautilus:
  Installed: 1:2.22.1-0ubuntu1
  Candidate: 1:2.22.1-0ubuntu1
  Version table:
 *** 1:2.22.1-0ubuntu1 0
        500 http://us.archive.ubuntu.com hardy/main Packages
        100 /var/lib/dpkg/status

[Sebastien Bacher (seb128)]

Thank you for your bug report. The log seems to indicate it tries to open the directory and not to create it, no? That doesn't seem to be a bug

[tgelter (timothy-gelter)]

That is correct. I was a bit quick in my explanation and didn't fully explain myself. Nautilus assumes that the directory has been created and then attempts to access it. If it wasn't possible to create the directory because the mounted directory is read-only, Nautilus shouldn't assume that it exists as it is doing in this instance.
If the .Trash-$USER directory couldn't be created, it shouldn't attempt to mount it. I still feel that this is a bug. Maybe it's as simple as changing the log message to indicate the actual problem rather than just saying that it couldn't be mounted, etc.

[tgelter (timothy-gelter)]

I went ahead and edited the description/tags of this bug report to match the actual issue.
If this still isn't an issue, please comment and I'll ignore this bug report.

[Sebastien Bacher (seb128)]

Thanks for your bug report. This bug has been reported to the developers of the software. You can track it and make comments here: http://bugzilla.gnome.org/show_bug.cgi?id=525779

[Brent Nelson (brent-phys-deactivatedaccount)]

Please also see duplicate bug #210586. gvfsd-trash continuously tries to access the automount points (and keeps spawning and spawning, although it does allow old processes to die before spawning more, so it doesn't consume all available memory). The result is that, as soon as someone logs in, the machine will automount every NFS filesystem in our department (~145, in our case) and keep the mounts active (some may expire, only to be remounted shortly thereafter).

This results in a mount storm (in the past, on slower processors, such storms have rendered machines unusable for ~10-20 minutes in our environment), and it really defeats much of the purpose of automounting if all mounts are kept active.

[Sebastien Bacher (seb128)]

could somebody describe an easy way to configure automount?

[tgelter (timothy-gelter)]

A very simple case would be something like this:

/etc/auto.master
******************************
/mountpoint /etc/auto.fileDescribingMount

/etc/auto.fileDescribingMount
******************************
directoryToMountTo -opts /dir/to/mount

Working example:
/etc/auto.master
******************************
/ftp /etc/auto.ftp

/etc/auto.ftp
******************************
myDir -ro,soft ftp.example.com:/pub/myDir

[tgelter (timothy-gelter)]

so the mounts would be:
/dir/to/mount/ on /mountpoint/directoryToMountTo type nfs (opts)
ftp.example.com:/pub/myDir on /ftp/myDir type ftp (ro,soft)

[Brent Nelson (brent-phys-deactivatedaccount)]

I tried again today, with gvfs 0.2.2svn20080403-0ubuntu1. I found that the mount storm is currently pretty slow, but the gvfsd-trash processes seem to be accumulating endlessly. After ~45 minutes, it has mounted everything in our department, and I have over 3000 gvfsd-trash processes (and 1.1GB consumed on a freshly booted machine with a completely idle logged-in Gnome session).

[kenjo (ken-kenjo)]

I also have this problem. the nfs server log gets flooded by mount request for .TrashXXXXX. in my case it's the home directories that is auto mounted and nautilus/gvfs tries to create directories in /home that is never going to work

Something needs to learn to detect that /home/user and /home is not located on the same filesystem. but why look for a .Trash directory there in the firstplace ??

[Sebastien Bacher (seb128)]

there is one such directory on every mount that's why it's trying to use those

[Sebastian Kapfer (caci)]

On a medium-sized installation with a few thousand automounter entries, the resulting mount storm makes the system unusable. It is constantly spawning new automount processes (almost like a fork bomb) and keeps haldaemon at 99% CPU. Not sure how gvfs triggers this, a simple stat /home/.Trash (where /home is autofs) sure doesn't trigger a mount storm.

This is happening from time to time; the bug was also in feisty and seems to be more likely to occurr in hardy.

[guenthert (guenthert)]

> This is happening from time to time; the bug was also in feisty and seems to be more likely to occurr in hardy.
Yes, it is.

> there is one such directory on every mount that's why it's trying to use those
That's simply not the case and this erroneous assumption is the bug. Perhaps every mount on a stand-alone Ubuntu host using Gnome desktop has a .Trash directory, but that is (thankfully) certainly not the case in a heterogeneous network.

[Sebastien Bacher (seb128)]

> this erroneous assumption is the bug

it's not an assumption but it needs to look on the partition to know if there a directory there

[Brent Nelson (brent-phys-deactivatedaccount)]

On Tue, 29 Apr 2008, Sebastien Bacher wrote:

>> this erroneous assumption is the bug
>
> it's not an assumption but it needs to look on the partition to know if
> there a directory there
>

It may need to look at THE partition (i.e., the partition on which a file
is actually being deleted, although for a home directory that would still
be erroneous; it needs to look in the home directory, not the mountpoint
of the home partition); it certainly does not need to look at EVERY
partition the system can access, even when no deletion has actually
occurred.

This is quite unworkable on anything but a standalone desktop PC. We had
to disable the trash demon (chmod a-x) before we could deploy Hardy in our
environment. Our environment is rather typical of managed Linux/UNIX
systems.

Thanks,

Brent

[Sebastien Bacher (seb128)]

nobody denies that's an issue and it has been milestoned and is set as a high priority ubuntu bug, the reason why it access all the partitions is that the trash location is listing the different partition trash content so it's looking on every partition to know if there is something to list there

[Sebastian Kapfer (caci)]

I'm starting to believe we're looking at four different issues here

a) gvfsdtrash looking into every mountpoint for .Trash directories. This should not be, as such, a problem.

b) gvfsdtrash's accumulating endlessly and keeping automounted NFS busy. That is a bug.

c) gvfsdtrash not knowing that filesystems of type "autofs" are unlikely to contain .Trash directories (or need one). This should not be, as such, a problem, but creates ugly spam in the logs.

d) Something in hardy's GNOME stack confusing the hell of out the automouter so it goes through the automounter map and mounts everything in sight. This kills the machine if there is a large number of automountable directories. I'm not sure if anything in gvfs even knows which directories can be automounted, so it could be that this is in fact an automounter bug.

[Steve Langasek (vorlon)]

Looking into every mount point for .Trash directories is not a bug. Looking is the only way to see if there are contents there that should be displayed in the system trash can! (As Sebastien notes, the trash can is shown as the union of all the per-filesystem trashcans, so the only way to ensure all trash contents are displayed correctly is by checking each mount point.)

b) and c) are certainly bugs, I agree. unfortunately, b) may be an inherent conflict between how gvfsd-trash and autofs work; gvfsd-trash uses inotify to monitor the trash can, which means the reference count on the mount point never drops to zero and autofs doesn't unmount. c) should be straightforward to fix.

d) is very odd indeed. Does running "ls /home" have this same effect? The last time I used autofs this was not the case, so I wonder what's happening to cause this.

[Paul Smith (psmith-gnu)]

I'm seeing this too, and something about it is causing my system to hang. Basically, when I come in the morning I can't do hardly anything, including reboot. Investigation has shown that it's because any attempt to read the /proc/mounts file causes a permanent hang on the process (can't even kill -9 etc.). Lots and lots of programs, even ones like "ls", try to read this file. I have to power-cycle the system to recover. My system load is above 7, even though my CPU usage is negligible (a sure sign that processes are hung in the kernel).

Last night I started a script that ran date, then cat /proc/mounts, then sleep 15 and left it running all night. I got the hang at 22:51 and looking through my logs, sure enough right at that exact time I saw a slew of these .Trash access errors show up in my /var/log/syslog file. I'm going to continue to test this theory to see if it was just coincidence or not.

[Paul Smith (psmith-gnu)]

Just a comment on point (c) above: I don't think this is quite as innocuous as you suppose. While looking at the root of a normal automount point for a non-existent directory (or two, in this case) isn't so bad (except for the annoying log spam), there are various types of maps which take any random set of characters and try to do something with them.

For example, if you try to access /net/.Trash then (on systems where it's enabled) a script /etc/auto.net is invoked with an argument of ".Trash" as the putative hostname. This leads to attempts to do DNS lookups, run showmount, and other dodgy operations which, on a well-behaved system, should still not be a major concern but are significantly more costly than a simple directory lookup failure.

Also, some maps, commonly /home maps, avoid very long lists in their maps by using map key substitution; so there's one entry containing a "&" and whatever text is used for that, a mount request is made for that (for example in /home/psmith "&" would be "psmith" and the mount request would be made with that--if you try /home/.Trash then & is ".Trash" and a mount request is made for that). This can also cause problems, most especially for the server which is constantly rejecting all these invalid mount requests from all the hosts on the network. It can also cause delays if the server or network is busy.

I do agree this should be fixed: I can't think of any situation where it would be legitimate and reasonable to have a .Trash directory in an automount map mount point (mount point of type autofs). Everything in an autofs mount point is, by definition, a partition mount itself which means that any .Trash entry would have to be a seperately mounted partition... which means that the real trash would be in .../.Trash/.Trash --- right?

It seems reasonable to me to actually make the fs types that gvfsd-trash looks in be configurable, in some way. This is a larger change of course, but it's the common way to avoid hardcoding this stuff (for things like slocate etc.)

If someone has a patch they would like me to test, I can do that.

[Walter Tautz (wtautz)]

Just to comment further, I have a remote home directory that I like to be able to access from
my Ubuntu box. In particular I have an sshfs+autofs setup:

/etc/auto.master:
/autofs/sshfs/cs_student/home auto_home_sshfs -nobrowse

where auto_home_sshfs is a shell script in /etc :

#!/bin/bash
echo "`date`: $@" >>/tmp/`basename $0`.$$
# Shell script that excepts one argument, namely the userid
case $1 in
 .Trash*)
  exit 1;;
 *)
  echo "-fstype=fuse,rw,nodev,nonempty,noatime,allow_other,max_read=65536 :sshfs\#$1@server\:"
esac

 ssh-askpass (a gnome Gui tool that gets launched by something gvfsd?) kept prompting
me for .Trash@server and also .<email address hidden> (1000 is my uid on the local box)
passwords.

Alas it would be nice not to be prompted for a password since I have setup a private key login (it works with sshfs
directly invoked from the command line)

[Paul Smith (psmith-gnu)]

Here is a bog-simple patch I created for gvfs-0.2.4 that skips any root directory with a filesystem type of "autofs".

To my mind this is just a workaround: the real answer is that the list of filesystem types to skip should be configurable, I presume via an entry in gconf. However, that involves a LOT more work in code I'm not familiar with and this fix is easy. This has been working for me for the day so far: no messages in the log file.

I really wanted to do this because I've been seeing a problem where, every few days, my system gets sluggish, NFS mounts hang, FireFox downloads hang, etc. I don't know what's going on (nothing much in the logs) but the one thing all these situations have in common is when I do "ps" I see 25-30 instances of gvfsd-trash running instead of one. The only thing concerning in the logs are these .Trash references. I have no other proof that these things are related but I figured I'd fix this one and we'll see if it helps.

[Etienne Goyer (etienne-goyer-outlands)]

I built a gvfs package for hardy using the patch Paul Smith posted above. You can get it from:

    https://edge.launchpad.net/~etienne-goyer-outlands/+archive

I am fairly certain the patch is correct, as there is no point in actually polling for a trash in an autofs base mountpoint anyway. File system mounted by automount actually get their own mtab entry, with their own file system type (by default, NFS), which are treated just like any other file system mounted manually or through /etc/fstab. automount base mountpoint are a special case, and should be handled as such. At least, that is how I understand it!

Sebastien/Steve: if you need a testbed, ping me out-of-band.

[Paul Smith (psmith-gnu)]

Thanks Etienne. Someday I'm going to learn how to use Launchpad PPA!

As for the fix, I'll say this: first, the concept behind the patch is unquestionably 100% correct. As you point out, it's fundamentally wrong to look for .Trash directories in automount points, because they aren't "real" directories. It's like trying to look for /proc/.Trash or /sys/.Trash or something... except worse because this lookup actually does more than just fail with ENOENT.

Second, I'm 99% sure that the patch does do what I expected it to do, and that it works properly. I'm definitely no expert in glib but this change seems very straightforward.

Third, as I mentioned above I'm confident that this is not the most elegant way to solve the problem. However, it may be worthwhile to keep this as an Ubuntu patch, even if upstream Gnome folks don't want to apply it there, until they work out and implement the more elegant solution.

[Sebastien Bacher (seb128)]

the new stable version has been uploaded as an hardy update candidate now

[Paul Smith (psmith-gnu)]

Hi Sebastien; not sure what this means, to be uploaded as a hardy update candidate... I haven't seen anything come through in my -proposed repository. Or is this a prelude to that? I guess someone will update the bug if/when the fix migrates into the repositories?

Thanks.

[Sebastien Bacher (seb128)]

I've uploaded it but the updates are frozen for 8.04.1 right now, it should be accepted in a few days normally

[Etienne Goyer (etienne-goyer-outlands)]

Looking at the ChangeLog of GVFS 0.2.5, there is mention of this bug being fixed. A casual inspection of the code confirm it. Sebastien, did a patch to fix this bug was added to the package you uploaded?

[Etienne Goyer (etienne-goyer-outlands)]

Above comment should read "..., there is *no* mention of this bug being fixed...". Sorry for the confusion!

[Sebastien Bacher (seb128)]

the patch attached to this bug has been added to the 0.2.5 update

[Martin Pitt (pitti)]

Accepted into -proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Etienne Goyer (etienne-goyer-outlands)]

Did a very quick check, and it seem to work. The Trash behave as expected: show full when something is deleted in one of the autofs mount, we can empty trash just fine and we do not get the error in the log anymore. Looks good to me!

[Etienne Goyer (etienne-goyer-outlands)]

I have deleted the patched 2.4.0 package in my PPA, as it been made obsolete by the new 2.5.0 upstream in hardy-proposed.

[Paul Smith (psmith-gnu)]

I can also verify that this fixes the problem of extra Trash messages.

I also have to say that I haven't seen the hang issue since I applied my changes, either. Don't know if it's a coincidence, but I'm happy!

[Jon Reeves (jonreeves)]

I've just tried it and the Trash messages have gone but the machine still hangs when accessing automounts

Nautilus is maxing the CPU - killing this brings it back until I go into the automount directories again

Any help greatly appreciated

[Paul Smith (psmith-gnu)]

If the messages have stopped appearing in your logs, then the problem you're having is clearly not related to this bug, since this bug is about autofs .Trash requests. Bug comments are not really the best way to ask for help, unless it's about the bug in question. You should either file a bug against the package which appears to be at fault here, which would be either Nautilus or autofs (do you get the same behavior when you try to access the automount from the command line? That should tell you which package is likely at fault), or else maybe ask on the Ubuntu user forums.

When you do find the right place to ask, be sure to give more information about exactly which versions you're using, how your automount maps are distributed, what they contain (for example the map that contains the directory you're trying to access via Nautilus), whether it happens every single time or only when the filesystem is not mounted already (or only if you try to re-access the filesystem after automount has unmounted it), etc.

[Martin Pitt (pitti)]

Copied to hardy-updates.

[Launchpad Janitor (janitor)]

This bug was fixed in the package gvfs - 0.99.3-0ubuntu2

---------------
gvfs (0.99.3-0ubuntu2) intrepid; urgency=low

  * debian/patches/91_no_autofs_trashs.patch:
    - don't look for trash on autofs mounts, change from Paul Smith on launchpad
      (lp: #210468)

 -- Sebastien Bacher <email address hidden> Tue, 29 Jul 2008 21:26:06 +0200

[Guillaume F (marsguo)]

Hi, I know this is an old bug and that it's marked as fixed, but I've been having exactly the same problem lately on Wily. Is there something i can do?