mountall doesn't treat glusterfs correctly

[Bruno MACADRE]

I've two servers replicating data by Gluster every servers are clients too.

The problem is that the glusterfs was never mounted at boot 'cause mountall doesn't call correct mount command and doesn't treat gluster filesystem as remote filesystem.

Some informations :
# lsb_realease -rd
Description: Ubuntu 13.04
Release: 13.04

# apt-cache policy mountall
mountall:
  Installed : 2.48build1
  Candidate : 2.48build1
 Table de version :
 *** 2.48build1 0
        500 http://fr.archive.ubuntu.com/ubuntu/ raring/main amd64 Packages
        100 /var/lib/dpkg/status

# mountall --version
mountall 2.44

Entry in the fstab :
192.162.0.1:test-vol /srv/conf glusterfs defaults,_netdev 0 0

After boot, this entry wasn't mounted. But if I try :
# mount -t glusterfs 192.162.0.1:test-vol /srv/test-vol
It works perfectly

I put a '--verbose' into /etc/init/mountall.conf file to see what happened. I obtain this :

/ est local
/proc est virtual
/sys est virtual
/sys/fs/cgroup est virtual
/sys/fs/fuse/connections est virtual
/sys/kernel/debug est virtual
/sys/kernel/security est virtual
/dev est virtual
/dev/pts est virtual
/tmp est local
/run est virtual
/run/lock est virtual
/run/shm est virtual
/run/user est virtual
/srv est local
/var est local
UUID=ac001a41-9a55-49d6-86c8-1e9e8df41054 est swap
/mnt/shared-conf est local
/srv/conf est nowait <== Why not remote ?
mounting event sent for /sys/fs/cgroup
mounting event sent for /sys/fs/fuse/connections
mounting event sent for /sys/kernel/debug
mounting event sent for /sys/kernel/security
mounting event sent for /run/lock
mounting event sent for /run/shm
mounting event sent for /run/user
...
mounting event handled for /srv/conf
montage de /srv/conf
mounted event handled for /srv/conf
local 5/5 remote 0/0 virtual 12/12 swap 1/1
                                  ^

And no /srv/conf mounted...

With this command I've found more information :
# mountall --verbose
...
Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
  or: glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
Try `glusterfs --help' or `glusterfs --usage' for more information.
mount /srv/conf [11156] s'est terminé normalement
...

We can see that the call of mount was not correct (see Usage).

If I go deeper :
# mountall --verbose --debug
...
run_mount: mtab /srv/conf
spawn: mount -f -t fuse.glusterfs -o defaults,_netdev 10.130.163.253:volume-conf /srv/conf
spawn: mount /srv/conf [11339]
Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
  or: glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
Try `glusterfs --help' or `glusterfs --usage' for more information.
mount /srv/conf [11339] s'est terminé normalement
...

When I try the mount mentioned above I've got the same behaviour (send Usage) 'cause fuse.glusterfs wasn't recognized. If I put glusterfs instead the mount works as expected.

More, since mountall doesn't treat glusterfs as remote filesystem, it try to mount it even if local filsystems aren't mounted yet. For example, if /var wasn't cleanly unmounted, it will be checked at boot and mountall try to mount glusterfs during check. This cause some troubles because glusterfsd need acces to /var/log which is not mounted yet...

[Bruno MACADRE (bruno-macadre)]

I disagree this duplicate (I remove them) 'cause this problem is pretty different than bug #1103047.

In my bug I've found that mountall generate bad mount commandline for glusterfs (see the Usage in mountall --verbose).

This bug is a CRITICAL bug 'cause my servers doesn't start anymore....

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paul Boven (p-boven)]

Running into the same issue here, I can't get glusterfs to mount at boot.

System: Ubuntu 13.04

Output from mountall --verbose:

mount /export/brick0 [2508] exited normally
Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
  or: glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
Try `glusterfs --help' or `glusterfs --usage' for more information.
mount /gluster [2509] exited normally

The 'exited normally' seems wrong, too.

Output from mountall --verbose --debug:
run_mount: mtab /gluster
spawn: mount -f -t fuse.glusterfs -o defaults localhost:/gv0 /gluster
spawn: mount /gluster [2599]
Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
  or: glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
Try `glusterfs --help' or `glusterfs --usage' for more information.
mount /gluster [2599] exited normally

My fstab entry:
localhost:/gv0 /gluster glusterfs defaults,nobootwait 0 0

It seems that mountall tries to emit "mount -f -t fuse.glusterfs", which fails.
mount -f -t glusterfs (with the same arguments) does work, but I can find no documentation on mountall and where it could
have decided to prepend the 'fuse' part.

Note that I have /sbin/mount.glusterfs, which is not part of Ubuntu but from a PPA.

[Steve Langasek (vorlon)]

On Thu, Sep 26, 2013 at 06:35:14PM -0000, Paul Boven wrote:
> Running into the same issue here, I can't get glusterfs to mount at
> boot.

> System: Ubuntu 13.04

> Output from mountall --verbose:

> mount /export/brick0 [2508] exited normally
> Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
> or: glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
> Try `glusterfs --help' or `glusterfs --usage' for more information.
> mount /gluster [2509] exited normally

> The 'exited normally' seems wrong, too.

These are all bugs in the glusterfs mount helper, not in mountall. It does
not implement the expected interfaces required for mount.$fstype helpers.

> Output from mountall --verbose --debug:
> run_mount: mtab /gluster
> spawn: mount -f -t fuse.glusterfs -o defaults localhost:/gv0 /gluster
> spawn: mount /gluster [2599]
> Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
> or: glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
> Try `glusterfs --help' or `glusterfs --usage' for more information.
> mount /gluster [2599] exited normally

> My fstab entry:
> localhost:/gv0 /gluster glusterfs defaults,nobootwait 0 0

> It seems that mountall tries to emit "mount -f -t fuse.glusterfs", which fails.

No, that command almost certainly succeeds, because all it's doing is
updating the /etc/mtab to match the list of filesystems that are already
mounted. Something else has mounted /gluster for you, according to
/proc/mounts, and mountall is making sure /etc/mtab is kept up to date.

The failing call is the *second* one, where again, mountall is trying to
mount /gluster according to the fstab.

You can reproduce this failure by running the following from the
commandline:

  mount -t glusterfs -o defaults localhost:/gv0 /gluster

[Louis Zuckerman (semiosis)]

Steve,

Where can I find more information about "the expected interfaces required for mount.$fstype helpers?"

I will work with the upstream project to get that resolved.

Thanks!

[Steve Langasek (vorlon)]

> Where can I find more information about "the expected interfaces
> required for mount.$fstype helpers?"

Effectively, in the mount(8) manpage. Anything that is a valid commandline for the mount command should also be a valid commandline for the mount helper. This includes arbitrary ordering of the arguments (testing with 13.10, glusterfs complains if the -o$options are listed before the device name), and making sure that any options which are listed as "generic" options to mount don't cause errors if passed, even if they might not have meaning for glusterfs.

[Louis Zuckerman (semiosis)]

I am skeptical of this mountall diagnosis. I spoke with Paul Boven on IRC and my diagnosis of his problem is that the mount was tried before the glusterfs service was completely ready -- which was a known bug with my PPA packages he uses.

I finally (last night) fixed that bug, for good as far as I can tell. If anyone can try reproducing this bug with my latest packages I would really appreciate your feedback.

The PPA: https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.4/+packages

Furthermore, in regards to the mountall diagnosis, if that were really the problem then I believe it would be affecting all glusterfs versions on Ubuntu going back years, which is definitely not the case. I have been battling mount on boot issues for as long as I have been maintaining ubuntu packages of glusterfs (two years) and never run into an issue with command line arguments between mountall & glusterfs -- for whatever that's worth.

Lastly, my PPA (linked above) is the official upstream distribution channel for glusterfs ubuntu packages. I'd like to work with the distro community to merge my changes into the universe packages. Please get in touch with me if you have any interest in this, I'd really like to hear from you.

Thanks!

[J. Pablo Fernández (pupeno)]

I think I might be experiencing this bug or something similar, with the 3.5 packages on Ubuntu 12.04. GlusterFS volumes are tried to be mounted even before GlusterFS is run. I posted a lot of details about the issue in a serverfault question: http://serverfault.com/questions/604860/glusterfs-is-failing-to-mount-on-boot