lxcfs startup failues because fuse directory not empty

[Jon Watte]

During start-up, I get about 8 complaints that LXCFS is starting and stopping, until it goes red and says it failed.

The lxcfs.service unit journal contains a number of these:

Mar 18 21:12:36 ripper systemd[1]: lxcfs.service: Unit entered failed state.
Mar 18 21:12:36 ripper systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 18 21:12:36 ripper systemd[1]: lxcfs.service: Service hold-off time over, scheduling restart.
Mar 18 21:12:36 ripper systemd[1]: Stopped FUSE filesystem for LXC.
Mar 18 21:12:36 ripper systemd[1]: Started FUSE filesystem for LXC.
Mar 18 21:12:37 ripper lxcfs[1539]: hierarchies:
Mar 18 21:12:37 ripper lxcfs[1539]: 0: fd: 5: hugetlb
Mar 18 21:12:37 ripper lxcfs[1539]: 1: fd: 6: cpu,cpuacct
Mar 18 21:12:37 ripper lxcfs[1539]: 2: fd: 7: cpuset
Mar 18 21:12:37 ripper lxcfs[1539]: 3: fd: 8: blkio
Mar 18 21:12:37 ripper lxcfs[1539]: 4: fd: 9: rdma
Mar 18 21:12:37 ripper lxcfs[1539]: 5: fd: 10: pids
Mar 18 21:12:37 ripper lxcfs[1539]: 6: fd: 11: memory
Mar 18 21:12:37 ripper lxcfs[1539]: 7: fd: 12: freezer
Mar 18 21:12:37 ripper lxcfs[1539]: 8: fd: 13: perf_event
Mar 18 21:12:37 ripper lxcfs[1539]: 9: fd: 14: net_cls,net_prio
Mar 18 21:12:37 ripper lxcfs[1539]: 10: fd: 15: devices
Mar 18 21:12:37 ripper lxcfs[1539]: 11: fd: 16: name=systemd
Mar 18 21:12:37 ripper lxcfs[1539]: 12: fd: 17: unified
Mar 18 21:12:37 ripper lxcfs[1539]: fuse: mountpoint is not empty
Mar 18 21:12:37 ripper lxcfs[1539]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Mar 18 21:12:37 ripper systemd[1]: lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Mar 18 21:12:37 ripper fusermount[1580]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
Mar 18 21:12:37 ripper systemd[1]: lxcfs.service: Unit entered failed state.

Looking at /var/lib/lxcfs, it does, in fact, contain two directories "cgroup" and "proc"
So, either:
1) This should only start once, but it's started multiple times, or
2) This directory should be empty but the script fails to make it so

ls -l /var/lib/lxcfs shows:

[21:26] jwatte@ripper:~$ ls -la /var/lib/lxcfs/
total 16
drwxr-xr-x 4 root root 4096 Aug 26 2017 ./
drwxr-xr-x 69 root root 4096 Mar 18 21:10 ../
drwxr-xr-x 13 root root 4096 Aug 26 2017 cgroup/
dr-xr-xr-x 2 root root 4096 Aug 26 2017 proc/

So, something somewhere made this directory non-empty, and now it fails every time on start. (I sure didn't put anything here!) August 26 matches when I first installed this system, at the time running 17.04.

I recommend making the startup script simply nuke the contents of this directory.

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: lxcfs 2.0.8-0ubuntu1~17.10.2
ProcVersionSignature: Ubuntu 4.13.0-37.42-generic 4.13.13
Uname: Linux 4.13.0-37-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.7-0ubuntu3.7
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Sun Mar 18 21:23:52 2018
InstallationDate: Installed on 2017-08-26 (204 days ago)
InstallationMedia: Ubuntu-Server 17.04 "Zesty Zapus" - Release amd64 (20170412)
SourcePackage: lxcfs
UpgradeStatus: Upgraded to artful on 2018-02-05 (41 days ago)

[Andreas Hasenack (ahasenack)]

Thanks for filing this bug in Ubuntu.

I don't think it's safe for the service to just simply nuke the contents of /var/lib/lxcfs prior to mounting/starting.

It does sound like a local configuration problem, or maybe a bug that existed in 17.04 (which is what you first installed). It will be hard to reproduce now if it depends on 17.04, as that release is end-of-life.

In your case, since the contents are cgroup and proc with the fuse system NOT MOUNTED (please make sure), you can delete those and then restart the service.

Something like this:
sudo systemctl stop lxcfs.service

# confirm with mount that it is unmounted
mount -t fuse.lxcfs

# delete or move away the contents of /var/lib/lxcfs, your choice

# start the service
sudo systemctl start lxcfs.service

# confirm it's mounted
mount -t fuse.lxcfs

If you come across a way to reproduce the problem from the beginning, like a fresh install, or an upgrade from a previous release of ubuntu, then please update this bug. In the meantime, I'll mark it as incomplete.

[Launchpad Janitor (janitor)]

[Expired for lxcfs (Ubuntu) because there has been no activity for 60 days.]