Too many levels of symbolic links /proc/sys/fs/binfmt_misc

[Jay R. Wren]

At first I thought this was an LXD issue, but it turns out the issue is in systemd

https://github.com/lxc/lxd/issues/1727#issuecomment-194416558

ls -l /proc/sys/fs/binfmt_misc fails with error:

Too many levels of symbolic links

I restarted binfmt manually by running:

    sudo /usr/sbin/update-binfmts --disable
    sudo /usr/sbin/update-binfmts --enable

I think using systemd to do this would have also worked:

    sudo service binfmt-support restart

[Martin Pitt (pitti)]

I tried to reproduce this with the standard images.linuxcontainers.org trusty container image, on both a wily and a xenial host, but this doesn't happen with obvious steps (otherwise we would have noticed this much earlier already).

From the github issue I take it you are using Ubuntu wily -- which lxd package version do you use? The 0.20-0ubuntu4.1 shipped in Wily or some backport? I tried the wily version and that does not even work any more ("error: json: cannot unmarshal string into Go value of type int64" when trying to list or launch from images:), and there's no newer version in wily-backports.

How exactly did you create the container?

How often does this happen, under the same circumstances? Is this a race condition which only happens sometimes, or is this always reproducible? Can you reproduce this with running

    sudo systemctl stop proc-sys-fs-binfmt_misc.mount
    lxc launch [...]

several times? Stopping proc-sys-fs-binfmt_misc.mount will stop the actual mount and go back to systemd providing a "proxy" mount until the first access.

Please give me the output of "systemctl status proc-sys-fs-binfmt_misc.mount" and "mount | grep binfmt" before and after the failed "lxc launch".

Does it always work after the first failure and after /proc/sys/fs/binfmt_misc actually gets mounted?

[Stéphane Graber (stgraber)]

The issue isn't inside a container, for that matter lxd is completely irrelevant here.

We're seeing that weird symlink loop happen randomly on xenial machines.
Normal users likely won't notice it though as they don't use that mountpoint, LXC users do though as LXC bind-mounts it into the container which fails miserably when it's an infinite loop of symlinks.

[Stéphane Graber (stgraber)]

Just tested a dozen Xenial systems I have around and only found the issue on one, my laptop today:

root@castiana:~# ls -lh /proc/sys/fs/binfmt_misc
ls: cannot open directory '/proc/sys/fs/binfmt_misc': Too many levels of symbolic links
root@castiana:~#

[Stéphane Graber (stgraber)]

stgraber@dakara:~$ rssh castiana.lan.mtl
root@castiana:~# ls -lh /proc/sys/fs/binfmt_misc
ls: cannot open directory '/proc/sys/fs/binfmt_misc': Too many levels of symbolic links
root@castiana:~# cat /proc/mounts | grep binfmt
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
root@castiana:~# systemctl -a | grep binfmt
  proc-sys-fs-binfmt_misc.automount loaded active running Arbitrary Executable File Formats File System Automount Point
  proc-sys-fs-binfmt_misc.mount loaded inactive dead Arbitrary Executable File Formats File System
  binfmt-support.service loaded active exited Enable support for additional executable binary formats
  systemd-binfmt.service loaded inactive dead Set Up Additional Binary Formats
root@castiana:~#

[Stéphane Graber (stgraber)]

root@castiana:~# systemctl status proc-sys-fs-binfmt_misc.mount
● proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System
   Loaded: loaded (/lib/systemd/system/proc-sys-fs-binfmt_misc.mount; static; vendor preset: enabled)
   Active: inactive (dead) since Sat 2016-03-12 13:21:58 EST; 3 days ago
    Where: /proc/sys/fs/binfmt_misc
     What: binfmt_misc
     Docs: https://www.kernel.org/doc/Documentation/binfmt_misc.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

Mar 11 00:56:08 castiana systemd[1]: Mounting Arbitrary Executable File Formats File System...
Mar 11 00:56:08 castiana systemd[1]: Mounted Arbitrary Executable File Formats File System.
root@castiana:~#

[Stéphane Graber (stgraber)]

I'm trying to get a system that doesn't even have lxc installed to get into the issue.

[Stéphane Graber (stgraber)]

Not having much success reproducing this on a clean machine. In the mean time, I figured I'd fix my laptop and try to see what changes after a restart:

root@castiana:~# ls -lh /proc/sys/fs/binfmt_misc/
ls: cannot access '/proc/sys/fs/binfmt_misc/': Too many levels of symbolic links

root@castiana:~# cat /proc/mounts | grep binfmt
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0

root@castiana:~# systemctl restart proc-sys-fs-binfmt_misc.automount

root@castiana:~# cat /proc/mounts | grep binfmt
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0

root@castiana:~# ls -lh /proc/sys/fs/binfmt_misc/
total 0
-rw-r--r-- 1 root root 0 Mar 11 00:56 python2.7
-rw-r--r-- 1 root root 0 Mar 11 00:56 python3.5
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-aarch64
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-alpha
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-arm
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-armeb
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-cris
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-m68k
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-microblaze
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-mips
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-mips64
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-mips64el
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-mipsel
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-ppc
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-ppc64
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-ppc64abi32
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-ppc64le
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-s390x
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-sh4
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-sh4eb
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-sparc
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-sparc32plus
-rw-r--r-- 1 root root 0 Mar 11 00:56 qemu-sparc64
--w------- 1 root root 0 Mar 11 00:56 register
-rw-r--r-- 1 root root 0 Mar 11 00:56 status
root@castiana:~#

So as far as one can tell from the kernel, nothing changed...

[Stéphane Graber (stgraber)]

I tried starting stopping trusty, wily and xenial containers, none of that triggers it.

What's odd though is that every report I've seen so far, show no binfmt mounted, just the autofs. But LXC itself accesses the path so should have triggered autofs for the previous container startups.

So either this only happens before the first container starts, or something is unmounting the real binfmt mount.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Casey Marshall (cmars)]

I'm not seeing the symbolic links error, but I am seeing this when trying to start a trusty container on xenial: https://github.com/lxc/lxd/issues/1836

[Stéphane Graber (stgraber)]

@cmars
Note that the too many levels of symlink is on the host, not in the container and is the reason why you needed to do the umount and remount.

In fact, your log file says you did get that error:
            lxc 20160331183429.245 ERROR lxc_utils - utils.c:safe_mount:1692 - Too many levels of symbolic links - Failed to mount /proc/sys/fs/binfmt_misc onto /usr/lib/x86_64-linux-gnu/lxc/proc/sys/fs/binfmt_misc

[René Jochum (pcdummy)]

I'm also affected by this bug.

Host: Ubuntu Xenial amd64 (all updates installed)
Containers: Multiple xenial containers, one privileged Trusty container.
Affected container: Unprivileged Trusty container (privileged works fine).

[Stéphane Graber (stgraber)]

Just found this report too https://bugzilla.redhat.com/show_bug.cgi?id=986339

[Stéphane Graber (stgraber)]

That redhat bug shows a similar end result though I doubt the cause is the same in Ubuntu's case. We wouldn't see such random occurrences of the bug if it was something as simple as a package shipping a completely broken binfmt snippet.

[Casey Marshall (cmars)]

When is this going to be fixed upstream? I'm having to remount on the host and restart containers quite often. I can't imagine this will be a good user experience for new LXD users on xenial.

[Stéphane Graber (stgraber)]

We'd have to find a way to reproduce it and find what's actually causing it before we have any chance of fixing it.

[Eric Desrochers (slashd)]

I don't have the error "Too many levels of symbolic links" when doing an "ls" command but I do have a problem using "df" on Xenial after installing Openstack with lxd using "conjure-up"

"df" works fine, but after a reboot, it hangs.

A "strace df" shows that it hangs at doing :
..
stat("/sys/fs/cgroup/memory", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/sys/fs/cgroup/hugetlb", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/sys/fs/cgroup/cpuset", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/proc/sys/fs/binfmt_misc",

$ systemctl status proc-sys-fs-binfmt_misc.mount
● proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System
   Loaded: loaded (/lib/systemd/system/proc-sys-fs-binfmt_misc.mount; static; vendor preset: enabled)
   Active: inactive (dead) since Sat 2016-06-25 16:54:21 EDT; 9min ago
    Where: /proc/sys/fs/binfmt_misc
     What: binfmt_misc
     Docs: https://www.kernel.org/doc/Documentation/binfmt_misc.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
  Process: 6247 ExecMount=/bin/mount binfmt_misc /proc/sys/fs/binfmt_misc -t binfmt_misc (code=exited, status=0/SUCCESS)

Jun 25 16:54:21 <HOST> systemd[1]: Mounting Arbitrary Executable File Formats File System...
Jun 25 16:54:21 <HOST> systemd[1]: Mounted Arbitrary Executable File Formats File System.
Jun 25 16:54:22 <HOST> systemd[1]: proc-sys-fs-binfmt_misc.mount: Start request repeated too quickly.
Jun 25 16:54:22 <HOST> systemd[1]: Failed to mount Arbitrary Executable File Formats File System.

My workaround is to perform the following command at every reboot :
$ sudo dpkg-reconfigure binfmt-support

Eric

[Eric Desrochers (slashd)]

To continue my previous comment...

It is reproducible with :

binfmt-support (2.1.6-1)
systemd (229-4ubuntu6)

[Casey Marshall (cmars)]

Still seeing this on xenial enough that I've written a script to perform the umounting when LXD gets stuck. Do I even need binfmt_misc on my system? Can I just remove it or would that break something?

[Casey Marshall (cmars)]

I apt purge'ed binfmt-support from my system and I'm still able to comment here :) We'll see if this fixes the issue.

If it does, why was it even installed on my system by default to begin with?

[Casey Marshall (cmars)]

For any poor soul who wanders here, here is the best known workaround for this issue I've seen to date:

22-09-2016 15:23:18 < stgraber!~stgraber@ubuntu/member/stgraber: cmars: if you mask (systemctl mask) the systemd units related to binfmt (systemctl -a | grep binfmt) and reboot, then the problem is gone for good. This is related to systemd's use of automount and not to binfmt-misc (which just ships a bunch of binfmt hooks)

FYI, I think this will disable binfmt.

[Vance Morris (vmorris)]

2c - just hit this myself on xenail s390x with a soaking openstack-on-lxd installation.

masking proc-sys-fs-binfmt_misc.automount seemed to be a bad idea... systemd is in a bad state..

ubuntu@zs93kvi:~$ systemctl -a | grep binfmt
  proc-sys-fs-binfmt_misc.automount loaded active running Arbitrary Executable File Formats File System Automount Point
  proc-sys-fs-binfmt_misc.mount loaded inactive dead Arbitrary Executable File Formats File System
  systemd-binfmt.service loaded inactive dead Set Up Additional Binary Formats
ubuntu@zs93kvi:~$ sudo systemctl mask proc-sys-fs-binfmt_misc.automount
Created symlink from /etc/systemd/system/proc-sys-fs-binfmt_misc.automount to /dev/null.
ubuntu@zs93kvi:~$
Broadcast message from systemd-journald@zs93kvi (Fri 2017-05-05 10:25:46 EDT):

systemd[1]: Caught <ABRT>, dumped core as pid 23484.

Broadcast message from systemd-journald@zs93kvi (Fri 2017-05-05 10:25:46 EDT):

systemd[1]: Freezing execution.

[Vance Morris (vmorris)]

Forcing a reload of the LPAR brings the system back online with juju, lxd, and df happy again.

[Andreas Hasenack (ahasenack)]

This worked on my xenial host to unwedge the trusty containers:

    sudo service binfmt-support restart

[Stéphane Graber (stgraber)]

I've not seen this issue in quite a long time at least on bionic/focal, so will tentatively mark it as fix released. If someone still hits this, please tell us on what release and we'll add some SRU tasks.