OverlayFS: Wrong mnt_id and path reported in /proc in linux-3.13

[oleg]

I am running Ubuntu Trusty (3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux).

There are two issues in OverlayFS as follows:

 1. /proc/<pid>/fdinfo/<fd> reports wrong mnt_id (non-existent in /proc/<pid>/mountinfo)

 2. /proc/<pid>/fd/<fd> shows incorrect path for the symlink target

These issues were previously reported and patched for the Ubuntu Vivid kernel (3.19) in bug #1479468, but the patch was not applied to the 3.13 kernel, possibly because it contains an older version of overlayfs.

These issues are now adversely affecting some versions of lxc when used in conjunction with the 3.13 kernel, because lxc has recently been patched to pass /proc/<pid>/fd/<fd> to the mount syscall. This change to lxc was made in response to cve-2015-1335; see bug #1476662.

The kernel issues cause some overlayfs-backed containers to fail to start, with error message,

strace: mount("/home/ubuntu/test.txt", "/proc/self/fd/16", 0x7ffd4c3f7fdb, MS_BIND, NULL) = -1 EACCES (Permission denied)

The mount fails because /proc/self/fd/16, which should point to a file in overlayfs, has an invalid value.

The version of lxc in the trusty-updates repository, lxc-1.0.7-0ubuntu0.7,
is not affected. The following versions of lxc are affected:

repository lxc version
----------------------------------------------------------------------------
ppa:ubuntu-lxc/lxc-git-stable-1.0 1.0.7+stable~20151014-2143-0ubuntu1~trusty
ppa:ubuntu-lxc/lxc-git-stable-1.1 1.1.3+stable~20151007-2301-0ubuntu1~trusty
ppa:ubuntu-lxc/lxc-git-master 1.1.3+master~20151012-0541-0ubuntu1~trusty

To reproduce the lxc failure-to-start issue:

# Install the 3.13 kernel and also an affected version of lxc.
lxc-create -t download -n trusty -- -d ubuntu -r trusty -a amd64;
touch $HOME/test.txt;
echo "lxc.mount.entry = $HOME/test.txt home/ubuntu/test.txt none bind,create=file 0 0" >> $HOME/.local/share/lxc/trusty/config;
lxc-clone -s -B overlayfs trusty trusty_overlay;
lxc-start -n trusty_overlay; # fails to start

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1507463

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[oleg (overlayfs)]

apport information

[Chris J Arges (arges)]

I tried the following:

# Install the 3.13 kernel and also an affected version of lxc.
lxc-create -t download -n trusty -- -d ubuntu -r trusty -a amd64;
touch $HOME/test.txt;
echo "lxc.mount.entry = $HOME/test.txt home/ubuntu/test.txt none bind,create=file 0 0" >> $HOME/.local/share/lxc/trusty/config;
lxc-clone -s -B overlayfs trusty trusty_overlay;
lxc-start -n trusty_overlay; # fails to start

In a clean and fully updated Trusty VM and could not reproduce the issue with the trusty version of the packages.
ii lxc 1.0.7-0ubuntu0.7 amd64 Linux Containers userspace tools

I also tried the stable version of LXC (https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/lxc-stable) and was able to get this working.
ii lxc 1.1.4-0ubuntu1~ubuntu14.04.1~ppa1 amd64 Linux Containers userspace tools

So seems to be fixed in 1.1.4, and I suspect this commit fixes the issue:
commit e228426cf483dc6a406276550b9cc7222e7c2602
Author: Serge Hallyn <email address hidden>
Date: Mon Sep 21 17:09:25 2015 +0000

    overlayfs_mount: create delta dir if it doesn't exist

Can you re-test with 1.1.4 and see if that fixes the issue? Thanks,
--chris

[oleg (overlayfs)]

I retested using lxc-1.1.4 from the lxc-stable ppa (https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/lxc-stable), which I added to a fresh VM image from the Trusty daily-builds, https://cloud-images.ubuntu.com/trusty/current.

"lxc-start -n trusty_overlay" fails as before, with the same error message,

ERROR lxc_utils - utils.c:safe_mount:1641 - Permission denied - Failed to mount /home/ubuntu/test.txt onto /usr/lib/x86_64-linux-gnu/lxc/home/ubuntu/test.txt
ERROR lxc_conf - conf.c:mount_entry:1731 - Permission denied - failed to mount '/home/ubuntu/test.txt' on '/usr/lib/x86_64-linux-gnu/lxc/home/ubuntu/test.txt'
ERROR lxc_conf - conf.c:lxc_setup:3745 - failed to setup the mount entries for 'trusty_overlay'

[oleg (overlayfs)]

Chris, it would be of interest to see whether you can reproduce the underlying kernel bug using this script,

#!/bin/bash
fatal() { echo "error: $@"; exit 1; }

echo -e "testing for overlayfs kernel bug in kernels <= 3.17.";
echo -e "(This script does not work for the newer version of overlayfs in kernels 3.18+)";

mkdir test_dir || fatal "mkdir test_dir";
sudo mount -t tmpfs none test_dir || fatal "mount tmpfs";
cd test_dir;
mkdir lowerdir upperdir overlayfs;
sudo mount -t overlayfs -o lowerdir=lowerdir,upperdir=upperdir none overlayfs || fatal "mount overlayfs";

exec 6> file_tmpfs.txt;
exec 7> lowerdir/file_lowerdir.txt;
exec 8> upperdir/file_upperdir.txt;
exec 9> overlayfs/file_overlayfs.txt;

echo -e "\ncontents of /proc/$BASHPID/fd/ . A broken symbolic link to file_overlayfs.txt indicates a kernel bug";
echo "--------------------------------------------------------------------------------------------------------";
file /proc/$BASHPID/fd/[6-9];

# cleanup
exec 9>&-;
exec 8>&-;
exec 7>&-;
exec 6>&-;
sudo umount overlayfs;
cd ..;
sudo umount test_dir && rmdir test_dir;
#----------end of script

Running the script as root, sudo ./script, I get the following output,

  testing for overlayfs kernel bug in kernels <= 3.17.
  (This script does not work for the newer version of overlayfs in kernels 3.18+)

  contents of /proc/1916/fd/ . A broken symbolic link to file_overlayfs.txt indicates a kernel bug
  --------------------------------------------------------------------------------------------------------
  /proc/1916/fd/6: symbolic link to `/home/ubuntu/test_dir/file_tmpfs.txt'
  /proc/1916/fd/7: symbolic link to `/home/ubuntu/test_dir/lowerdir/file_lowerdir.txt'
  /proc/1916/fd/8: symbolic link to `/home/ubuntu/test_dir/upperdir/file_upperdir.txt'
  /proc/1916/fd/9: broken symbolic link to `/file_overlayfs.txt'

It seems that the kernel bug is causing fd 9 to point to /file_overlayfs.txt, when it should point to /home/ubuntu/test_dir/overlayfs/file_overlayfs.txt .

[oleg (overlayfs)]

The following lxc commit introduced safe_mount(), which passes fd's as arguments to mount(),

CVE-2015-1335: Protect container mounts against symlinks
https://github.com/lxc/lxc/commit/6de26af93d3dd87c8b21a42fdf20f30fa1c1948d

Here is the relevant part of the commit, which creates a file descriptor and passes it to mount().

+ destfd = open_without_symlink(dest, rootfs);
+ if (destfd < 0) {
+ if (srcfd != -1)
+ close(srcfd);
+ return destfd;
+ }
+
+ ret = snprintf(destbuf, 50, "/proc/self/fd/%d", destfd);
+ if (ret < 0 || ret > 50) {
+ if (srcfd != -1)
+ close(srcfd);
+ close(destfd);
+ ERROR("Out of memory");
+ return -EINVAL;
+ }
+
+ ret = mount(mntsrc, destbuf, fstype, flags, data);

The strace of mount() shows,

strace: mount("/home/ubuntu/test.txt", "/proc/self/fd/16", 0x7ffd4c3f7fdb, MS_BIND, NULL) = -1 EACCES (Permission denied)

The mount fails because /proc/self/fd/16, which should point to a file in overlayfs, has an invalid value.

[oleg (overlayfs)]

Related bug #1347746 : Overlayfs breaks /proc/self/exe link

[oleg (overlayfs)]

This bug does still affect lxc on Ubuntu Trusty when using the 3.13 kernel in conjunction with the latest daily-build of lxc, 1.1.5+master~20151110-0623-0ubuntu1~trusty (from the lxc-daily ppa, https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/daily).