rm -r * fails to delete directories when using overlayfs in a user-namespace

[oleg]

rm -r * fails to delete directories when using overlayfs in a user-namespace.

If overlayfs is mounted as follows,

   mount -n -t overlay overlay -o lowerdir=lowerdir,upperdir=upperdir,workdir=workdir mntpt

and if lowerdir contains lowerdir/foo/bar.txt, then,

   rm -r mntpt/foo

fails with error message,

   "rm: cannot remove ‘mntpt/foo’: Operation not permitted"

strace shows,

    unlinkat(AT_FDCWD, "mntpt/foo", AT_REMOVEDIR) = -1 EPERM (Operation not permitted)

OS details: 64-bit ubuntu 14.04.2 with the linux-generic-lts-vivid kernel (3.19).

The bug does not occur on ubuntu 14.04.2 with the default linux-generic 3.13 kernel.

The bug occurs for any vanilla 3.18+ kernel to which the 1-line patch, "overlayfs: allow unprivileged mounts" is added. (The patch is at, http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/commit/?id=78ec45495b15d27d4cc6d05cc454e30ec5b587ea)

The bug occurs regardless of whether the filesystem underlying overlayfs is tmpfs or ext4.

The bug also occurs when running ubuntu 15.04 (ubuntu vivid) in a virtual machine (qemu-system-x86_64). The host is ubuntu 14.04.2 with the default 3.13 kernel.

A script which reproduces the bug is attached and is also available at http://paste.ubuntu.com/11974137/

Hardware: intel core2 duo processor in a macbook-4.1

[Thomas Müller (muellerthomas977)]

I can reproduce this problem on a fresh machine set-up at AWS (kernel Linux ip-172-31-6-51 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux).

Below the output of the script reproduction script linked above:

```
mounted overlayfs

ls -RF mntpt
 -------------
mntpt:
foo/

mntpt/foo:
bar.txt

rm -r mntpt/*
 -------------
rm: cannot remove ‘mntpt/foo’: Operation not permitted
exit code=1

ls -RF mntpt
 -------------
mntpt:
foo/

mntpt/foo:

unmounted overlayfs
cleaning up
```

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Serge Hallyn (serge-hallyn)]

This is not a bug. Easiest way is to

lxc-usernsexec -- rm -rf $container_dir/
rm -rf $container_dir/

[oleg (overlayfs)]

I don't quite get the point underlying Serge's preceding comment, so I'll describe the problem I'm experiencing with lxc and see whether Serge classifies it as a bug.

On an ubuntu-vivid (amd64) host, create an unprivileged vivid container named 'vivid',
   lxc-create -n vivid -t download -- -d ubuntu -r vivid -a amd64
Start and attach to the container and create foo/bar.txt . Stop the container.

Then form a second container named vivid_overlay,
   lxc-clone -s -B overlayfs vivid vivid_overlay
Start and attach to vivid_overlay.
Delete bar.txt (this should succeed).
Attempt to delete foo. I get "Operation not permitted", even as root in the container.

The same issue arises if I replace the overlay container with an ephemeral container,
   lxc-start-ephemeral -o vivid -n vivid_ephemeral

[Serge Hallyn (serge-hallyn)]

Ah, sorry. I thought you were trying to manually delete the whole container (instead of using lxc-destroy).

[Serge Hallyn (serge-hallyn)]

(confirmed here on my wily laptop - thanks for reporting this)

[oleg (overlayfs)]

This bug does still affect lxc (as per comment #5) when using kernel 3.18 or above in conjunction with the latest daily-build of lxc, 1.1.5+master~20151110-0623-0ubuntu1~trusty (from the lxc-daily ppa, https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/daily).

[Serge Hallyn (serge-hallyn)]

@oleg,

yes, but it is not an lxc bug, there's nothing lxc can do about it. Stéphane un-marked it from lxc to make the lxc bug view more usable so we can use it rather than ignore it :)

[oleg (overlayfs)]

@Serge,

Thanks for the clarification; I'll revert the status to un-marked.