"non-accessable symlink" errors when using aufs-shaddowed read-only root filesystem

[robbak]

When I updated my pxeboot/nfs system to 10.10, several applets failed to load, and the log files contained the following messages:
Oct 19 22:25:58 computer08 kernel: [ 66.326808] non-accessible hardlink creation was attempted by: gconfd-2 (fsuid 1000)
Oct 19 22:26:05 computer08 kernel: [ 73.491138] non-accessible hardlink creation was attempted by: gconfd-2 (fsuid 114)
Oct 19 22:26:05 computer08 kernel: [ 73.580608] non-accessible hardlink creation was attempted by: gdm-session-wor (fsuid 1000)
Oct 19 22:26:09 computer08 kernel: [ 77.545565] non-accessible hardlink creation was attempted by: indicator-apple (fsuid 1000)
Oct 19 22:26:09 computer08 kernel: [ 77.670861] non-accessible hardlink creation was attempted by: indicator-apple (fsuid 1000)

Setting kernel.yama.protected_nonaccess_hardlinks=0 allows everything to work.

The read-only root is configured according to the instructions at http://www.logicsupply.com/blog/2009/01/27/how-to-build-a-read-only-linux-system . It uses a initramfs script to create a tmpfs, move /, and mount the aufs union filesystem on / early in the boot process.

[Rémi Rérolle (remi.rerolle)]

Can be easily reproduced with the following steps:

mkdir ro rw aufs
touch ro/foo
sudo mount -t aufs -o br:rw:ro none aufs
mv aufs/{foo,bar}

Gives the following output:

mv: cannot move `aufs/foo' to `aufs/bar': Operation not permitted

And:

dmesg | tail -1
[179473.298795] non-accessible hardlink creation was attempted by: mv (fsuid 1000)

I confirm the kernel.yama.protected_nonaccess_hardlinks=0 workaround

[gcc (chris+ubuntu-qwirx)]

Confirmed. This also causes problems if you try to use aufs to create a sandbox directory for LTSP guest users, like this:

http://kristianlyng.wordpress.com/2010/03/19/a-sandboxed-home-directory/

This fails because xauth can't remove and replace the .Xauthority file when you log in graphically. sshd tells xauth to remove the existing cookie and create a new one. xauth creates .Xauthority-n, tries to remove .Xauthority (which fails), and then tries to hardlink .Xauthority to .Xauthority-n (which fails because .Xauthority still exists):

[pid 1150] send(10, "<39>Dec 7 16:34:12 sshd[1150]: debug1: Forked child 10969.", 59, MSG_NOSIGNAL <unfinished ...>
[pid 10969] execve("/usr/sbin/sshd", ["/usr/sbin/sshd", "-R"], [/* 5 vars */]) = 0
[pid 10969] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb771cb48) = 11031
Process 11031 attached

[pid 10969] send(4, "<38>Dec 7 16:34:20 sshd[10969]: User child is on pid 11031", 59, MSG_NOSIGNAL) = 59
[pid 11031] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb771cb48) = 11032
Process 11032 attached

[pid 11032] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb771cb48) = 11033
Process 11033 attached
[pid 11033] execve("/bin/sh", ["sh", "-c", "/usr/bin/xauth -q -"], [/* 13 vars */] <unfinished ...>

[pid 11032] write(4, "remove unix:38.0\nadd unix:38.0 MIT-MAGIC-COOKIE-1 9296d13c776bd89c10fe65673f3088e5\n", 83 <unfinished ...>

[pid 11033] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7880938) = 11034
Process 11034 attached

[pid 11034] execve("/usr/bin/xauth", ["/usr/bin/xauth", "-q", "-"], [/* 14 vars */]) = 0
[pid 11034] open("/home/guest_091221/.Xauthority", O_RDONLY) = 3
[pid 11034] close(3) = 0
[pid 11034] read(0, "remove unix:38.0\nadd unix:38.0 MIT-MAGIC-COOKIE-1 9296d13c776bd89c10fe65673f3088e5\n", 4096) = 83
[pid 11034] unlink("/home/guest_091221/.Xauthority") = -1 EPERM (Operation not permitted)
[pid 11034] link("/home/guest_091221/.Xauthority-n", "/home/guest_091221/.Xauthority") = -1 EEXIST (File exists)
[pid 11034] write(2, "/usr/bin/xauth: unable to link authority file /home/guest_091221/.Xauthority, use /home/guest_091221/.Xauthority-n\n", 116) = 116

Here's how to reproduce it without sshd and xauth:

manager@zambiaserver2:~$ mkdir base
manager@zambiaserver2:~$ touch base/.Xauthority

manager@zambiaserver2:~$ mkdir tmp
manager@zambiaserver2:~$ sudo mount -t tmpfs none tmp

manager@zambiaserver2:~$ mkdir aufs
manager@zambiaserver2:~$ sudo mount -t aufs -o dirs=tmp=rw:base=ro none aufs

manager@zambiaserver2:~$ ls -la aufs
total 4
drwxrwxrwt 4 root root 100 2010-12-07 16:08 .
drwxr-xr-x 39 manager manager 4096 2010-12-07 16:04 ..
-rw-r--r-- 1 manager manager 0 2010-12-07 16:03 .Xauthority

manager@zambiaserver2:~$ rm aufs/.Xauthority
rm: cannot remove `aufs/.Xauthority': Operation not permitted
manager@zambiaserver2:~$ sudo tail -1 /var/log/kern.log
Dec 7 16:09:04 zambiaserver2 kernel: [ 3445.254650] non-accessible hardlink creation was attempted by: rm (fsuid 1000)

manager@zambiaserv...

[Andy Whitcroft (apw)]

This was fixed by this commit:

    UBUNTU: ubuntu: Yama: if an underlying filesystem provides a permissions op

    When we are checking permissions on hardlinks we use generic_permissions()
    to work out if the user actually has read/write permissions and only
    then allow the link. However where the underlying filesystem supplies
    a permissions() op there is no guarentee that the inode ownership is
    actually valid and we must use that op instead.

    Add a new function mirroring the core fragment from inode_permission
    using the filesystem specific permissions() op falling back to
    generic_permissions() when it is not present.

    With this in place links in overlayfs behave as expected.

    Signed-off-by: Andy Whitcroft <email address hidden>

[rbhkamal (rbhkamal)]

Where do get the fix for this? Thanks

[robbak (robbak)]

Is this still happening? I no longer have the system that I found this
bug on. I posted the workaround in this threads first message, and a
solution was committed to Ubuntu long ago. Should not be a problem in
current ubuntu. Yes, my searches say that the fix was committed in
July 2011.

If you are still having this issue, and you have updated to a current
version, then post a new bug: You've caught a regression. If you have
not updated, then do.

On 7 October 2012 04:00, rbhkamal <email address hidden> wrote:
> Where do get the fix for this? Thanks
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/663069
>
> Title:
> "non-accessable symlink" errors when using aufs-shaddowed read-only
> root filesystem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/663069/+subscriptions

[Ken Sharp (kennybobs)]

Upgrade from what to what?

I see this on a tmpfs+aufs mount.

[Donald Allwright (donald-allwright)]

I'm still seeing this on 12.04, but not on 13.04. Reproducible as follows:

me@laptop ~/temp $ mkdir ro rw aufs
me@laptop ~/temp $ touch ro/dummy
me@laptop ~/temp $ sudo mount -t aufs -o dirs=rw:ro=ro none aufs
[sudo] password for me:
me@laptop ~/temp $ ls aufs/
dummy
me@laptop ~/temp $ rm aufs/dummy
rm: cannot remove `aufs/dummy': Operation not permitted
me@laptop ~/temp $

On 13.04 I don't get the error message and the file is 'deleted' (i.e, a whiteout file is correctly created). Looks like the bug fix hasn't been ported to the LTS release, but probably needs doing so.