Filesystem mount from lxc template causes filesystem permission breakages

[Stuart Bishop]

In juju-core 1.17.5, creating new lxc machines is now much faster as it appears to be using a template machine. In addition, the root filesystem is mounted from the template machine.

Unfortunately, this causes filesystem permissions to screw up.

juju deploy ubuntu
juju ssh ubuntu/0
sudo chown ubuntu:ubuntu /etc/ssl/private
ls /etc/ssl/private

That final 'ls' fails with a permission denied. This is possibly a security precaution in lxc or the filesystem.

This issue breaks the postgresql charm. The PostgreSQL packages require and use the ssl-cert package, which changes /etc/ssl/private to be group readable by the ssl-cert group. The postgres user, a member of the ssl-cert group, is unable to read the private key stored in this directory.

[Tim Penhey (thumper)]

This isn't an aufs permission bug it seems.

[Tim Penhey (thumper)]

sudo lxc-create -t ubuntu-cloud -n base -- -r precise
sudo lxc-clone --snapshot -o base -n clone-aufs --backingstore aufs
sudo lxc-clone -o base -n clone-dir

I then started each container in different terminals

sudo lxc-start -n clone-dir
sudo lxc-start -n clone-aufs

and executed the following two commands:

sudo chown ubuntu:ubuntu /etc/ssl/private
ls /etc/ssl/private

This worked on clone-dir, and got permission failed on clone-aufs

[Serge Hallyn (serge-hallyn)]

Thanks, that is a great reproducer.

Note that doing
   rm -rf /etc/ssl/private
   mkdir /etc/ssl/private

works around this, and explains why this *may* in fact be on purpose. If you only do
    sudo chown ubuntu:ubuntu /etc/ssl/private

then the underlying directory is still owned by root and not readable by ubuntu. So this could be seen as allowing for an information leak, although it seems like a stretch to me since root has to do the chown in the first place.

It wasn't immediately obvious to me where in the code (linux/ubuntu/aufs/) this is happening.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1293549

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Serge Hallyn (serge-hallyn)]

@Tim,

is rmdir/mkdir an acceptable workaround for juju (ie can we lower priority of the bug) or will that not be doable?

[Adam Conrad (adconrad)]

17:07 < infinity> hallyn: Well, that certainly shouldn't change the permissions on the underlying dir, as that's immutable.
17:07 < infinity> hallyn: But copying up, changing permissions, and using that copy would seem sane.

[Tim Penhey (thumper)]

@Serge, while it may be a work around for some, it certainly is going to be a problem in the big picture.

We are running code written by others (juju hooks).

I'm going to submit a branch to juju to disable aufs by default, but enable the setting so people can try it out.

[Tim Penhey (thumper)]

Shouldn't aufs catch a chmod to a directory that isn't in the delta and copy it across?

[Tim Penhey (thumper)]

or a chown

[Andy Whitcroft (apw)]

A naieve test of aufs directly shows that chown and chmod do cause a copy up of the underlying files as expected.

In the read only layer before chmod/chown:

  drwxrwxr-x 2 apw apw 4096 Mar 18 09:16 D1
  drwxrwxr-x 2 apw apw 4096 Mar 18 09:16 D2
  drwxrwxr-x 2 apw apw 4096 Mar 18 09:16 D3

In the mount after:

  drwxrwxr-x 2 sbuild sbuild 4096 Mar 18 09:16 D1
  drwxrwxrwx 2 apw apw 4096 Mar 18 09:16 D2
  drwxrwxrwx 2 apw apw 4096 Mar 18 09:16 D3

The underlying permissions remain unchanged after these operations. This all seems semantically correct.

I need a description of how we are using aufs in these this scenario (in comment #2), for instance are we modifing the actual underlying files while mounted which would be a no-no.

[Andy Whitcroft (apw)]

Ok a more refined reproducer shows that this is an issue triggered by different permissions on the various layers. Although the directory is reported correctly the permissions for each layer are used on that layer to control who can actually see the contents of the directory. If you cannot rx the lower levels you cannot read the directory at all. This is a deliberate design decision.