Don't require use of mediate_deleted with LXC (was: apparmor prevents dpkg-divert and localedef from working in a container)

[Stéphane Graber]

I moved the daily flavour upgrade testing to a container but it's now failing when running ubuntu-vm-builder, here are the entries from dmesg:
[ 2038.491817] type=1400 audit(1333119659.468:51): apparmor="DENIED" operation="getattr" info="Failed name lookup - deleted entry" error=-2 parent=19255 profile="lxc-container-upgrader01" name="/tmp/tmpQ1TioA/var/lib/dpkg/diversions" pid=19259 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 2149.277909] type=1400 audit(1333119770.257:52): apparmor="DENIED" operation="getattr" info="Failed name lookup - deleted entry" error=-2 parent=25847 profile="lxc-container-upgrader01" name="/tmp/tmpQ1TioA/var/lib/dpkg/diversions" pid=25849 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 2263.501949] type=1400 audit(1333119884.482:53): apparmor="DENIED" operation="chmod" info="Failed name lookup - deleted entry" error=-2 parent=5444 profile="lxc-container-upgrader01" name="/tmp/tmpQ1TioA/usr/lib/locale/locale-archive.Ou6sxd" pid=5450 comm="localedef" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[ 2264.736948] type=1400 audit(1333119885.718:54): apparmor="DENIED" operation="chmod" info="Failed name lookup - deleted entry" error=-2 parent=5511 profile="lxc-container-upgrader01" name="/tmp/tmpQ1TioA/usr/lib/locale/locale-archive.D05snx" pid=5531 comm="localedef" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[ 2367.429100] type=1400 audit(1333119988.408:55): apparmor="DENIED" operation="getattr" info="Failed name lookup - deleted entry" error=-2 parent=5553 profile="lxc-container-upgrader01" name="/tmp/tmpQ1TioA/var/lib/dpkg/diversions" pid=9783 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

The apparmor profile used for this container is attached.

[Stéphane Graber (stgraber)]

Reason for critical is that it's making random commands in container fail.
We've already got a few bug reports against udev, postgresql, ... all caused by that issue.

[James Page (james-page)]

Interestingly when I ser the lxc-container-default profile to complain:

sudo aa-complain /etc/apparmor.d/lxc/lxc-default

I no longer get the issue in the lxc instance - however neither do I get any complaints.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John Johansen (jjohansen)]

While I haven't tried this yet, my initial thought when seeing it works in complain mode, but there are no messages is that this is something that is being specifically denied in the profile.

to confirm this we need to disable quieting of explicitly denied messages, we can do this as root with

echo -n "noquiet" > /sys/module/apparmor/parameters/audit

[JP Viljoen (froztbyte)]

Friend of mine solved this, asked me if I can post it, so here goes:

/etc/apparmor.d/lxc/lxc-default profile needs 'flags=(mediate_deleted)' appended to it, and the problem should go away. Documentation reference for this is at http://wiki.apparmor.net/index.php/FAQ#Failed_name_lookup_-_deleted_entry and attached also find find a modified lxc-default entry.

[John Johansen (jjohansen)]

This does indeed seem to be the problem. The current labeling done by apparmor is not enough to avoid needing the mediate_deleted flag on the lxc profiles. Adding the flag will force apparmor to do a name lookup for entries that have been deleted (the name can be reliably be reconstructed), instead of using the default of the cached file label.

I have opened Bug #970647 for the failure to log rejects due to the deleted entry logic.

[Serge Hallyn (serge-hallyn)]

@JP

great! Thanks for that. I'll add that for now as a workaround.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 0.7.5-3ubuntu49

---------------
lxc (0.7.5-3ubuntu49) precise; urgency=low

  * debian/lxc-default.apparmor: add mediate_deleted flag (LP: #969299)
 -- Serge Hallyn <email address hidden> Mon, 02 Apr 2012 09:38:21 -0500

[Jamie Strandboge (jdstrand)]

Marking the apparmor task as Won't Fix since the lxc work around is in place. If we pursue this in SRU, it will be through bug #970647.

[Serge Hallyn (serge-hallyn)]

Based on the duplicates, I'm not sure the workaround is working as well as we'd hoped.

John, what are the prospects of bug 970647? How complicated is the fix for it?

[Francesco Del Degan (pr0gg3d)]

I'm sorry if this is not the place to report this, but running localedef into a lxc ubuntu container it's affecting quantal right now.

The log line is

[26775.302073] type=1400 audit(1353478924.553:73): apparmor="DENIED" operation="chmod" info="Failed name lookup - deleted entry" error=-2 parent=14028 profile="/usr/bin/lxc-start" name="/usr/lib/locale/locale-archive.fyr1kX" pid=14336 comm="localedef" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

I just fixed adding mediate_deleted into /etc/apparmor.d/usr.bin.lxc-start, but i don't know it this is the right fix.

[John Johansen (jjohansen)]

Francesco,

The mediate_deleted flag should fix the rejection shown in comment #12

[John Johansen (jjohansen)]

Serge,

see comments on bug 970647, there is some progress but I have not found a specific bug affecting logging of this case. The larger fix which is the extended labeling, is in progress and will enter into the apparmor-dev ppa soon for testing.

[Serge Hallyn (serge-hallyn)]

Francesco,

The DENIED message doesn't look right. It says your containern is running in the lxc-start pfofile? it should have transitioned to a container profile when /sbin/init was executed.

I think it is worth opening a new bug about your issue, so we can make sure there isn't more going on.

[Iain Lane (laney)]

I get this (newly?) when trying to update within sbuild within lxc

[ 1927.282880] type=1400 audit(1383816970.374:86): apparmor="DENIED" operation="getattr" info="Failed name lookup - deleted entry" error=-2 parent=11717 profile="/usr/bin/lxc-start" name="/var/lib/schroot/mount/trusty-amd64-c7aa6e25-c1a2-401f-864d-d0b82f4002b5/var/lib/dpkg/diversions" pid=12244 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Serge Hallyn (serge-hallyn)]

Quoting Iain Lane (<email address hidden>):
> I get this (newly?) when trying to update within sbuild within lxc
>
> [ 1927.282880] type=1400 audit(1383816970.374:86): apparmor="DENIED"
> operation="getattr" info="Failed name lookup - deleted entry" error=-2
> parent=11717 profile="/usr/bin/lxc-start" name="/var/lib/schroot/mount

lxc-start -> that is not the profile you should be under.

Is this by chance a 3.12 kernel?

[Iain Lane (laney)]

On Thu, Nov 07, 2013 at 03:20:29PM -0000, Serge Hallyn wrote:
> Quoting Iain Lane (<email address hidden>):
> > I get this (newly?) when trying to update within sbuild within lxc
> >
> > [ 1927.282880] type=1400 audit(1383816970.374:86): apparmor="DENIED"
> > operation="getattr" info="Failed name lookup - deleted entry" error=-2
> > parent=11717 profile="/usr/bin/lxc-start" name="/var/lib/schroot/mount
>
> lxc-start -> that is not the profile you should be under.
>
> Is this by chance a 3.12 kernel?

Sure is. 3.12.0-1-generic

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Serge Hallyn (serge-hallyn)]

Quoting Iain Lane (<email address hidden>):
> On Thu, Nov 07, 2013 at 03:20:29PM -0000, Serge Hallyn wrote:
> > Quoting Iain Lane (<email address hidden>):
> > > I get this (newly?) when trying to update within sbuild within lxc
> > >
> > > [ 1927.282880] type=1400 audit(1383816970.374:86): apparmor="DENIED"
> > > operation="getattr" info="Failed name lookup - deleted entry" error=-2
> > > parent=11717 profile="/usr/bin/lxc-start" name="/var/lib/schroot/mount
> >
> > lxc-start -> that is not the profile you should be under.
> >
> > Is this by chance a 3.12 kernel?
>
> Sure is. 3.12.0-1-generic

The fix for that should be in the trusty kernel I believe mid-next week.

Would you mind opening a new bug against lxc saying that if the
container is in profile lxc-start, and apparmor support is lacking,
it must run unconfined or refuse to run?

[Sidnei da Silva (sidnei)]

Confirmed fixed in 3.13.0-2-generic, where in 3.13.0-1-generic it was still failing.

[Randall Leeds (randall-leeds)]

Any chance this will be fixed in saucy?