classic snap files logs with apparmor ALLOWED messages

[Jamie Strandboge]

From https://lists.ubuntu.com/archives/snapcraft/2017-January/002370.html:

"When building a c++ app on a raspberry pi 3 in classic mode,
I noticed two things that hurt performance substantially.

1) the obvious: if you crank config.txt's gpu_mem up above the
default, life is bad :-) Don't do that on build machines.

2) the not so obvious: apparmor is keeping systemctl's journal process
at 100% on one core logging crud like

Jan 06 18:03:37 rbb-ubu1604pi3-1 audit[3249]: AVC apparmor="ALLOWED"
operation="open"
profile="snap.classic.classic//null-/usr/bin/systemd-run//null-/usr/sbin/chroot//null-/var/snap/classic/common/classic/bin/dash//null-/var/snap/classic/common/classic/usr/bin/script//null-/var/snap/classic/common/classic/bin/bash//null-/var/snap/classic/common/classic/usr/bin/sudo//null-/var/snap/classic/common/classic/bin/bash//null-/var/snap/classic/common/classic/home/obdank/src/ob-repobot/bau//null-/var/snap/classic/common/classic/home/obdank/src/ob-repobot/yovo/buildshim//null-/var/snap/classic/common/classic/home/obdank/src/ob-repobot/yovo/cmake.d/buildshim-ubu//null-/var/snap/classic/common/classic/usr/bin/dpkg-buildpackage//null-/var/snap/classic/common/classic/home/obdank/src/yovo/debian/rules//null-/var/snap/classic/common/classic/usr/bin/dh//null-/var/snap/classic/common/classic/home/obdank/src/yovo/debian/rules//null-/var/snap/classic/common/classic/usr/bin/dh_auto_build//null-/var/snap/classic/common/classic/usr/bin/make//null-/var/snap/classic/common/classic/usr/bin/make//null-/var/snap/classic/common/classic/usr/bin/make//null-/var/snap/classic/common/classic/bin/dash//null-/var/snap/classic/common/classic/usr/bin/g++-5//null-/var/snap/classic/common/classic/usr/lib/gcc/arm-linux-gnueabihf/5/cc1plus"
name="/var/snap/classic/common/classic/usr/include/c++/5/cstring"
pid=3249 comm="cc1plus" requested_mask="r" denied_mask="r" fsuid=1000
ouid=0
Jan 06 18:03:37 rbb-ubu1604pi3-1 audit[3254]: AVC apparmor="ALLOWED"
operation="mknod"
profile="snap.classic.classic//null-/usr/bin/systemd-run//null-/usr/sbin/chroot//null-/var/snap/classic/common/classic/bin/dash//null-/var/snap/classic/common/classic/usr/bin/script//null-/var/snap/classic/common/classic/bin/bash//null-/var/snap/classic/common/classic/usr/bin/sudo//null-/var/snap/classic/common/classic/bin/bash//null-/var/snap/classic/common/classic/home/obdank/src/ob-repobot/bau//null-/var/snap/classic/common/classic/home/obdank/src/ob-repobot/yovo/buildshim//null-/var/snap/classic/common/classic/home/obdank/src/ob-repobot/yovo/cmake.d/buildshim-ubu//null-/var/snap/classic/common/classic/usr/bin/dpkg-buildpackage//null-/var/snap/classic/common/classic/home/obdank/src/yovo/debian/rules//null-/var/snap/classic/common/classic/usr/bin/dh//null-/var/snap/classic/common/classic/home/obdank/src/yovo/debian/rules//null-/var/snap/classic/common/classic/usr/bin/dh_auto_build//null-/var/snap/classic/common/classic/usr/bin/make//null-/var/snap/classic/common/classic/usr/bin/make//null-/var/snap/classic/common/classic/usr/bin/make//null-/var/snap/classic/common/classic/bin/dash//null-/var/snap/classic/common/classic/usr/bin/g++-5"
name="/var/snap/classic/common/classic/tmp/ccpOmyw6.s" pid=3254
comm="c++" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

Is there an option for reducing apparmor verbosity on ALLOWED operations?
(speaking strictly as a foreigner in the land of apparmor and snappy)

Thanks!
"

What is happening is that the classic snap is currently using a chroot with devmode. devmode causes policy violations to be allowed but logged but because it is using a chroot instead of a pivot_root, all the file accesses don't match the rules from the default template. To silence the logging, the policy needs to be updated. There are several options on how to do this and it needs a little bit of design on which implementation is best.

WORKAROUND: you can add this to /var/lib/snapd/apparmor/profiles/snap.classic.classic before the final '}':

  mount fstype=devpts options=(rw) devpts -> /dev/pts/,
  /bin/mountpoint ixr,
  @{PROC}/[0-9]*/mountinfo r,
  /var/lib/extrausers/{,*} r,
  capability fsetid,
  capability dac_override,
  /etc/sudoers.d/{,*} r,
  /usr/bin/systemd-run Uxr,
  /bin/systemctl Uxr,

then load the profile into the kernel with:

$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.classic.classic

Now you should be able to 'sudo classic' and not see any apparmor logging. Note that you may have to re-add the above rules to the profile (eg, if the snap is removed/installed/updated/etc).

[Jamie Strandboge (jdstrand)]

I'm not sure of the long term goals of the classic snap but there are several options on how to fix this for the current situation:

1. add a 'classic' interface that adds rules like those in the workaround and have the classic snap 'plugs: [ classic ]'
2. special case the classic snap in snapd and add rules like those in the workaround unconditionally
3. use 'confinement: classic' for the classic snap

'3' seems like the cleanest way to do this but someone would have to confirm that using 'classic' confinement does what you want for this snap since the mount namespace setup that is used with strict and devmode isn't present with 'classic' confinement (but since 'chroot' is used I suspect that is probably ok).

[Dan Kegel (obdank)]

that workaround seems to have helped, thanks.

[Michael Vogt (mvo)]

Hey Jamie, thanks for your suggestions! Using (3) seems like the best option, however we do not allow installing classic snaps on a Ubuntu Core system (all-snap) system. So I think the next best option is (1).

I pushed https://github.com/snapcore/snapd/pull/2604 with a simple interface like you described above. AIUI to full support this we will:
1. land this interface
2. add a package declaration that allows auto-connect of the "classic" snap with the "classic-dimension" interface

This should also allow using the classic snap in non-devmode (with the limitations that poking around is really limited by apparmor but it should be fine for e.g. building).

[Michael Vogt (mvo)]

Oh and of course

3. add 'plugs: [ classic-dimension ]' to the classic snap :)

[Gustavo Niemeyer (niemeyer)]

Is this about _the_ classic snap, or arbitrary snaps in classic confinement mode?

I take it as the latter, given the description:

"When building a c++ app on a raspberry pi 3 in classic mode,"

But I'm not quite sure.

Do we have this issue in classic confinement, or only in the classic snap in core?

[Oliver Grawert (ogra)]

@gustavo: the latter ... it is essentially about everything you install in --devmode ... (every apparmored command then causes one "ALLOWED" line in the log), the classic snap is worse in that regard since everything you do inside the chroot needs to go through apparmor (because it is a chroot command)

michael has a nice fix for this on github already (which is why i re-assigned the bug)

[Dan Kegel (dank)]

What's the status on this? I'm being bitten by the slow builds again.

[Oliver Grawert (ogra)]

this has been resolved with the latest classic snap in the edge channel ...

[Dan Kegel (dank)]

Tested with iirc

$ sudo snap refresh classic --edge --devmode

Seems much better now, thanks. Looking forward to that landing on stable.

(My test was slightly invalid because I also did 'sudo apt dist-upgrade' inside classic, but it only updated 15 or so packages.)

[Dan Kegel (dank)]

omg it is so much better now. with the new classic, make -j3 keeps three cores pegged solid, as the good $deity intended.

[Mark Shuttleworth (sabdfl)]

:)

[Dan Kegel (dank)]

And -j4 keeps four cores pegged solid. Overhead seems low now.
debuild times went from 11-20 hours down to 2.5 hours for one package.
Happy happy joy joy!

(For the record, I did have to adapt slightly to squeeze a -j4 build of a large c++ project
onto a pi3; I use
-ftrack-macro-expansion=0 --param ggc-min-expand=30 --param ggc-min-heapsize=65536
to reduce gcc's memory consumption. And the i7 still creams it, but what do you want for $35?)