AppArmor: general protection fault: 0000 [#1] SMP

[Alban Crequy]

How to reproduce it:

1. Create a concatenated cache: (it might not be a correct thing to do, but the kernel is not supposed to crash anyway)

. /lib/apparmor/functions
recache_profiles
rm -f "${PROFILES}"/cache/.ubercache
for i in "${PROFILES}"/cache/*; do
  cat "$i" >> "${PROFILES}"/cache/.ubercache
done

2. sudo /etc/init.d/apparmor teardown

3. sudo /sbin/apparmor_parser --add -B /etc/apparmor.d/cache/.ubercache
   => $?=239 (EEXIST)
The issue with EEXIST is how apparmor_parser uploads the binary profiles to the kernel in parser/parser_interface.c sd_load_buffer().

If /etc/apparmor.d/cache/.ubercache contains n profiles, sd_load_buffer() was performing n write() to /sys/kernel/security/apparmor/.load in this way:

1. write() with all profiles in the range [1, n]
2. write() with all profiles in the range [2, n]
3. write() with all profiles in the range [3, n]
...
n. write() with the n-th profile

The first write() is successful and the n-1 other write() return EEXIST because the profiles is already uploaded in the first write().

Comment near parser/parser_interface.c sd_load_buffer():
/* bleah the kernel should just loop and do multiple load, but to support
 * older systems we need to do this

4. sudo /sbin/apparmor_parser --replace -B /etc/apparmor.d/cache/.ubercache
   => segmentation fault (because of the kernel crash)

5. sudo /sbin/apparmor_parser --replace -B /etc/apparmor.d/cache/.ubercache
   => freeze in state "D"

ProblemType: KernelOops
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-24-generic 3.13.0-24.47
ProcVersionSignature: Ubuntu 3.13.0-24.47-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
Annotation: Your system might become unstable now and might need to be restarted.
ApportVersion: 2.13.3-0ubuntu1
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Thu May 8 15:43:47 2014
Failure: oops
HibernationDevice: RESUME=UUID=ae00639c-b206-4387-b731-1a52e58547cf
InstallationDate: Installed on 2014-03-21 (48 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140321)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 002: ID 80ee:0021 VirtualBox USB Tablet
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: innotek GmbH VirtualBox
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-24-generic root=UUID=1ec94169-4f22-4000-90d6-c14f61a59998 ro quiet splash vt.handoff=7
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions: kerneloops-daemon 0.12+git20090217-3ubuntu6
RfKill:

SourcePackage: linux
Title: general protection fault: 0000 [#1] SMP
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH

[Brad Figg (brad-figg)]

This change was made by a bot.

[Tyler Hicks (tyhicks)]

Assigning this to jj and subscribing ubuntu-security, since it is an AppArmor kernel bug, so that it doesn't get lost.

[Alban Crequy (muadda)]

This kernel oops was produced without patching apparmor or the kernel. But if you are curious, here is a bit more context why I created a concatenated cache:

I want to make the boot faster on a system which does not need to check whether the profiles changed. I have a single cache file (named .ubercache) for all profiles and I get it uploaded in the kernel by apparmor_parser in a single write() syscall. I have a patch on parser/parser_interface.c sd_load_buffer() to avoid multiple write().

[Brad Figg (brad-figg)]

This change was made by a bot.

[John Johansen (jjohansen)]

Alban, I'd be interested in seeing what you patch to sd_load_buffer() looks like. We have just landed a patch upstream to fix the multiple write problem for 2.8, 29, and dev.

There are two things to note about profile loads that have multiple profiles in them.
1. Older kernels don't actually support it.
2. The load is treated as an atomic set. That is either all profiles within the set must load/replace successfully or none of them will

[Alban Crequy (muadda)]

My patch would not work with older kernel then... But here it is.

I believe the line:
- bsize = size - (b - buffer);
is not correct. In the first iteration of the loop, b==buffer, so bsize==size. So it is sending all the profiles in a single write() syscall. The second iteration will have write() return EEXIST, unless "--replace" is used. But in Ubuntu upstart scripts, "--replace" is always used.

[John Johansen (jjohansen)]

Right, the add option will return EEXIST if any profile exists, and fails the load. This is why --replace is used it will load or replace.

In upstream kernel 3.12 is the first to support loading multiple profiles, in a single load.

Ubuntu has been carrying (and testing the patch) since Saucy (3.11 based kernel).

When the kernel supports loading multiple profiles it exports the file (assuming securityfs is mounted at /sys/kernel/security)
   /sys/kernel/security/apparmor/features/policy/set_load

Previous kernels will just ignore extra data after the first profile (multiple profile load was always planned).

Concatenating profile cache files into a single file should work, as long as each file is a valid cache file, and the load is a single atomic write. This can be done by writing the file to the raw interface directly or via the parser using the -B option. The parser has the logic to break apart the large file if the kernel doesn't support multiple profile loads, so I would recommend using it, but it isn't required.

I'm still looking into what exactly is causing the kernel oops.

[John Johansen (jjohansen)]

The upstream commit that fixes the multiple profile load problem for the parser is

commit 2510 on the dev branch or
commit 2127 on the 2.8 branch

[John Johansen (jjohansen)]

Alban,

can you attach your concatted uber cache file? This has worked (not oops) for me with every combination of the cache I throw at it.

Alban also note that the parser caching doesn't currently support auto rebuilding, etc of caches with this uber cache yet. Are you interested in beta testing a patch that provides native support for this?

[John Johansen (jjohansen)]

Just an update as I did some testing on concated cache timings, on a kernel that supports multiple profile loads

so for 100 iterations of profile loading I got
2.8.95 parser, profile set from cache: 0m22.961s
2.8.95 parser, profile set concated: 2m16.653s
patched parser, profile set from cache: 0m15.365s
patched parser, profile set concated: 0m8.108s

So with out the above mentioned parser patch, I would avoid using concated caches as it results in a significant slow down
With the patch doing regular caching, the load is about 33% faster. This is partly because there are fewer writes but also because some of the cache files do have multiple profiles in them (very much like the concated cache file).

The concated cache with the patch isn't quite 2x as fast as regular caching. Hopefully this will help you make an informed decision as to whether this is something you want to continue doing atm.

With that said, there are other benefits to multiple profiles in the same cache file that we will be taking advantage of in the future. Atomic set replacement, the ability to share dfas and other parts of the policy for better compression, etc. So better support for doing this stuff is coming.

[Alban Crequy (muadda)]

Attached: the compressed apparmor binary cache file that cause the oops.

apparmor 2.8.95~2430-0ubuntu5
linux-image-3.13.0-24-generic 3.13.0-24.47