BUG: unable to handle kernel NULL pointer dereference (aa_label_merge)

[Roger Lehmann]

I had a kernel problem with one of my high availability VMs. This was not the first occurence of the problem, it seems to appear every now and then.
The VM froze because of the Kernel bug appended as a text file. I had to hard reset it.
I am running Ubuntu 14.04.2 Kernel 3.16.0-34-generic with libapache2-mod-apparmor 2.8.95~2430-0ubuntu5.1 and apache 2.4.7-1ubuntu4.4 (mpm_prefork).
Unfortunately I am unable to reproduce the bug at will. This is a serious problem for high availability VMs.
Looking at the trace, I assume this is a bug in AppArmor.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 15 14:13 seq
 crw-rw---- 1 root audio 116, 33 May 15 14:13 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=UUID=69e1aff7-c3d1-4bf4-8685-bd3fc2942f60
InstallationDate: Installed on 2015-03-25 (61 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.16.0-37-generic root=UUID=fea3075f-a9a3-414b-992a-b812d86cd04e ro
ProcVersionSignature: Ubuntu 3.16.0-37.51~14.04.1-generic 3.16.7-ckt9
RelatedPackageVersions:
 linux-restricted-modules-3.16.0-37-generic N/A
 linux-backports-modules-3.16.0-37-generic N/A
 linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.16.0-37-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

WifiSyslog:

_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: rel-1.7.5.1-0-g8936dbb-20141113_115728-nilsson.home.kraxel.org
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-2.2
dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.7.5.1-0-g8936dbb-20141113_115728-nilsson.home.kraxel.org:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.2:cvnQEMU:ct1:cvrpc-i440fx-2.2:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-2.2
dmi.sys.vendor: QEMU

[Kees Cook (kees)]

I saw the same thing this morning.

Ubuntu 14.04.2 LTS
Kernel linux-image-3.16.0-30-generic
libapache2-mod-apparmor 2.8.95~2430-0ubuntu5.1
apache2-mpm-prefork 2.4.7-1ubuntu4.4

Attached is first the warning (like in bug 1447530), and then the Oops, 9 seconds later.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1448912

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Roger Lehmann (roger-lehmann-u)]

apport information

[Roger Lehmann (roger-lehmann-u)]

apport information

[Roger Lehmann (roger-lehmann-u)]

apport information

[Roger Lehmann (roger-lehmann-u)]

apport information

[Roger Lehmann (roger-lehmann-u)]

apport information

[Roger Lehmann (roger-lehmann-u)]

apport information

[Roger Lehmann (roger-lehmann-u)]

apport information

[Roger Lehmann (roger-lehmann-u)]

apport information

[Kees Cook (kees)]

Three more crashes today, one after the other (the trigger must be some kind of Apache access pattern, still trying to figure that out.)

[Kees Cook (kees)]

Here is the Oops from a jj-special kernel. similar, but different.

[John Johansen (jjohansen)]

Yep, thanks. I think I've got it now. I am working on a fix

[Aidan Hobson Sayers (aidanhs)]

Not sure if this is the same issue or not, my null pointer is *not* in aa_label_merge - "BUG: unable to handle kernel NULL pointer dereference (aa_file_perm)".

It does have some significant similarities in the trace though. I've had this on 14.04 kernel 3.16 and 15.04 kernel 3.19.

[Aidan Hobson Sayers (aidanhs)]

I should note that the above is using AUFS (via docker) - https://github.com/docker/docker/issues/14816

[Kees Cook (kees)]

The debug kernels reduced the frequency of the Oopsing, but this has made AppArmor unusable with Apache for me.

[John Johansen (jjohansen)]

there are new test images in
http://people.canonical.com/~jj/lp1448912/

there has been some testing of these and so far they appear to fix the issue

[Kees Cook (kees)]

It's been 3 days running the test kernel and I've seen no problems. Very encouraging!

[Kees Cook (kees)]

Still no issues for me. Yay! :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-9.24

---------------
linux (4.4.0-9.24) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1551319

  * AppArmor logs denial for when the device path is ENOENT (LP: #1482943)
    - SAUCE: apparmor: fix log of apparmor audit message when kern_path() fails

  * BUG: unable to handle kernel NULL pointer dereference (aa_label_merge) (LP:
    #1448912)
    - SAUCE: apparmor: Fix: insert race between label_update and label_merge
    - SAUCE: apparmor: Fix: ensure aa_get_newest will trip debugging if the
      replacedby is not setup
    - SAUCE: apparmor: Fix: label merge handling of marking unconfined and stale
    - SAUCE: apparmor: Fix: refcount race between locating in labelset and get
    - SAUCE: apparmor: Fix: ensure new labels resulting from merge have a
      replacedby
    - SAUCE: apparmor: Fix: label_vec_merge insertion
    - SAUCE: apparmor: Fix: deadlock in aa_put_label() call chain
    - SAUCE: apparmor: Fix: add required locking of __aa_update_replacedby on
      merge path
    - SAUCE: apparmor: Fix: convert replacedby update to be protected by the
      labelset lock
    - SAUCE: apparmor: Fix: update replacedby allocation to take a gfp parameter

  * apparmor kernel BUG kills firefox (LP: #1430546)
    - SAUCE: apparmor: Disallow update of cred when then subjective != the
      objective cred
    - SAUCE: apparmor: rework retrieval of the current label in the profile update
      case

  * sleep from invalid context in aa_move_mount (LP: #1539349)
    - SAUCE: apparmor: fix sleep from invalid context

  * s390x: correct restore of high gprs on signal return (LP: #1550468)
    - s390/compat: correct restore of high gprs on signal return

  * missing SMAP support (LP: #1550517)
    - x86/entry/compat: Add missing CLAC to entry_INT80_32

  * Floating-point exception handler receives empty Data-Exception Code in
    Floating Point Control register (LP: #1548414)
    - s390/fpu: signals vs. floating point control register

  * kvm fails to boot GNU Hurd kernels with 4.4 Xenial kernel (LP: #1550596)
    - KVM: x86: fix conversion of addresses to linear in 32-bit protected mode

  * Surelock GA2 SP1: capiredp01: cxl_init_adapter fails for CAPI devices
    0000:01:00.0 and 0005:01:00.0 after upgrading to 840.10 Platform firmware
    build fips840/b1208b_1604.840 (LP: #1532914)
    - cxl: Fix PSL timebase synchronization detection

  * [Feature]EDAC support for Knights Landing (LP: #1519631)
    - EDAC, sb_edac: Set fixed DIMM width on Xeon Knights Landing

  * Various failures of kernel_security suite on Xenial kernel on s390x arch
    (LP: #1531327)
    - [config] s390x -- CONFIG_DEFAULT_MMAP_MIN_ADDR=65536

  * Unable to install VirtualBox Guest Service in 15.04 (LP: #1434579)
    - [Config] Provides: virtualbox-guest-modules when appropriate

  * linux is missing provides for virtualbox-guest-modules [i386 amd64 x32] (LP:
    #1507588)
    - [Config] Provides: virtualbox-guest-modules when appropriate

  * Backport more recent driver for SKL, KBL and BXT graphics (LP: #1540390)
    - SAUCE: i915_bpo: Provide a backport driver for SKL, KBL & BXT graphics
    - SA...

[Tyler Hicks (tyhicks)]

I've marked the releases which have went end-of-life as "Won't Fix". I'm not sure if John is planning on SRU'ing the fix to Trusty so I'll leave that bug task alone.