AppArmor causes hard lock when free vmalloced memory in interrupt context

[John Johansen]

AppArmor will cause a hard lockup due to calling vfree in interrupt context, backtrace attached below

[John Johansen (jjohansen)]

[ 80.250147] ------------[ cut here ]------------
[ 80.251233] kernel BUG at /home/jj/ubuntu/lucid/ubuntu-lucid/mm/vmalloc.c:1406!
[ 80.252769] invalid opcode: 0000 [#1] SMP
[ 80.253702] last sysfs file: /sys/devices/pci0000:00/0000:00:04.0/class
[ 80.255107] CPU 0
[ 80.255559] Modules linked in: binfmt_misc ppdev lp parport fbcon tileblit font bitblit softcursor psmouse serio_raw vga16fb vgastate i2c_piix4 virtio_blk virtio_net floppy virtio_pci virtio_ring virtio
[ 80.259814] Pid: 671, comm: Xorg Not tainted 2.6.32-20-generic #30 Bochs
[ 80.260019] RIP: 0010:[<ffffffff8111f65c>] [<ffffffff8111f65c>] vfree+0x2c/0x30
[ 80.260019] RSP: 0018:ffff880001c03d50 EFLAGS: 00010206
[ 80.260019] RAX: 0000000000000100 RBX: 0000000000000000 RCX: ffffffff817af230
[ 80.260019] RDX: 000000000020e000 RSI: ffffffff8127df30 RDI: ffffc9000020e000
[ 80.260019] RBP: ffff880001c03d50 R08: 0000000000000000 R09: 0000000000000000
[ 80.260019] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88001cc2b540
[ 80.260019] R13: ffff88001f8c5840 R14: ffff88001f8c5840 R15: ffff880001c10610
[ 80.260019] FS: 00007f919246c700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
[ 80.260019] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 80.260019] CR2: 00007f0a62c70000 CR3: 000000001085a000 CR4: 00000000000006f0
[ 80.260019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 80.260019] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 80.260019] Process Xorg (pid: 671, threadinfo ffff880011db0000, task ffff8800127a44d0)
[ 80.260019] Stack:
[ 80.260019] ffff880001c03d60 ffffffff8127dd15 ffff880001c03d80 ffffffff8127df0a
[ 80.260019] <0> ffff88001cc2b540 ffffffff8127df30 ffff880001c03d90 ffffffff8127df3e
[ 80.260019] <0> ffff880001c03db0 ffffffff812b45d7 ffff880012425800 ffffffff81280e80
[ 80.260019] Call Trace:
[ 80.260019] <IRQ>
[ 80.260019] [<ffffffff8127dd15>] free_table+0x45/0x50
[ 80.260019] [<ffffffff8127df0a>] aa_dfa_free+0x2a/0x50
[ 80.260019] [<ffffffff8127df30>] ? aa_dfa_free_kref+0x0/0x10
[ 80.260019] [<ffffffff8127df3e>] aa_dfa_free_kref+0xe/0x10
[ 80.260019] [<ffffffff812b45d7>] kref_put+0x37/0x70
[ 80.260019] [<ffffffff81280e80>] ? aa_free_profile_kref+0x0/0x20
[ 80.260019] [<ffffffff81280dfa>] aa_free_profile+0xaa/0x130
[ 80.260019] [<ffffffff81280e92>] aa_free_profile_kref+0x12/0x20
[ 80.260019] [<ffffffff812b45d7>] kref_put+0x37/0x70
[ 80.260019] [<ffffffff8127d4ce>] aa_free_task_context+0x2e/0x70
[ 80.260019] [<ffffffff812829ea>] apparmor_cred_free+0x1a/0x30
[ 80.260019] [<ffffffff81250d86>] security_cred_free+0x16/0x20
[ 80.260019] [<ffffffff8108b6e6>] put_cred_rcu+0x26/0xb0
[ 80.260019] [<ffffffff810c9f9d>] __rcu_process_callbacks+0x10d/0x340
[ 80.260019] [<ffffffff81038a61>] ? kvm_clock_read+0x21/0x30
[ 80.260019] [<ffffffff810ca1fb>] rcu_process_callbacks+0x2b/0x50
[ 80.260019] [<ffffffff8106e3a7>] __do_softirq+0xb7/0x1e0
[ 80.260019] [<ffffffff8109445a>] ? tick_program_event+0x2a/0x30
[ 80.260019] [<ffffffff810142ec>] call_softirq+0x1c/0x30
[ 80.260019] [<ffffffff81015cb5>] do_softirq+0x65/0xa0...

[Alex Harrington (alexharrington)]

We think we're seeing this problem - very similar console output after the crash on two HP DL320s servers running 10.04 64 bit. They've been rock solid for 3 years running 8.04 but started crashing about once every two weeks after the upgrade to 10.04.

[John Johansen (jjohansen)]

Alex,

Do you have a log log files, or console output that you can attach? If not test could you take a picture of the your console?

[Alex Harrington (alexharrington)]

John,

I'm afraid I don't. Next time they go I'll grab a photo. I've disabled and uninstalled AppArmor on them and added a further 4GB of RAM which I'm hoping will work around it. Unfortunately they're serving 600 odd workstations in a production environment so I don't want to do anything that's likely to make it happen.

Alex