Kernel panic when we call pipework to setup virtual network for docker containers

[Peng]

I found a kernel panic issue when I was using pipework to config the network of a docker container on an x86 board with all-snap image. The issue is related to the auditing module of Linux kernel. So it should be an issue of pc-kernel-snap.

I created a simple test snap to reproduce the issue and upload it to github.
https://github.com/pliu6/docker-snap-test

Software environment to reproduce the bug:
#snap list
Name Version Rev Developer Notes
core 16.04.1 714 canonical -
docker 1.11.2-9 56 canonical devmode
dockertest 0.0.1 x12 devmode
pc 16.04-0.8 9 canonical -
pc-kernel 4.4.0-53-2 51 canonical -

The log is below:
[ 504.783341] BUG: unable to handle kernel paging request at fffffffffffffff3
[ 504.867186] IP: [<ffffffff813fe6c0>] strlen+0x0/0x20
[ 504.926879] PGD 1e0d067 PUD 1e0f067 PMD 0
[ 504.976588] Oops: 0000 [#1] SMP
[ 505.015690] Modules linked in: veth xt_addrtype br_netfilter ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_conntrack
iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc overlay aufs arc4 ath9k ath9k_common ath9k_hw ath mac80211 c
fg80211 kvm_amd uas kvm irqbypass k10temp r8169 mii sp5100_tco mac_hid i2c_piix4 shpchp iptable_filter ip_tables ip6table_filter ip6_tables x_tables aut
ofs4 mmc_block sdhci_acpi sdhci_pci sdhci virtio_scsi nls_iso8859_1 usb_storage ahci libahci
[ 505.599099] CPU: 1 PID: 2414 Comm: snap-confine Not tainted 4.4.0-53-generic #74-Ubuntu
[ 505.694977] Hardware name: PC Engines APU, BIOS SageBios_PCEngines_APU-45 04/05/2014
[ 505.787738] task: ffff880037637080 ti: ffff880061a70000 task.ti: ffff880061a70000
[ 505.877382] RIP: 0010:[<ffffffff813fe6c0>] [<ffffffff813fe6c0>] strlen+0x0/0x20
[ 505.966192] RSP: 0018:ffff880061a73a20 EFLAGS: 00010246
[ 506.029835] RAX: ffff880061a73b20 RBX: fffffffffffffff3 RCX: 0000000000000000
[ 506.115320] RDX: 000000000000014e RSI: fffffffffffffff3 RDI: fffffffffffffff3
[ 506.200802] RBP: ffff880061a73a38 R08: ffff88005c835138 R09: ffff880061a73a94
[ 506.286283] R10: 000000000000000e R11: ffff88005c835131 R12: ffff88007aff0480
[ 506.371767] R13: ffff880037637080 R14: ffffffff81399fc0 R15: 00000000fffffff3
[ 506.457251] FS: 00007fa9f36aa740(0000) GS:ffff88007df00000(0000) knlGS:0000000000000000
[ 506.554170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 506.623014] CR2: fffffffffffffff3 CR3: 000000007853e000 CR4: 00000000000006e0
[ 506.708497] Stack:
[ 506.732624] ffffffff81122a1a ffff88007aff0480 ffff880061a73b00 ffff880061a73a60
[ 506.822056] ffffffff8139a028 ffff88007aff0480 ffff880061a73b00 ffff880037637080
[ 506.911490] ffff880061a73ad8 ffffffff8136f088 ffffffff812285c0 ffff880061a73af0
[ 507.000926] Call Trace:
[ 507.030263] [<ffffffff81122a1a>] ? audit_log_untrustedstring+0x1a/0x30
[ 507.109502] [<ffffffff8139a028>] audit_cb+0x68/0x3f0
[ 507.170027] [<ffffffff8136f088>] common_lsm_audit+0x1b8/0x740
[ 507.239910] [<ffffffff812285c0>] ? alloc_inode+0x50/0x90
[ 507.304593] [<ffffffff812265b6>] ? prepend_path+0xc6/0x2a0
[ 507.371358] [<ffffffff8138161f>] aa_audit+0x5f/0x170
[ 507.431880] [<ffffffff81399fb2>] audit_mount+0x152/0x160
[ 507.496567] [<ffffffff8139a67d>] match_mnt_path_str+0x1dd/0x490
[ 507.568529] [<ffffffff812278b8>] ? dentry_path+0x18/0x70
[ 507.633213] [<ffffffff8139aa0a>] match_mnt+0xda/0x150
[ 507.694776] [<ffffffff8139b280>] aa_bind_mount+0x100/0x180
[ 507.761540] [<ffffffff813903e0>] wrap_apparmor_sb_mount+0x1c0/0x270
[ 507.837664] [<ffffffff81345db7>] security_sb_mount+0x57/0x80
[ 507.906506] [<ffffffff8123029b>] do_mount+0xab/0xde0
[ 507.967032] [<ffffffff811efd74>] ? __kmalloc_track_caller+0x1b4/0x250
[ 508.045236] [<ffffffff810ef581>] ? hrtimer_try_to_cancel+0xd1/0x130
[ 508.121361] [<ffffffff811acc72>] ? memdup_user+0x42/0x70
[ 508.186042] [<ffffffff812312ff>] SyS_mount+0x9f/0x100
[ 508.247607] [<ffffffff81836072>] entry_SYSCALL_64_fastpath+0x16/0x71
[ 508.324765] Code: 89 f8 48 89 e5 f6 82 a0 05 a5 81 20 74 10 48 83 c0 01 0f b6 10 f6 82 a0 05 a5 81 20 75 f0 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 <8
0> 3f 00 55 48 89 e5 74 11 48 89 f8 48 83 c0 01 80 38 00 75 f7
[ 508.564156] RIP [<ffffffff813fe6c0>] strlen+0x0/0x20
[ 508.624889] RSP <ffff880061a73a20>
[ 508.666696] CR2: fffffffffffffff3
[ 508.706425] ---[ end trace 9a8196367a1a3630 ]---

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1651944

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Peng (pengliu6)]

The bug was found on the all-snap image. Therefore I was not able to run apport-collect command.

[Peng (pengliu6)]

Changed the target to be apparmor because the kernel panic is triggered by apparmor.

[Seth Arnold (seth-arnold)]

I added linux back due to the uncanny resemblance to:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1508737
and
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1586997

Thanks

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1651944

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.10 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc2

[John Johansen (jjohansen)]

Ignore the request to test the upstream kernel, for the moment.

In this case the apparmor code that is in the trace does not exist upstream. Instead could you test the kernel in
  http://people.canonical.com/~jj/lp1648143/

While listed as being for bug 1648143, it contains several fixes including a fix to the bind mount code. That will be pushed up to the ubuntu kernel this week.

If this still exhibits the fault then, please test the upstream kernel to verify that the bug is indeed in apparmor, and not being triggered else where and showing up in apparmor.

[Peng (pengliu6)]

Actually, this bug was found with Ubuntu all-snap version. So I am not able to try the kernel.deb. Could you tell me how to get the latest version of the kernel snap for 16.04?
Thanks!

[John Johansen (jjohansen)]

sudo snap refresh <package>

should refresh the kernel snap. However the suspected fix will not be in any snap kernel, nor can I atm build you a kernel snap to test with.