Kernel OOPS: BUG: unable to handle kernel NULL pointer dereference; IP at ip6_datagram_connect+0x249/0x500

[Michał "rysiek" Woźniak]

We are running Ubuntu 15.10, in a server environment where we have IPsec transport set between servers both for IPv4 and IPv6. We can *reliably* reproduce this error by running:

```
tcpdump "ip and ( host host1.example.com or host host2.example.com or host host3.example.com or host host4.example.com or host host5.example.com )"
```
...where host1-host5.example.com are actual hostnames of servers connected with IPsec transport with the problematic host. Those hosts are currently running either 15.04, or Debian 8, and the affected server is the first one we upgraded to 15.10.

This immediately produces the kernel oops.

Some more info on the host:

```
# uname -a
Linux host6 4.2.0-27-generic #32-Ubuntu SMP Fri Jan 22 04:49:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
```

Version signature:

```
# cat /proc/version_signature
Ubuntu 4.2.0-27.32-generic 4.2.8-ckt1
```

An example oops (more in the attached file):

```
[23882.053990] BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0
[23882.054044] IP: [<ffffffff817bba89>] ip6_datagram_connect+0x249/0x500
[23882.054080] PGD 0
[23882.054103] Oops: 0000 [#7] SMP
[23882.054129] Modules linked in: aufs xt_multiport ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables esp6 ah6 xfrm6_mode_transport nfnetlink_queue nfnetlink_log nfnetlink bluetooth drbg ansi_cprng authenc echainiv esp4 ah4 xfrm4_mode_transport xt_TCPMSS deflate ctr twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_aesni_avx_x86_64 camellia_x86_64 serpent_avx_x86_64 serpent_sse2_x86_64 xts serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_avx_x86_64 cast5_generic cast_common des_generic cmac xcbc rmd160 crypto_null af_key xfrm_algo xt_nat xt_tcpudp veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter
[23882.054488] ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm eeepc_wmi asus_wmi sparse_keymap crct10dif_pclmul crc32_pclmul aesni_intel ppdev shpchp aes_x86_64 lrw gf128mul lpc_ich glue_helper ablk_helper input_leds cryptd parport_pc parport serio_raw tpm_infineon mac_hid 8250_fintek nfsd auth_rpcgss nfs_acl lockd grace sunrpc autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear r8169 ahci libahci mii megaraid_sas wmi video
[23882.054773] CPU: 7 PID: 5954 Comm: tcpdump Tainted: G D 4.2.0-27-generic #32-Ubuntu
[23882.054819] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1106 10/17/2011
[23882.054864] task: ffff8808153ce040 ti: ffff88046c9e0000 task.ti: ffff88046c9e0000
[23882.054907] RIP: 0010:[<ffffffff817bba89>] [<ffffffff817bba89>] ip6_datagram_connect+0x249/0x500
[23882.054955] RSP: 0018:ffff88046c9e3da8 EFLAGS: 00010202
[23882.054980] RAX: ffff880816c10038 RBX: ffff880816c10000 RCX: 000000000000ffff
[23882.055008] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[23882.055036] RBP: ffff88046c9e3e48 R08: ffff880816c10390 R09: ffff880815fc5c80
[23882.055064] R10: ffffffff81cf7c00 R11: 0000000000000002 R12: 0000000000000000
[23882.055092] R13: 0000000000000000 R14: ffff880816c10120 R15: ffff880816c10390
[23882.055121] FS: 00007f4a150af700(0000) GS:ffff88083fbc0000(0000) knlGS:0000000000000000
[23882.055165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[23882.055191] CR2: 00000000000000a0 CR3: 00000000b6161000 CR4: 00000000000406e0
[23882.055219] Stack:
[23882.055241] ffff880816c10390 ffff880816c10038 0000000000000000 00000000d23f16ed
[23882.055291] ffff8800b6477980 0000000100000000 0011000000000000 f804012a00000000
[23882.055341] 00000000f5511001 f804012a02000000 000000002f845101 0000000002000000
[23882.055390] Call Trace:
[23882.055420] [<ffffffff8175b6a1>] inet_dgram_connect+0x41/0x80
[23882.055451] [<ffffffff816c8879>] SYSC_connect+0xd9/0x110
[23882.055483] [<ffffffff8121b895>] ? fd_install+0x25/0x30
[23882.055511] [<ffffffff816c7734>] ? sock_map_fd+0x44/0x70
[23882.055540] [<ffffffff816c961e>] SyS_connect+0xe/0x10
[23882.055569] [<ffffffff817f1c72>] entry_SYSCALL_64_fastpath+0x16/0x75
[23882.055598] Code: ff ff ff 4c 8b 85 60 ff ff ff 49 89 47 28 4d 89 47 30 41 f6 85 17 01 00 00 40 0f 85 ae 01 00 00 41 f6 45 60 10 0f 85 7e 02 00 00 <49> 8b 85 a0 00 00 00 48 85 c0 0f 84 67 02 00 00 8b 40 2c 41 89
[23882.055768] RIP [<ffffffff817bba89>] ip6_datagram_connect+0x249/0x500
[23882.055801] RSP <ffff88046c9e3da8>
[23882.055824] CR2: 00000000000000a0
[23882.056185] ---[ end trace 91f389eb505db06a ]---
```

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1545031

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.5 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5-rc3-wily/

[Michał "rysiek" Woźniak (rysiek+launchpad)]

@Brad we would feel much more comfortable with providing the logfiles needed manually due to nature of the work we do. What's missing? What's needed?

@Joseph this started happening directly after dist-upgrade from 15.04 to 15.10; the previous kernel version was whatever 15.04 was using, and it was stable, no problems there. The setup was *identical*.

We might consider testing the new kernel, I'll get back to you on that.

[Michał "rysiek" Woźniak (rysiek+launchpad)]

As suggested I have tested on the mainline kernel:

# uname -a
Linux host6 4.5.0-040500rc3-generic #201602071930 SMP Mon Feb 8 00:34:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I can confirm the bug is *fixed* in mainline.

However, this is not a solution for us, as we are using docker with the `aufs` backing storage driver, and `aufs` is not supported in the mainline kernel.

[Michał "rysiek" Woźniak (rysiek+launchpad)]

We have downgraded to kernel 3.19.x on the affected host and can also confirm the bug is not present there:

# uname -a
Linux host6 3.19.0-47-generic #53-Ubuntu SMP Mon Jan 18 14:02:48 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Currently we will have to run the 3.19-series kernel until the bug gets fixed. We are unable to use a mainstream kernel on that host, as we rely on aufs, which is not available in that kernel.