After upgrading to kernel 3.13.0-51 we've been seeing kernel Oops on all our machines during boot.
Example from one of our trusty machines:
[ 12.357276] BUG: unable to handle kernel NULL pointer dereference at (null) [ 12.357886] IP: [<ffffffff8136cb80>] strlen+0x0/0x30 [ 12.358457] PGD 230fe9067 PUD 230d5c067 PMD 0 [ 12.359034] Oops: 0000 [#1] SMP [ 12.359590] Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack crct10dif_pclmul iptable_filter crc32_pclmul ip_tables ghash_clmulni_intel aesni_intel aes_x86_64 ppdev lrw x_tables gf128mul vmwgfx glue_helper ablk_helper cryptd ttm drm vmw_balloon serio_raw shpchp parport_pc lp i2c_piix4 parport mac_hid vmw_vmci psmouse mptspi vmw_pvscsi e1000 mptscsih floppy vmxnet3 mptbase [ 12.364773] CPU: 2 PID: 1718 Comm: fail2ban-server Not tainted 3.13.0-51-generic #84-Ubuntu [ 12.365587] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 [ 12.367276] task: ffff880230fc3000 ti: ffff8802308c4000 task.ti: ffff8802308c4000 [ 12.368159] RIP: 0010:[<ffffffff8136cb80>] [<ffffffff8136cb80>] strlen+0x0/0x30 [ 12.369073] RSP: 0018:ffff8802308c5d60 EFLAGS: 00010212 [ 12.369963] RAX: 000000000000000d RBX: 000000000000000d RCX: 0000000000002df0 [ 12.370973] RDX: 0000000000000012 RSI: 0000000000000000 RDI: 0000000000000000 [ 12.372005] RBP: ffff8802308c5d90 R08: ffff8800b9218648 R09: ffff8802308c5d60 [ 12.372988] R10: 0000000000000002 R11: ffff88023082e180 R12: 0000000000000012 [ 12.373901] R13: 0000000000000000 R14: ffff880231f1b3f8 R15: ffff8800b9218460 [ 12.374827] FS: 00007f196f84c740(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000 [ 12.375752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 12.376667] CR2: 0000000000000000 CR3: 0000000230872000 CR4: 00000000000407e0 [ 12.377684] Stack: [ 12.378612] ffffffff810fb39b 0000000000000000 0000000000000004 ffff88022ff74838 [ 12.379559] ffff8800b9218400 ffff8800b9218460 ffff8802308c5df8 ffffffff810fdb36 [ 12.380516] ffffffff811d56e0 000000042ff74838 ffff880231f1b3c0 ffff88022febecf8 [ 12.381506] Call Trace: [ 12.382630] [<ffffffff810fb39b>] ? audit_compare_dname_path+0x2b/0xa0 [ 12.383784] [<ffffffff810fdb36>] __audit_inode_child+0xb6/0x330 [ 12.384912] [<ffffffff811d56e0>] ? d_instantiate+0x50/0x70 [ 12.386013] [<ffffffff811ca060>] vfs_mknod+0x110/0x160 [ 12.387145] [<ffffffff816bf475>] unix_bind+0x2a5/0x360 [ 12.388207] [<ffffffff810ff142>] ? __audit_sockaddr+0x42/0x80 [ 12.389250] [<ffffffff8160d4c0>] SYSC_bind+0xe0/0x120 [ 12.390297] [<ffffffff8172e9fa>] ? do_page_fault+0x1a/0x70 [ 12.391303] [<ffffffff8160e4de>] SyS_bind+0xe/0x10 [ 12.392426] [<ffffffff817330bd>] system_call_fastpath+0x1a/0x1f [ 12.393581] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 <80> 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 [ 12.396831] RIP [<ffffffff8136cb80>] strlen+0x0/0x30 [ 12.397812] RSP <ffff8802308c5d60> [ 12.398769] CR2: 0000000000000000 [ 12.399743] ---[ end trace 2c5a33d31a03347e ]---
We've also seen this on our precise machines that are running the backported trusty kernel.
When reverting to kernel 3.13.0-49 this no longer occurs.
uname -a: Linux search-2 3.13.0-51-generic #84-Ubuntu SMP Wed Apr 15 12:08:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
cat /proc/version_signature: Ubuntu 3.13.0-51.84-generic 3.13.11-ckt18
After upgrading to kernel 3.13.0-51 we've been seeing kernel Oops on all our machines during boot.
Example from one of our trusty machines:
[ 12.357276] BUG: unable to handle kernel NULL pointer dereference at (null) b80>] strlen+0x0/0x30 vmci_transport vsock ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_ netbios_ ns nf_conntrack_ broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack crct10dif_pclmul iptable_filter crc32_pclmul ip_tables ghash_clmulni_intel aesni_intel aes_x86_64 ppdev lrw x_tables gf128mul vmwgfx glue_helper ablk_helper cryptd ttm drm vmw_balloon serio_raw shpchp parport_pc lp i2c_piix4 parport mac_hid vmw_vmci psmouse mptspi vmw_pvscsi e1000 mptscsih floppy vmxnet3 mptbase ffffffff8136cb8 0>] [<ffffffff8136c b80>] strlen+0x0/0x30 8c5d60 EFLAGS: 00010212 0(0000) GS:ffff88023fd0 0000(0000) knlGS:000000000 0000000 39b>] ? audit_compare_ dname_path+ 0x2b/0xa0 b36>] __audit_ inode_child+ 0xb6/0x330 6e0>] ? d_instantiate+ 0x50/0x70 060>] vfs_mknod+ 0x110/0x160 475>] unix_bind+ 0x2a5/0x360 142>] ? __audit_ sockaddr+ 0x42/0x80 4c0>] SYSC_bind+ 0xe0/0x120 9fa>] ? do_page_ fault+0x1a/ 0x70 4de>] SyS_bind+0xe/0x10 0bd>] system_ call_fastpath+ 0x1a/0x1f b80>] strlen+0x0/0x30
[ 12.357886] IP: [<ffffffff8136c
[ 12.358457] PGD 230fe9067 PUD 230d5c067 PMD 0
[ 12.359034] Oops: 0000 [#1] SMP
[ 12.359590] Modules linked in: tcp_diag inet_diag vmw_vsock_
[ 12.364773] CPU: 2 PID: 1718 Comm: fail2ban-server Not tainted 3.13.0-51-generic #84-Ubuntu
[ 12.365587] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[ 12.367276] task: ffff880230fc3000 ti: ffff8802308c4000 task.ti: ffff8802308c4000
[ 12.368159] RIP: 0010:[<
[ 12.369073] RSP: 0018:ffff880230
[ 12.369963] RAX: 000000000000000d RBX: 000000000000000d RCX: 0000000000002df0
[ 12.370973] RDX: 0000000000000012 RSI: 0000000000000000 RDI: 0000000000000000
[ 12.372005] RBP: ffff8802308c5d90 R08: ffff8800b9218648 R09: ffff8802308c5d60
[ 12.372988] R10: 0000000000000002 R11: ffff88023082e180 R12: 0000000000000012
[ 12.373901] R13: 0000000000000000 R14: ffff880231f1b3f8 R15: ffff8800b9218460
[ 12.374827] FS: 00007f196f84c74
[ 12.375752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 12.376667] CR2: 0000000000000000 CR3: 0000000230872000 CR4: 00000000000407e0
[ 12.377684] Stack:
[ 12.378612] ffffffff810fb39b 0000000000000000 0000000000000004 ffff88022ff74838
[ 12.379559] ffff8800b9218400 ffff8800b9218460 ffff8802308c5df8 ffffffff810fdb36
[ 12.380516] ffffffff811d56e0 000000042ff74838 ffff880231f1b3c0 ffff88022febecf8
[ 12.381506] Call Trace:
[ 12.382630] [<ffffffff810fb
[ 12.383784] [<ffffffff810fd
[ 12.384912] [<ffffffff811d5
[ 12.386013] [<ffffffff811ca
[ 12.387145] [<ffffffff816bf
[ 12.388207] [<ffffffff810ff
[ 12.389250] [<ffffffff8160d
[ 12.390297] [<ffffffff8172e
[ 12.391303] [<ffffffff8160e
[ 12.392426] [<ffffffff81733
[ 12.393581] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 <80> 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80
[ 12.396831] RIP [<ffffffff8136c
[ 12.397812] RSP <ffff8802308c5d60>
[ 12.398769] CR2: 0000000000000000
[ 12.399743] ---[ end trace 2c5a33d31a03347e ]---
We've also seen this on our precise machines that are running the backported trusty kernel.
When reverting to kernel 3.13.0-49 this no longer occurs.
uname -a:
Linux search-2 3.13.0-51-generic #84-Ubuntu SMP Wed Apr 15 12:08:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
cat /proc/version_ signature: 51.84-generic 3.13.11-ckt18
Ubuntu 3.13.0-