Trusty kernel 3.13.0-124.173, current on -proposed, has a regression introduced by the patches for bug#1690094.
This is causing the openvswitch autopkgtest testcase to hang: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-trusty/trusty/amd64/o/openvswitch/20170710_093748_a5f0a@/log.gz
because of the following kernel bug:
====================================================================== [ 16.371056] random: nonblocking pool is initialized [ 22.187846] gre: GRE over IPv4 demultiplexor driver [ 22.190604] openvswitch: Open vSwitch switching datapath [ 27.919352] gre: GRE over IPv4 demultiplexor driver [ 27.920959] openvswitch: Open vSwitch switching datapath [ 28.230062] IPv6: ADDRCONF(NETDEV_UP): s1-eth1: link is not ready [ 28.255859] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth1: link becomes ready [ 28.294253] IPv6: ADDRCONF(NETDEV_UP): s1-eth2: link is not ready [ 28.310647] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth2: link becomes ready [ 28.341512] IPv6: ADDRCONF(NETDEV_UP): s1-eth3: link is not ready [ 28.358174] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth3: link becomes ready [ 28.386774] IPv6: ADDRCONF(NETDEV_UP): s1-eth4: link is not ready [ 28.402249] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth4: link becomes ready [ 28.431979] IPv6: ADDRCONF(NETDEV_UP): s1-eth5: link is not ready [ 28.446848] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth5: link becomes ready [ 28.482404] IPv6: ADDRCONF(NETDEV_UP): s1-eth6: link is not ready [ 28.498450] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth6: link becomes ready [ 28.530356] IPv6: ADDRCONF(NETDEV_UP): s1-eth7: link is not ready [ 28.546206] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth7: link becomes ready [ 28.578384] IPv6: ADDRCONF(NETDEV_UP): s2-eth1: link is not ready [ 28.594164] IPv6: ADDRCONF(NETDEV_CHANGE): s2-eth1: link becomes ready [ 28.623885] IPv6: ADDRCONF(NETDEV_UP): s2-eth2: link is not ready [ 28.642570] IPv6: ADDRCONF(NETDEV_CHANGE): s2-eth2: link becomes ready [ 28.669156] IPv6: ADDRCONF(NETDEV_UP): s2-eth3: link is not ready [ 28.683255] IPv6: ADDRCONF(NETDEV_CHANGE): s2-eth3: link becomes ready [ 28.716490] IPv6: ADDRCONF(NETDEV_UP): s2-eth4: link is not ready [ 28.734239] IPv6: ADDRCONF(NETDEV_CHANGE): s2-eth4: link becomes ready [ 28.763779] IPv6: ADDRCONF(NETDEV_UP): s2-eth5: link is not ready [ 28.782221] IPv6: ADDRCONF(NETDEV_CHANGE): s2-eth5: link becomes ready [ 28.810559] IPv6: ADDRCONF(NETDEV_UP): s2-eth6: link is not ready [ 28.826181] IPv6: ADDRCONF(NETDEV_CHANGE): s2-eth6: link becomes ready [ 28.856232] IPv6: ADDRCONF(NETDEV_UP): s2-eth7: link is not ready [ 28.875082] IPv6: ADDRCONF(NETDEV_CHANGE): s2-eth7: link becomes ready [ 28.901120] IPv6: ADDRCONF(NETDEV_UP): s1-eth8: link is not ready [ 28.909372] IPv6: ADDRCONF(NETDEV_CHANGE): s1-eth8: link becomes ready [ 28.986164] device ovs-system entered promiscuous mode [ 29.001788] device s1 entered promiscuous mode [ 29.021015] BUG: unable to handle kernel NULL pointer dereference at 00000000000000a8 [ 29.021600] IP: [<ffffffff8163f61b>] if_nlmsg_size+0xfb/0x240 [ 29.021990] PGD 3bfec067 PUD 36c7e067 PMD 0 [ 29.022303] Oops: 0000 [#1] SMP [ 29.022540] Modules linked in: veth openvswitch gre vxlan ip_tunnel libcrc32c 9p ppdev kvm_intel kvm 9pnet_virtio serio_raw 9pnet parport_pc parport i2c_piix4 mac_hid psmouse floppy pata_acpi [last unloaded:] [ 29.023992] CPU: 0 PID: 2255 Comm: ovs-vswitchd Not tainted 3.13.0-124-generic #173-Ubuntu [ 29.024012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014 [ 29.024012] task: ffff88003d5f1800 ti: ffff880036c7c000 task.ti: ffff880036c7c000 [ 29.024012] RIP: 0010:[<ffffffff8163f61b>] [<ffffffff8163f61b>] if_nlmsg_size+0xfb/0x240 [ 29.024012] RSP: 0018:ffff880036c7d850 EFLAGS: 00010286 [ 29.024012] RAX: ffff88003bb41000 RBX: ffff88003ac9b000 RCX: 00000000000000d0 [ 29.024012] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88003ac9b000 [ 29.024012] RBP: ffff880036c7d888 R08: 0000000000000000 R09: 0000000000000000 [ 29.024012] R10: 00000000000044ed R11: 006d65747379732d R12: 0000000000000344 [ 29.024012] R13: 0000000000000000 R14: ffffffffa011c000 R15: 0000000000000014 [ 29.024012] FS: 00007f2a595e3980(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000 [ 29.024012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 29.024012] CR2: 00000000000000a8 CR3: 000000003d5ff000 CR4: 00000000000006f0 [ 29.024012] Stack: [ 29.024012] ffff88003bb41000 0000000000000000 00000000000000d0 0000000000000010 [ 29.024012] ffffffff81cdaf00 0000000000000000 ffff88003ac9b000 ffff880036c7d8d8 [ 29.024012] ffffffff81642f82 735f7265776f6cfa ff00316874652d31 ffff88003ac9b000 [ 29.024012] Call Trace: [ 29.024012] [<ffffffff81642f82>] rtmsg_ifinfo_build_skb+0x32/0xe0 [ 29.024012] [<ffffffff81643046>] rtmsg_ifinfo+0x16/0x50 [ 29.024012] [<ffffffff816430b9>] rtnetlink_event+0x39/0x40 [ 29.024012] [<ffffffff81739f1c>] notifier_call_chain+0x4c/0x70 [ 29.024012] [<ffffffff81093566>] raw_notifier_call_chain+0x16/0x20 [ 29.024012] [<ffffffff8162b085>] call_netdevice_notifiers_info+0x35/0x60 [ 29.024012] [<ffffffff81631f1b>] __netdev_upper_dev_link+0x3fb/0x4a0 [ 29.024012] [<ffffffff81631ff5>] netdev_master_upper_dev_link+0x15/0x20 [ 29.024012] [<ffffffffa015557c>] netdev_create+0xdc/0x170 [openvswitch] [ 29.024012] [<ffffffffa015482a>] ovs_vport_add+0x4a/0xd0 [openvswitch] [ 29.024012] [<ffffffffa014c222>] new_vport+0x12/0x50 [openvswitch] [ 29.024012] [<ffffffffa014ec6a>] ovs_vport_cmd_new+0x12a/0x220 [openvswitch] [ 29.024012] [<ffffffff8165f79d>] genl_family_rcv_msg+0x18d/0x370 [ 29.024012] [<ffffffff8165f980>] ? genl_family_rcv_msg+0x370/0x370 [ 29.024012] [<ffffffff8165fa11>] genl_rcv_msg+0x91/0xd0 [ 29.024012] [<ffffffff8165dab9>] netlink_rcv_skb+0xa9/0xc0 [ 29.024012] [<ffffffff8165dfb8>] genl_rcv+0x28/0x40 [ 29.024012] [<ffffffff8165d1aa>] netlink_unicast+0xda/0x1b0 [ 29.024012] [<ffffffff8165d58e>] netlink_sendmsg+0x30e/0x680 [ 29.024012] [<ffffffff8165a2b4>] ? netlink_rcv_wake+0x44/0x60 [ 29.024012] [<ffffffff8165b367>] ? netlink_recvmsg+0x1c7/0x3c0 [ 29.024012] [<ffffffff8161659b>] sock_sendmsg+0x8b/0xc0 [ 29.024012] [<ffffffff81632d38>] ? __netdev_update_features+0x18/0x380 [ 29.024012] [<ffffffff81634357>] ? ethtool_get_value+0x37/0x60 [ 29.024012] [<ffffffff816362ca>] ? dev_ethtool+0xd5a/0x1c80 [ 29.024012] [<ffffffff816169a9>] ___sys_sendmsg+0x389/0x3a0 [ 29.024012] [<ffffffff8163e4ce>] ? rtnl_unlock+0xe/0x10 [ 29.024012] [<ffffffff81645b0e>] ? dev_ioctl+0x1ce/0x590 [ 29.024012] [<ffffffff8165be43>] ? netlink_table_ungrab+0x33/0x40 [ 29.024012] [<ffffffff8165c755>] ? netlink_insert+0x145/0x240 [ 29.024012] [<ffffffff81613995>] ? sock_do_ioctl+0x45/0x50 [ 29.024012] [<ffffffff81613ec0>] ? sock_ioctl+0x1f0/0x2c0 [ 29.024012] [<ffffffff81617792>] __sys_sendmsg+0x42/0x80 [ 29.024012] [<ffffffff816177e2>] SyS_sendmsg+0x12/0x20 [ 29.024012] [<ffffffff8173e35d>] system_call_fastpath+0x1a/0x1f [ 29.024012] Code: c0 74 10 48 89 df ff d0 83 c0 07 83 e0 fc 48 98 49 01 c7 48 89 df e8 45 b5 fe ff 48 85 c0 0f 84 e9 00 00 00 48 8b 90 08 07 00 00 <48> 8b 8a a8 00 00 00 31 d2 48 85 c9 74 0c 48 89 de 48 89 c [ 29.024012] RIP [<ffffffff8163f61b>] if_nlmsg_size+0xfb/0x240 [ 29.024012] RSP <ffff880036c7d850> [ 29.024012] CR2: 00000000000000a8 [ 29.046589] ---[ end trace 19b0345dcdf6940a ]--- ======================================================================
Trusty kernel 3.13.0-124.173, current on -proposed, has a regression introduced by the patches for bug#1690094.
This is causing the openvswitch autopkgtest testcase to hang: /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- trusty/ trusty/ amd64/o/ openvswitch/ 20170710_ 093748_ a5f0a@/ log.gz
https:/
because of the following kernel bug:
======= ======= ======= ======= ======= ======= ======= ======= ======= ======= NETDEV_ UP): s1-eth1: link is not ready NETDEV_ CHANGE) : s1-eth1: link becomes ready NETDEV_ UP): s1-eth2: link is not ready NETDEV_ CHANGE) : s1-eth2: link becomes ready NETDEV_ UP): s1-eth3: link is not ready NETDEV_ CHANGE) : s1-eth3: link becomes ready NETDEV_ UP): s1-eth4: link is not ready NETDEV_ CHANGE) : s1-eth4: link becomes ready NETDEV_ UP): s1-eth5: link is not ready NETDEV_ CHANGE) : s1-eth5: link becomes ready NETDEV_ UP): s1-eth6: link is not ready NETDEV_ CHANGE) : s1-eth6: link becomes ready NETDEV_ UP): s1-eth7: link is not ready NETDEV_ CHANGE) : s1-eth7: link becomes ready NETDEV_ UP): s2-eth1: link is not ready NETDEV_ CHANGE) : s2-eth1: link becomes ready NETDEV_ UP): s2-eth2: link is not ready NETDEV_ CHANGE) : s2-eth2: link becomes ready NETDEV_ UP): s2-eth3: link is not ready NETDEV_ CHANGE) : s2-eth3: link becomes ready NETDEV_ UP): s2-eth4: link is not ready NETDEV_ CHANGE) : s2-eth4: link becomes ready NETDEV_ UP): s2-eth5: link is not ready NETDEV_ CHANGE) : s2-eth5: link becomes ready NETDEV_ UP): s2-eth6: link is not ready NETDEV_ CHANGE) : s2-eth6: link becomes ready NETDEV_ UP): s2-eth7: link is not ready NETDEV_ CHANGE) : s2-eth7: link becomes ready NETDEV_ UP): s1-eth8: link is not ready NETDEV_ CHANGE) : s1-eth8: link becomes ready 61b>] if_nlmsg_ size+0xfb/ 0x240 ffffffff8163f61 b>] [<ffffffff8163f 61b>] if_nlmsg_ size+0xfb/ 0x240 c7d850 EFLAGS: 00010286 0(0000) GS:ffff88003fc0 0000(0000) knlGS:000000000 0000000 f82>] rtmsg_ifinfo_ build_skb+ 0x32/0xe0 046>] rtmsg_ifinfo+ 0x16/0x50 0b9>] rtnetlink_ event+0x39/ 0x40 f1c>] notifier_ call_chain+ 0x4c/0x70 566>] raw_notifier_ call_chain+ 0x16/0x20 085>] call_netdevice_ notifiers_ info+0x35/ 0x60 f1b>] __netdev_ upper_dev_ link+0x3fb/ 0x4a0 ff5>] netdev_ master_ upper_dev_ link+0x15/ 0x20 57c>] netdev_ create+ 0xdc/0x170 [openvswitch] 82a>] ovs_vport_ add+0x4a/ 0xd0 [openvswitch] 222>] new_vport+0x12/0x50 [openvswitch] c6a>] ovs_vport_ cmd_new+ 0x12a/0x220 [openvswitch] 79d>] genl_family_ rcv_msg+ 0x18d/0x370 980>] ? genl_family_ rcv_msg+ 0x370/0x370 a11>] genl_rcv_ msg+0x91/ 0xd0 ab9>] netlink_ rcv_skb+ 0xa9/0xc0 fb8>] genl_rcv+0x28/0x40 1aa>] netlink_ unicast+ 0xda/0x1b0 58e>] netlink_ sendmsg+ 0x30e/0x680 2b4>] ? netlink_ rcv_wake+ 0x44/0x60 367>] ? netlink_ recvmsg+ 0x1c7/0x3c0 59b>] sock_sendmsg+ 0x8b/0xc0 d38>] ? __netdev_ update_ features+ 0x18/0x380 357>] ? ethtool_ get_value+ 0x37/0x60 2ca>] ? dev_ethtool+ 0xd5a/0x1c80 9a9>] ___sys_ sendmsg+ 0x389/0x3a0 4ce>] ? rtnl_unlock+ 0xe/0x10 b0e>] ? dev_ioctl+ 0x1ce/0x590 e43>] ? netlink_ table_ungrab+ 0x33/0x40 755>] ? netlink_ insert+ 0x145/0x240 995>] ? sock_do_ ioctl+0x45/ 0x50 ec0>] ? sock_ioctl+ 0x1f0/0x2c0 792>] __sys_sendmsg+ 0x42/0x80 7e2>] SyS_sendmsg+ 0x12/0x20 35d>] system_ call_fastpath+ 0x1a/0x1f 61b>] if_nlmsg_ size+0xfb/ 0x240 ======= ======= ======= ======= ======= ======= ======= ======= =======
[ 16.371056] random: nonblocking pool is initialized
[ 22.187846] gre: GRE over IPv4 demultiplexor driver
[ 22.190604] openvswitch: Open vSwitch switching datapath
[ 27.919352] gre: GRE over IPv4 demultiplexor driver
[ 27.920959] openvswitch: Open vSwitch switching datapath
[ 28.230062] IPv6: ADDRCONF(
[ 28.255859] IPv6: ADDRCONF(
[ 28.294253] IPv6: ADDRCONF(
[ 28.310647] IPv6: ADDRCONF(
[ 28.341512] IPv6: ADDRCONF(
[ 28.358174] IPv6: ADDRCONF(
[ 28.386774] IPv6: ADDRCONF(
[ 28.402249] IPv6: ADDRCONF(
[ 28.431979] IPv6: ADDRCONF(
[ 28.446848] IPv6: ADDRCONF(
[ 28.482404] IPv6: ADDRCONF(
[ 28.498450] IPv6: ADDRCONF(
[ 28.530356] IPv6: ADDRCONF(
[ 28.546206] IPv6: ADDRCONF(
[ 28.578384] IPv6: ADDRCONF(
[ 28.594164] IPv6: ADDRCONF(
[ 28.623885] IPv6: ADDRCONF(
[ 28.642570] IPv6: ADDRCONF(
[ 28.669156] IPv6: ADDRCONF(
[ 28.683255] IPv6: ADDRCONF(
[ 28.716490] IPv6: ADDRCONF(
[ 28.734239] IPv6: ADDRCONF(
[ 28.763779] IPv6: ADDRCONF(
[ 28.782221] IPv6: ADDRCONF(
[ 28.810559] IPv6: ADDRCONF(
[ 28.826181] IPv6: ADDRCONF(
[ 28.856232] IPv6: ADDRCONF(
[ 28.875082] IPv6: ADDRCONF(
[ 28.901120] IPv6: ADDRCONF(
[ 28.909372] IPv6: ADDRCONF(
[ 28.986164] device ovs-system entered promiscuous mode
[ 29.001788] device s1 entered promiscuous mode
[ 29.021015] BUG: unable to handle kernel NULL pointer dereference at 00000000000000a8
[ 29.021600] IP: [<ffffffff8163f
[ 29.021990] PGD 3bfec067 PUD 36c7e067 PMD 0
[ 29.022303] Oops: 0000 [#1] SMP
[ 29.022540] Modules linked in: veth openvswitch gre vxlan ip_tunnel libcrc32c 9p ppdev kvm_intel kvm 9pnet_virtio serio_raw 9pnet parport_pc parport i2c_piix4 mac_hid psmouse floppy pata_acpi [last unloaded:]
[ 29.023992] CPU: 0 PID: 2255 Comm: ovs-vswitchd Not tainted 3.13.0-124-generic #173-Ubuntu
[ 29.024012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 29.024012] task: ffff88003d5f1800 ti: ffff880036c7c000 task.ti: ffff880036c7c000
[ 29.024012] RIP: 0010:[<
[ 29.024012] RSP: 0018:ffff880036
[ 29.024012] RAX: ffff88003bb41000 RBX: ffff88003ac9b000 RCX: 00000000000000d0
[ 29.024012] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88003ac9b000
[ 29.024012] RBP: ffff880036c7d888 R08: 0000000000000000 R09: 0000000000000000
[ 29.024012] R10: 00000000000044ed R11: 006d65747379732d R12: 0000000000000344
[ 29.024012] R13: 0000000000000000 R14: ffffffffa011c000 R15: 0000000000000014
[ 29.024012] FS: 00007f2a595e398
[ 29.024012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 29.024012] CR2: 00000000000000a8 CR3: 000000003d5ff000 CR4: 00000000000006f0
[ 29.024012] Stack:
[ 29.024012] ffff88003bb41000 0000000000000000 00000000000000d0 0000000000000010
[ 29.024012] ffffffff81cdaf00 0000000000000000 ffff88003ac9b000 ffff880036c7d8d8
[ 29.024012] ffffffff81642f82 735f7265776f6cfa ff00316874652d31 ffff88003ac9b000
[ 29.024012] Call Trace:
[ 29.024012] [<ffffffff81642
[ 29.024012] [<ffffffff81643
[ 29.024012] [<ffffffff81643
[ 29.024012] [<ffffffff81739
[ 29.024012] [<ffffffff81093
[ 29.024012] [<ffffffff8162b
[ 29.024012] [<ffffffff81631
[ 29.024012] [<ffffffff81631
[ 29.024012] [<ffffffffa0155
[ 29.024012] [<ffffffffa0154
[ 29.024012] [<ffffffffa014c
[ 29.024012] [<ffffffffa014e
[ 29.024012] [<ffffffff8165f
[ 29.024012] [<ffffffff8165f
[ 29.024012] [<ffffffff8165f
[ 29.024012] [<ffffffff8165d
[ 29.024012] [<ffffffff8165d
[ 29.024012] [<ffffffff8165d
[ 29.024012] [<ffffffff8165d
[ 29.024012] [<ffffffff8165a
[ 29.024012] [<ffffffff8165b
[ 29.024012] [<ffffffff81616
[ 29.024012] [<ffffffff81632
[ 29.024012] [<ffffffff81634
[ 29.024012] [<ffffffff81636
[ 29.024012] [<ffffffff81616
[ 29.024012] [<ffffffff8163e
[ 29.024012] [<ffffffff81645
[ 29.024012] [<ffffffff8165b
[ 29.024012] [<ffffffff8165c
[ 29.024012] [<ffffffff81613
[ 29.024012] [<ffffffff81613
[ 29.024012] [<ffffffff81617
[ 29.024012] [<ffffffff81617
[ 29.024012] [<ffffffff8173e
[ 29.024012] Code: c0 74 10 48 89 df ff d0 83 c0 07 83 e0 fc 48 98 49 01 c7 48 89 df e8 45 b5 fe ff 48 85 c0 0f 84 e9 00 00 00 48 8b 90 08 07 00 00 <48> 8b 8a a8 00 00 00 31 d2 48 85 c9 74 0c 48 89 de 48 89 c
[ 29.024012] RIP [<ffffffff8163f
[ 29.024012] RSP <ffff880036c7d850>
[ 29.024012] CR2: 00000000000000a8
[ 29.046589] ---[ end trace 19b0345dcdf6940a ]---
=======