kernel crash in ipv4_dst_destroy

[Tom Eichhorn]

Hi,

we have seen a GPF resulting in ipv4_dst_destroy:

messages-2015-11-08:Nov 8 20:23:05 bbk130129 kernel: [205505.064821] device tap406dbc6f-a8 left promiscuous mode
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205505.534610] general protection fault: 0000 [#1] SMP
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205505.578139] Modules linked in: kvm_intel kvm vport_vxlan vport_gre gre openvswitch dm_multipath scsi_dh nbd sch_fq_codel xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat bridge ebtable_filter ebtables ipmi_devintf btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c vhost_net vhost macvtap macvlan ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_limit xt_multiport xt_CT iptable_raw xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables mlx4_en(OE) mlx4_core(OE) mlx_compat(OE) isofs ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich hpilo ioatdma ipmi_si 8250_fintek ipmi_msghandler wmi acpi_power_meter mac_hid shpchp bonding 8021q garp mrp stp llc lp parport nls_iso8859_1 ixgbe dca vxlan ip6_udp_tunnel tg3 udp_tunnel mpt3sas(OE) raid_class ptp configfs pps_core mdio scsi_transport_sas
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.275481] CPU: 42 PID: 258 Comm: rcuos/35 Tainted: G OE 3.19.0-32-generic #37~14.04.1-Ubuntu
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.349044] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.404252] task: ffff882fa5823110 ti: ffff882fa5848000 task.ti: ffff882fa5848000
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.463760] RIP: 0010:[<ffffffff816de782>] [<ffffffff816de782>] ipv4_dst_destroy+0x42/0x80
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.527658] RSP: 0018:ffff882fa584bde8 EFLAGS: 00010246
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.578843] RAX: dead000000200200 RBX: ffff882f1e961200 RCX: 0000000000000060
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.636886] RDX: dead000000100100 RSI: 00000000fffffe01 RDI: ffffffff81fc5d84
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.696883] RBP: ffff882fa584bdf8 R08: ffff882f1e961200 R09: 00000001802a000b
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.756193] R10: ffffffff816b30f5 R11: ffffea00bc7a5800 R12: 0000000000000000
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.814938] R13: 0000000000000001 R14: ffff882f1e961080 R15: ffff882fa5823110
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.878741] FS: 0000000000000000(0000) GS:ffff885fbf440000(0000) knlGS:0000000000000000
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.943046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.997859] CR2: 00007ffd99ac3640 CR3: 0000000002c16000 CR4: 00000000001427e0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.057616] Stack:
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.093614] 0000000000000000 ffff882f1e961200 ffff882fa584be18 ffffffff816b30d2
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.155762] ffff882f1e961200 ffff882fbfcf4bc0 ffff882fa584be28 ffffffff816b33ae
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.217537] ffff882fa584beb8 ffffffff810d542f ffff882fa5823110 ffff882fa584be78
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.281530] Call Trace:
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.321761] [<ffffffff816b30d2>] dst_destroy+0x32/0xe0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.373239] [<ffffffff816b33ae>] dst_destroy_rcu+0xe/0x20
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.424174] [<ffffffff810d542f>] rcu_nocb_kthread+0x19f/0x590
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.474698] [<ffffffff810b4e30>] ? prepare_to_wait_event+0x110/0x110
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.530284] [<ffffffff810d5290>] ? rcu_spawn_all_nocb_kthreads.part.54+0x160/0x160
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.589559] [<ffffffff81093822>] kthread+0xd2/0xf0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.635397] [<ffffffff81093750>] ? kthread_create_on_node+0x1c0/0x1c0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.691201] [<ffffffff817b6d18>] ret_from_fork+0x58/0x90
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.741015] [<ffffffff81093750>] ? kthread_create_on_node+0x1c0/0x1c0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.798002] Code: 08 48 39 87 b0 00 00 00 74 4f 48 c7 c7 84 5d fc 81 e8 a3 82 0d 00 48 8b 83 b8 00 00 00 48 8b 93 b0 00 00 00 48 c7 c7 84 5d fc 81 <48> 89 42 08 48 89 10 48 b8 00 01 10 00 00 00 ad de 48 89 83 b0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205508.001730] RSP <ffff882fa584bde8>
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205508.087352] ---[ end trace 645de9b56a152109 ]---

What has happened is that openstack has destroyed a VM and a network interface has been removed from the kernel:

auth-2015-11-08.log:Nov 8 20:23:01 bbk130129 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf mm-ctl --unbind-port 406dbc6f-a8fb-42f1-8f8d-a668698c7676
auth-2015-11-08.log:Nov 8 20:23:01 bbk130129 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
auth-2015-11-08.log:Nov 8 20:23:05 bbk130129 sudo: pam_unix(sudo:session): session closed for user root
auth-2015-11-08.log:Nov 8 20:23:05 bbk130129 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link delete tap406dbc6f-a8

Some googling had brought up http://lists.openwall.net/netdev/2014/01/22/29 and http://lists.openwall.net/netdev/2014/01/18/5,
which might here also be the problem.

I have not yet been able to reproduce this bug, but will update this bug if it happens again.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.19.0-32-generic 3.19.0-32.37~14.04.1
ProcVersionSignature: Ubuntu 3.19.0-32.37~14.04.1-generic 3.19.8-ckt7
Uname: Linux 3.19.0-32-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.18
Architecture: amd64
Date: Mon Nov 9 09:09:04 2015
SourcePackage: linux-lts-vivid
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[csmcd5 (geek-n)]

On our product we have recently seen kernel crashes showing the same call chain as above. It happened on two different hosts only 26 minutes apart.

We have a Python program using scapy/sniff. Its effect is to repeatedly put all interfaces into promiscuous mode and then take them out. There are about 10 interfaces, mostly vlan, and it does not create/delete interfaces. This has been running on 7 hosts every 1-3 minutes for days.

Linux ir6 3.16.0-44-iridium #59~14.04.1 SMP PREEMPT Tue May 23 01:10:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I've seen other Google results for crashes in ipv4_dst_destroy that are preceded by "promiscuous mode" notices in the kernel log.