Kernel crash in vrouter enforces a node reboot

[Martin Gerhard Loschwitz]

After applying the fix for https://bugs.launchpad.net/juniperopenstack/r1.1/+bug/1398484 we're seeing another crash which appears to be unrelated to the first one:

[240362.566841] CPU: 12 PID: 42965 Comm: contrail-vroute Tainted: G OX 3.13.0-40-generic #69-Ubuntu
[240362.566902] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 08/26/2014
[240362.566937] task: ffff882c0b494800 ti: ffff8822a537e000 task.ti: ffff8822a537e000
[240362.574071] RIP: 0010:[<ffffffff8161fb69>] [<ffffffff8161fb69>] netif_napi_del+0x19/0x60
[240362.574130] RSP: 0018:ffff8822a537f680 EFLAGS: 00010202
[240362.574161] RAX: ffff88228ecfd528 RBX: ffff88228ecfd4e0 RCX: 0000000000000000
[240362.574215] RDX: 0000000000000000 RSI: 000000000000035a RDI: ffff88228ecfd4e0
[240362.574270] RBP: ffff8822a537f688 R08: ffff88228ecfd4dc R09: ffff882154049000
[240362.574324] R10: ffff881fff403300 R11: ffffffffa0636314 R12: ffff88228ecfd4f0
[240362.574378] R13: ffff88228ecfd400 R14: 00000000ffffffed R15: 0000000000000001
[240362.574434] FS: 00007fdb5cff3700(0000) GS:ffff883fff200000(0000) knlGS:0000000000000000
[240362.574489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[240362.574522] CR2: 0000000000000008 CR3: 0000002a3993e000 CR4: 00000000001427e0
[240362.574578] Stack:
[240362.574604] ffff88228ecfd400 ffff8822a537f6a8 ffffffffa063d6d9 ffff88228ecfd400
[240362.574674] 0000000000000000 ffff8822a537f6c8 ffffffffa0645658 ffff8822a537f6c8
[240362.574743] ffff88228ecfd400 ffff8822a537f6e0 ffffffffa06458fe ffff88228ecfda00
[240362.574812] Call Trace:
[240362.574849] [<ffffffffa063d6d9>] linux_if_del+0xd9/0x100 [vrouter]
[240362.574889] [<ffffffffa0645658>] eth_drv_del+0x38/0x60 [vrouter]
[240362.574926] [<ffffffffa06458fe>] vif_drv_delete+0x3e/0x60 [vrouter]
[240362.574965] [<ffffffffa0647144>] vr_interface_add+0x444/0x4e0 [vrouter]
[240362.575003] [<ffffffffa06369cb>] ? vr_interface_req_read+0x6fb/0x740 [vrouter]
[240362.575060] [<ffffffffa06472ba>] vr_interface_req_process+0xda/0x250 [vrouter]
[240362.575117] [<ffffffffa0637fba>] sandesh_decode_one+0x10a/0x1d0 [vrouter]
[240362.575158] [<ffffffffa0639e00>] ? thrift_transport_flush+0x20/0x20 [vrouter]
[240362.575215] [<ffffffffa0639e10>] ? thrift_memory_buffer_is_open+0x10/0x10 [vrouter]
[240362.575273] [<ffffffffa0639e20>] ? thrift_memory_buffer_open+0x10/0x10 [vrouter]
[240362.575332] [<ffffffffa0639e60>] ? thrift_memory_buffer_flush+0x10/0x10 [vrouter]
[240362.575390] [<ffffffffa0639e30>] ? thrift_memory_buffer_close+0x10/0x10 [vrouter]
[240362.575451] [<ffffffffa0639ef0>] ? thrift_memory_buffer_read+0x90/0x90 [vrouter]
[240362.575512] [<ffffffffa0639e40>] ? thrift_memory_buffer_read_end+0x10/0x10 [vrouter]
[240362.575571] [<ffffffffa0639e50>] ? thrift_memory_buffer_write_end+0x10/0x10 [vrouter]
[240362.575630] [<ffffffffa0638e80>] ? thrift_binary_protocol_write_message_begin+0x80/0x80 [vrouter]
[240362.575692] [<ffffffffa0638ce0>] ? thrift_binary_protocol_write_message_end+0x10/0x10 [vrouter]
[240362.575753] [<ffffffffa0638e00>] ? thrift_binary_protocol_read_double+0x10/0x10 [vrouter]
[240362.575811] [<ffffffffa0638cd0>] ? thrift_protocol_skip+0x270/0x270 [vrouter]
[240362.575867] [<ffffffffa0638cf0>] ? thrift_binary_protocol_write_sandesh_end+0x10/0x10 [vrouter]
[240362.575926] [<ffffffffa0638d00>] ? thrift_binary_protocol_write_struct_begin+0x10/0x10 [vrouter]
[240362.575985] [<ffffffffa0638fc0>] ? thrift_binary_protocol_write_bool+0x20/0x20 [vrouter]
[240362.576045] [<ffffffffa0638d10>] ? thrift_binary_protocol_write_struct_end+0x10/0x10 [vrouter]
[240362.576104] [<ffffffffa0638ea0>] ? thrift_binary_protocol_write_sandesh_begin+0x20/0x20 [vrouter]
[240362.576163] [<ffffffffa0638ec0>] ? thrift_binary_protocol_write_field_stop+0x20/0x20 [vrouter]
[240362.576223] [<ffffffffa0638d20>] ? thrift_binary_protocol_write_field_end+0x10/0x10 [vrouter]
[240362.576283] [<ffffffffa0638f40>] ? thrift_binary_protocol_write_map_begin+0x80/0x80 [vrouter]
[240362.576343] [<ffffffffa0638d30>] ? thrift_binary_protocol_write_map_end+0x10/0x10 [vrouter]
[240362.576402] [<ffffffffa0639020>] ? thrift_binary_protocol_write_field_begin+0x60/0x60 [vrouter]
[240362.576463] [<ffffffffa0638d40>] ? thrift_binary_protocol_write_list_end+0x10/0x10 [vrouter]
[240362.576522] [<ffffffffa0638fa0>] ? thrift_binary_protocol_write_list_begin+0x60/0x60 [vrouter]
[240362.576580] [<ffffffffa0639030>] ? thrift_binary_protocol_write_set_begin+0x10/0x10 [vrouter]
[240362.576638] [<ffffffffa06391d0>] ? thrift_binary_protocol_write_binary+0x80/0x80 [vrouter]
[240362.576698] [<ffffffffa0639250>] ? thrift_binary_protocol_write_u16+0x40/0x40 [vrouter]
[240362.576756] [<ffffffffa0639060>] ? thrift_binary_protocol_write_byte+0x30/0x30 [vrouter]
[240362.576814] [<ffffffffa0639210>] ? thrift_binary_protocol_write_i16+0x40/0x40 [vrouter]
[240362.576873] [<ffffffffa0639280>] ? thrift_binary_protocol_write_i32+0x30/0x30 [vrouter]
[240362.576932] [<ffffffffa06390c0>] ? thrift_binary_protocol_write_i64+0x60/0x60 [vrouter]
[240362.576990] [<ffffffffa06392b0>] ? thrift_binary_protocol_write_u32+0x30/0x30 [vrouter]
[240362.577050] [<ffffffffa0638d50>] ? thrift_binary_protocol_write_set_end+0x10/0x10 [vrouter]
[240362.577107] [<ffffffffa06392c0>] ? thrift_binary_protocol_write_ipv4+0x10/0x10 [vrouter]
[240362.577180] [<ffffffffa0639150>] ? thrift_binary_protocol_write_uuid_t+0x30/0x30 [vrouter]
[240362.577237] [<ffffffffa06392c0>] ? thrift_binary_protocol_write_ipv4+0x10/0x10 [vrouter]
[240362.577293] [<ffffffffa0639120>] ? thrift_binary_protocol_write_u64+0x60/0x60 [vrouter]
[240362.577349] [<ffffffffa06393e0>] ? thrift_binary_protocol_read_message_begin+0xd0/0xd0 [vrouter]
[240362.577407] [<ffffffffa0638d70>] ? thrift_binary_protocol_read_message_end+0x10/0x10 [vrouter]
[240362.577463] [<ffffffffa0639310>] ? thrift_binary_protocol_write_string+0x50/0x50 [vrouter]
[240362.577520] [<ffffffffa0638d60>] ? thrift_binary_protocol_write_double+0x10/0x10 [vrouter]
[240362.577577] [<ffffffffa0638d80>] ? thrift_binary_protocol_read_sandesh_end+0x10/0x10 [vrouter]
[240362.577635] [<ffffffffa0638da0>] ? thrift_binary_protocol_read_struct_begin+0x20/0x20 [vrouter]
[240362.577695] [<ffffffffa0639540>] ? thrift_binary_protocol_read_list_begin+0x90/0x90 [vrouter]
[240362.577751] [<ffffffffa0638db0>] ? thrift_binary_protocol_read_struct_end+0x10/0x10 [vrouter]
[240362.577809] [<ffffffffa0639400>] ? thrift_binary_protocol_read_sandesh_begin+0x20/0x20 [vrouter]
[240362.577868] [<ffffffffa0638dc0>] ? thrift_binary_protocol_read_field_end+0x10/0x10 [vrouter]
[240362.577924] [<ffffffffa06394b0>] ? thrift_binary_protocol_read_map_begin+0xb0/0xb0 [vrouter]
[240362.577980] [<ffffffffa0638dd0>] ? thrift_binary_protocol_read_map_end+0x10/0x10 [vrouter]
[240362.578036] [<ffffffffa06395d0>] ? thrift_binary_protocol_read_field_begin+0x90/0x90 [vrouter]
[240362.578094] [<ffffffffa0638de0>] ? thrift_binary_protocol_read_list_end+0x10/0x10 [vrouter]
[240362.578152] [<ffffffffa06395e0>] ? thrift_binary_protocol_read_set_begin+0x10/0x10 [vrouter]
[240362.578210] [<ffffffffa0639630>] ? thrift_binary_protocol_read_bool+0x50/0x50 [vrouter]
[240362.578265] [<ffffffffa0639680>] ? thrift_binary_protocol_read_byte+0x50/0x50 [vrouter]
[240362.578320] [<ffffffffa06396d0>] ? thrift_binary_protocol_read_i16+0x50/0x50 [vrouter]
[240362.578378] [<ffffffffa0639720>] ? thrift_binary_protocol_read_i32+0x50/0x50 [vrouter]
[240362.578434] [<ffffffffa06397a0>] ? thrift_binary_protocol_read_i64+0x80/0x80 [vrouter]
[240362.578489] [<ffffffffa06397f0>] ? thrift_binary_protocol_read_u16+0x50/0x50 [vrouter]
[240362.578545] [<ffffffffa0639850>] ? thrift_binary_protocol_read_ipv4+0x10/0x10 [vrouter]
[240362.578602] [<ffffffffa0639840>] ? thrift_binary_protocol_read_u32+0x50/0x50 [vrouter]
[240362.578657] [<ffffffffa0638df0>] ? thrift_binary_protocol_read_set_end+0x10/0x10 [vrouter]
[240362.578713] [<ffffffffa06399e0>] ? thrift_binary_protocol_read_binary+0xe0/0xe0 [vrouter]
[240362.578770] [<ffffffffa0639900>] ? thrift_binary_protocol_read_uuid_t+0x30/0x30 [vrouter]
[240362.578825] [<ffffffffa06399e0>] ? thrift_binary_protocol_read_binary+0xe0/0xe0 [vrouter]
[240362.578883] [<ffffffffa06398d0>] ? thrift_binary_protocol_read_u64+0x80/0x80 [vrouter]
[240362.578938] [<ffffffffa0637e90>] ? sandesh_hdr_free+0x10/0x10 [vrouter]
[240362.578974] [<ffffffffa0638136>] sandesh_decode+0x46/0x90 [vrouter]
[240362.579012] [<ffffffffa06404f4>] sandesh_proto_decode+0x24/0x30 [vrouter]
[240362.579051] [<ffffffffa063ff6e>] vr_message_request+0x3e/0x60 [vrouter]
[240362.579089] [<ffffffffa063f9c6>] netlink_trans_request+0x56/0x190 [vrouter]
[240362.579131] [<ffffffff8165276d>] genl_family_rcv_msg+0x18d/0x370
[240362.579165] [<ffffffff81652950>] ? genl_family_rcv_msg+0x370/0x370
[240362.579199] [<ffffffff816529e1>] genl_rcv_msg+0x91/0xd0
[240362.579232] [<ffffffff81650a69>] netlink_rcv_skb+0xa9/0xc0
[240362.579265] [<ffffffff81650f68>] genl_rcv+0x28/0x40
[240362.579298] [<ffffffff81650065>] netlink_unicast+0xd5/0x1b0
[240362.579332] [<ffffffff81650460>] netlink_sendmsg+0x320/0x770
[240362.579367] [<ffffffff8160abeb>] sock_sendmsg+0x8b/0xc0
[240362.579404] [<ffffffff811b2116>] ? __mem_cgroup_commit_charge+0x156/0x3d0
[240362.579438] [<ffffffff8160a8de>] ? move_addr_to_kernel.part.16+0x1e/0x60
[240362.586483] [<ffffffff8160b4a1>] ? move_addr_to_kernel+0x21/0x30
[240362.586517] [<ffffffff8160b473>] ___sys_sendmsg+0x3c3/0x3d0
[240362.586553] [<ffffffff8172b324>] ? __do_page_fault+0x204/0x560
[240362.586589] [<ffffffff81182215>] ? change_protection+0x65/0xb0
[240362.586622] [<ffffffff811823b1>] ? mprotect_fixup+0x151/0x290
[240362.586656] [<ffffffff8160bb72>] __sys_sendmsg+0x42/0x80
[240362.586689] [<ffffffff8160bbc2>] SyS_sendmsg+0x12/0x20
[240362.586723] [<ffffffff8172fd2d>] system_call_fastpath+0x1a/0x1f
[240362.586756] Code: 00 00 48 85 c0 74 05 48 89 df ff d0 5b 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 8d 47 48 48 89 e5 53 48 8b 4f 48 48 8b 57 50 48 89 fb <48> 89 51 08 48 89 0a 48 89 47 48 48 89 47 50 48 8b 7f 40 e8 af
[240362.587054] RIP [<ffffffff8161fb69>] netif_napi_del+0x19/0x60
[240362.587092] RSP <ffff8822a537f680>
[240362.587120] CR2: 0000000000000008
[240362.587478] ---[ end trace c7d2a283c66e5e35 ]---

[Martin Gerhard Loschwitz (martin-loschwitz)]

Additional information:

Dec 5 09:51:12 bka-001-08 kernel: [240362.565887] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
Dec 5 09:51:12 bka-001-08 kernel: [240362.565955] IP: [<ffffffff8161fb69>] netif_napi_del+0x19/0x60
Dec 5 09:51:12 bka-001-08 kernel: [240362.566001] PGD 215429c067 PUD 3fbdf48067 PMD 0
Dec 5 09:51:12 bka-001-08 kernel: [240362.566042] Oops: 0002 [#1] SMP
Dec 5 09:51:12 bka-001-08 kernel: [240362.566079] Modules linked in: vrouter(OX) veth vhost_net vhost macvtap macvlan joydev hid_generic usbhid hid ipt_MASQUERADE ipt_REJECT xt_CHECKSUM bridge ip6table_filter ip6_tables ebtable_nat ebtables xt_LOG xt_limit xt_tcpudp xt_comment xt_multiport xt_conntrack iptable_filter iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_crypt x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich kvm_intel crct10dif_pclmul kvm crc32_pclmul ghash_clmulni_intel dm_multipath aesni_intel scsi_dh aes_x86_64 lrw ipmi_devintf gf128mul ioatdma glue_helper bonding ablk_helper serio_raw 8021q hpwdt hpilo cryptd garp ipmi_si acpi_power_meter stp lpc_ich mrp llc sch_fq_codel lp mac_hid parport xfs btrfs xor raid6_pq libcrc32c ixgbe tg3 psmouse dca ptp pps_core hpsa mdio wmi [last unloaded: vrouter]

[Martin Gerhard Loschwitz (martin-loschwitz)]

Now seen on a second node:

253182.725246] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[253182.725298] IP: [<ffffffff8161fb69>] netif_napi_del+0x19/0x60
[253182.725333] PGD 17c6575067 PUD 1f8994b067 PMD 0
[253182.725367] Oops: 0002 [#1] SMP
[253182.725405] Modules linked in: vrouter(OX) veth vhost_net vhost macvtap macvlan joydev hid_generic usbhid hid ipt_MASQUERADE ipt_REJECT xt_CHECKSUM bridge ip6table_filter ip6_tables ebtable_nat ebtables iptable_nat nf_nat_ipv4 nf_nat iptable_mangle xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack nf_conntrack iptable_filter ip_tables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_crypt x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel gpio_ich kvm ipmi_devintf bonding xfs crct10dif_pclmul 8021q crc32_pclmul ghash_clmulni_intel aesni_intel garp aes_x86_64 dm_multipath hpilo ioatdma lrw stp gf128mul mrp llc ipmi_si lpc_ich scsi_dh serio_raw hpwdt acpi_power_meter lp glue_helper mac_hid parport ablk_helper cryptd sch_fq_codel btrfs xor raid6_pq libcrc32c ixgbe tg3 psmouse dca ptp pps_core hpsa mdio wmi [last unloaded: vrouter]
[253182.726083] CPU: 14 PID: 34661 Comm: contrail-vroute Tainted: G OX 3.13.0-40-generic #69-Ubuntu
[253182.726127] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 08/26/2014
[253182.726164] task: ffff881ec66bb000 ti: ffff881db56b8000 task.ti: ffff881db56b8000
[253182.726204] RIP: 0010:[<ffffffff8161fb69>] [<ffffffff8161fb69>] netif_napi_del+0x19/0x60
[253182.726250] RSP: 0018:ffff881db56b9680 EFLAGS: 00010202
[253182.726273] RAX: ffff883fcfd12128 RBX: ffff883fcfd120e0 RCX: 0000000000000000
[253182.726312] RDX: 0000000000000000 RSI: 00000000000000b6 RDI: ffff883fcfd120e0
[253182.726352] RBP: ffff881db56b9688 R08: ffff883fcfd120dc R09: ffff883f87e14000
[253182.726392] R10: ffff881fff403300 R11: ffffffffa0594314 R12: ffff883fcfd120f0
[253182.726431] R13: ffff883fcfd12000 R14: 00000000ffffffed R15: 0000000000000001
[253182.726481] FS: 00007f6bf73fc700(0000) GS:ffff883fff240000(0000) knlGS:0000000000000000
[253182.726522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[253182.726546] CR2: 0000000000000008 CR3: 0000001fd0d68000 CR4: 00000000001427e0
[253182.726585] Stack:
[253182.726604] ffff883fcfd12000 ffff881db56b96a8 ffffffffa059b6d9 ffff883fcfd12000
[253182.726659] 0000000000000000 ffff881db56b96c8 ffffffffa05a3658 ffff881db56b96c8
[253182.726712] ffff883fcfd12000 ffff881db56b96e0 ffffffffa05a38fe ffff883fcfd10400
[253182.726765] Call Trace:
[253182.726792] [<ffffffffa059b6d9>] linux_if_del+0xd9/0x100 [vrouter]
[253182.726821] [<ffffffffa05a3658>] eth_drv_del+0x38/0x60 [vrouter]
[253182.726849] [<ffffffffa05a38fe>] vif_drv_delete+0x3e/0x60 [vrouter]
[253182.726877] [<ffffffffa05a5144>] vr_interface_add+0x444/0x4e0 [vrouter]
[253182.726906] [<ffffffffa05949cb>] ? vr_interface_req_read+0x6fb/0x740 [vrouter]
[253182.726948] [<ffffffffa05a52ba>] vr_interface_req_process+0xda/0x250 [vrouter]
[253182.726990] [<ffffffffa0595fba>] sandesh_decode_one+0x10a/0x1d0 [vrouter]
[253182.727020] [<ffffffffa0597e0...

[Anand H. Krishnan (anandhk)]

Problem is understood. It seems like vif add is failing and the resultant cleanup results in the crash because the napi add didn't happen in the first place. Fix could be to add a simple check.

[Martin Gerhard Loschwitz (martin-loschwitz)]

Please let us know if we can do anything to help with finding a solution to this issue. It keeps our nodes in an unstable state as it crashes the kernel hard enough to even make "reboot -f" have no effect, so this is a real showstopper.

[Martin Gerhard Loschwitz (martin-loschwitz)]

We have now deployed the following patch supplied by Pedro:

diff --git a/dp-core/vr_interface.c b/dp-core/vr_interface.c
index f4a08e8..51fcd82 100644
--- a/dp-core/vr_interface.c
+++ b/dp-core/vr_interface.c
@@ -1407,7 +1407,9 @@ vr_interface_add(vr_interface_req *req, bool need_response)

     ret = vif_drv_add(vif, req);
     if (ret) {
- vif_delete(vif);
+ vr_printf(
+ "interface %s add (type %x, flags %x) failed with error: %d\n",
+ req->vifr_name, vif->vif_type, vif->vif_flags, ret);
         vif = NULL;
     }

This will work around the problem by not calling the problematic vif_delete function that was breaking the kernel in the first place. Pedro, however, believes that the problem is similar to https://bugs.launchpad.net/juniperopenstack/+bug/1399603 and based on the vrouter module making assumptions that were true in previous gcc versions but are no longer true now.

The patch above will produce debug output instead of crashing the machine; we will report back on the debug output as soon as we get it.

[Anand H. Krishnan (anandhk)]

Patch for the fix is under review. While the patch will fix the crash, we still need to understand why the interface addition is failing. If you have the output of the debug from Pedro's patch, please do send that to us. We might need the contrail-vrouter logs too (under /var/log).

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5372
Committed: http://github.org/Juniper/contrail-vrouter/commit/1d5faf97fdd4a3e4919304a04c9cbf1813c77a05
Submitter: Zuul
Branch: master

commit 1d5faf97fdd4a3e4919304a04c9cbf1813c77a05
Author: Anand H Krishnan <email address hidden>
Date: Sun Dec 7 22:51:50 2014 -0800

Do not destroy vif's napi context, if it was not initialised during addition

If the napi structure was not initialised during interface addition, because
of errors, it should not be touched during the delete too, since doing a
netif_napi_del results in crash. However, there are no reliable checks to find
whether napi was initialised or not. Hence, for now we will check for the existance
poll function, which should have been set if the napi structure was initialised.

Closes BUG: #1399577

Change-Id: I8cf439dc53805801a5ba301f542dedb2aaa5dee2

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5632
Committed: http://github.org/Juniper/contrail-vrouter/commit/6cbce2965240261b9cbed3999948178a5a63bd82
Submitter: Zuul
Branch: R2.0

commit 6cbce2965240261b9cbed3999948178a5a63bd82
Author: Anand H Krishnan <email address hidden>
Date: Sun Dec 7 22:51:50 2014 -0800

Do not destroy vif's napi context, if it was not initialised during addition

If the napi structure was not initialised during interface addition, because
of errors, it should not be touched during the delete too, since doing a
netif_napi_del results in crash. However, there are no reliable checks to find
whether napi was initialised or not. Hence, for now we will check for the existance
poll function, which should have been set if the napi structure was initialised.

Closes BUG: #1399577

Change-Id: I8cf439dc53805801a5ba301f542dedb2aaa5dee2

[Martin Gerhard Loschwitz (martin-loschwitz)]

While walking through our logs, we have just discovered this:

Dec 16 20:14:37 bka-001-05 kernel: [ 1962.816461] interface veth24d195f8-9 add (type 3, flags 1c1) failed with error: -19

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5766
Committed: http://github.org/Juniper/contrail-vrouter/commit/e3a0755d225de698d0dbf99ca25e0735f4e40bdc
Submitter: Zuul
Branch: R1.10

commit e3a0755d225de698d0dbf99ca25e0735f4e40bdc
Author: Anand H Krishnan <email address hidden>
Date: Sun Dec 7 22:51:50 2014 -0800

Do not destroy vif's napi context, if it was not initialised during addition

If the napi structure was not initialised during interface addition, because
of errors, it should not be touched during the delete too, since doing a
netif_napi_del results in crash. However, there are no reliable checks to find
whether napi was initialised or not. Hence, for now we will check for the existance
poll function, which should have been set if the napi structure was initialised.

Closes BUG: #1399577

Change-Id: I8cf439dc53805801a5ba301f542dedb2aaa5dee2