[eBay] Server reboots with 3.0.2.1-6 eBay build

[Mladen Maric]

eBay reported server freezing/reboots, after fix for issue identified in "2016-0829-1122, Tracebacks in DMESG contrail 3.0.2.1 on CentOS 7" has been fixed as part of PR1618375 (3.0.2.1-6).
https://review.opencontrail.org/#/c/23763/

This happens both with kernel 3.10.0-327.28.3.el7.x86_64 and the "officially supported" 3.10.0-327.10.1.el7.x86_64

Core dumps to be uploaded.

dmesg:
3994.317851] BUG: unable to handle kernel paging request at ffff883fccdb8000
[ 3994.317886] IP: [<ffffffffa0605f45>] vr_interface_add_response+0x75/0x170 [vrouter]
[ 3994.317955] PGD 1f32067 PUD 3fcd9e1063 PMD 3fccd65063 PTE 8000003fccdb8161
[ 3994.317977] Oops: 0003 [#1] SMP
[ 3994.317990] Modules linked in: xt_comment xt_multiport vfat fat mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase dell_rbu vhost_net vhost macvtap macvlan tun binfmt_misc vrouter(OE) ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_devintf pcspkr sb_edac mei_me edac_core
[ 3994.318227] lpc_ich mei shpchp mfd_core ipmi_si ipmi_msghandler wmi tpm_crb acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt drm_kms_helper crct10dif_pclmul ttm crct10dif_common crc32c_intel drm i40e igb ahci vxlan libahci dca ip6_udp_tunnel i2c_algo_bit udp_tunnel libata i2c_core ptp megaraid_sas pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: usb_storage]
[ 3994.318374] CPU: 15 PID: 11435 Comm: contrail-vroute Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1
[ 3994.318404] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.0.2 03/15/2016
[ 3994.318425] task: ffff881f93150000 ti: ffff881f91968000 task.ti: ffff881f91968000
[ 3994.318445] RIP: 0010:[<ffffffffa0605f45>] [<ffffffffa0605f45>] vr_interface_add_response+0x75/0x170 [vrouter]
[ 3994.318478] RSP: 0018:ffff881f9196b5f0 EFLAGS: 00010206
[ 3994.318493] RAX: 0000000000000041 RBX: ffff883f8c283000 RCX: ffff883fccdb7e00
[ 3994.318513] RDX: 0000000000000040 RSI: ffff883f84818000 RDI: ffff883f8c283000
[ 3994.318532] RBP: ffff881f9196b5f0 R08: 0000000000000000 R09: ffff883fccdb7e00
[ 3994.318551] R10: ffff881ffec07600 R11: ffffffffa05f1593 R12: ffff883faa6c7400
[ 3994.318570] R13: 0000000000000001 R14: ffffffffa061fc60 R15: ffff883fbccc3700
[ 3994.318590] FS: 00002b6d84801700(0000) GS:ffff883ffeae0000(0000) knlGS:0000000000000000
[ 3994.318611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3994.318627] CR2: ffff883fccdb8000 CR3: 0000003f55925000 CR4: 00000000003427e0
[ 3994.318645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3994.318664] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3994.318683] Stack:
[ 3994.318690] ffff881f9196b648 ffffffffa0607bc9 000000000000001b ffff881f9196b630
[ 3994.318718] ffffffffa05f8483 000000003a52b449 ffff883f8c284c00 0000000000000000
[ 3994.318749] ffff883f8c283000 ffffffffa061fc60 ffff883fbccc3700 ffff881f9196b688
[ 3994.318779] Call Trace:
[ 3994.318794] [<ffffffffa0607bc9>] vr_interface_make_req+0x469/0x510 [vrouter]
[ 3994.318818] [<ffffffffa05f8483>] ? lh_zalloc+0x33/0x40 [vrouter]
[ 3994.318841] [<ffffffffa060969e>] vr_interface_req_process+0x1fe/0x2b0 [vrouter]
[ 3994.318864] [<ffffffffa05f43f9>] sandesh_decode_one+0x119/0x200 [vrouter]
[ 3994.318887] [<ffffffffa05f6880>] ? thrift_binary_protocol_init+0x270/0x270 [vrouter]
[ 3994.319638] [<ffffffffa05f6890>] ? thrift_memory_buffer_is_open+0x10/0x10 [vrouter]
[ 3994.320372] [<ffffffffa05f68a0>] ? thrift_memory_buffer_open+0x10/0x10 [vrouter]
[ 3994.321102] [<ffffffffa05f68e0>] ? thrift_memory_buffer_flush+0x10/0x10 [vrouter]
[ 3994.321828] [<ffffffffa05f68b0>] ? thrift_memory_buffer_close+0x10/0x10 [vrouter]
[ 3994.322542] [<ffffffffa05f6970>] ? thrift_memory_buffer_read+0x90/0x90 [vrouter]
[ 3994.323247] [<ffffffffa05f68c0>] ? thrift_memory_buffer_read_end+0x10/0x10 [vrouter]
[ 3994.323951] [<ffffffffa05f68d0>] ? thrift_memory_buffer_write_end+0x10/0x10 [vrouter]
[ 3994.324654] [<ffffffffa05f63b0>] ? thrift_binary_protocol_write_string+0x50/0x50 [vrouter]
[ 3994.325354] [<ffffffffa05f5360>] ? thrift_binary_protocol_write_message_end+0x10/0x10 [vrouter]
[ 3994.326048] [<ffffffffa05f6480>] ? thrift_binary_protocol_write_set_begin+0x10/0x10 [vrouter]
[ 3994.326734] [<ffffffffa05f5350>] ? thrift_binary_protocol_write_bool+0x60/0x60 [vrouter]
[ 3994.327414] [<ffffffffa05f5370>] ? thrift_binary_protocol_write_sandesh_end+0x10/0x10 [vrouter]
[ 3994.328082] [<ffffffffa05f5380>] ? thrift_binary_protocol_write_struct_begin+0x10/0x10 [vrouter]
[ 3994.328732] [<ffffffffa05f61b0>] ? thrift_binary_protocol_read_binary+0xf0/0xf0 [vrouter]
[ 3994.329366] [<ffffffffa05f5390>] ? thrift_binary_protocol_write_struct_end+0x10/0x10 [vrouter]
[ 3994.330153] [<ffffffffa05f53a0>] ? thrift_binary_protocol_write_field_end+0x10/0x10 [vrouter]
[ 3994.330755] [<ffffffffa05f6550>] ? thrift_binary_protocol_write_message_begin+0xd0/0xd0 [vrouter]
[ 3994.331342] [<ffffffffa05f5400>] ? thrift_binary_protocol_write_field_stop+0x60/0x60 [vrouter]
[ 3994.331913] [<ffffffffa05f63d0>] ? thrift_binary_protocol_write_sandesh_begin+0x20/0x20 [vrouter]
[ 3994.332468] [<ffffffffa05f5410>] ? thrift_binary_protocol_write_map_end+0x10/0x10 [vrouter]
[ 3994.333008] [<ffffffffa05f6470>] ? thrift_binary_protocol_write_list_begin+0xa0/0xa0 [vrouter]
[ 3994.333540] [<ffffffffa05f5420>] ? thrift_binary_protocol_write_list_end+0x10/0x10 [vrouter]
[ 3994.334047] [<ffffffffa05f52f0>] ? thrift_binary_protocol_write_u64+0x80/0x80 [vrouter]
[ 3994.334534] [<ffffffffa05f51c0>] ? thrift_protocol_skip+0x270/0x270 [vrouter]
[ 3994.335009] [<ffffffffa05f59c0>] ? thrift_binary_protocol_read_ipv4+0x10/0x10 [vrouter]
[ 3994.335482] [<ffffffffa05f5830>] ? thrift_binary_protocol_read_double+0x10/0x10 [vrouter]
[ 3994.335943] [<ffffffffa05f51f0>] ? thrift_binary_protocol_write_byte+0x30/0x30 [vrouter]
[ 3994.336404] [<ffffffffa05f5bb0>] ? thrift_binary_protocol_read_u16+0x70/0x70 [vrouter]
[ 3994.336859] [<ffffffffa05f5880>] ? thrift_binary_protocol_write_i32+0x50/0x50 [vrouter]
[ 3994.337319] [<ffffffffa05f5270>] ? thrift_binary_protocol_write_i64+0x80/0x80 [vrouter]
[ 3994.337760] [<ffffffffa05f6250>] ? thrift_binary_protocol_write_field_begin+0xa0/0xa0 [vrouter]
[ 3994.338202] [<ffffffffa05f5fc0>] ? thrift_binary_protocol_read_sandesh_begin+0x20/0x20 [vrouter]
[ 3994.338657] [<ffffffffa05f5460>] ? thrift_binary_protocol_write_uuid_t+0x30/0x30 [vrouter]
[ 3994.339105] [<ffffffffa05f6360>] ? thrift_binary_protocol_write_binary+0xb0/0xb0 [vrouter]
[ 3994.339542] [<ffffffffa05f62b0>] ? thrift_binary_protocol_write_ipv4+0x60/0x60 [vrouter]
[ 3994.339963] [<ffffffffa05f6360>] ? thrift_binary_protocol_write_binary+0xb0/0xb0 [vrouter]
[ 3994.340383] [<ffffffffa05f5430>] ? thrift_binary_protocol_write_set_end+0x10/0x10 [vrouter]
[ 3994.340817] [<ffffffffa05f5fa0>] ? thrift_binary_protocol_read_message_begin+0xf0/0xf0 [vrouter]
[ 3994.341239] [<ffffffffa05f5480>] ? thrift_binary_protocol_read_message_end+0x10/0x10 [vrouter]
[ 3994.341660] [<ffffffffa05f5eb0>] ? thrift_binary_protocol_read_string+0xe0/0xe0 [vrouter]
[ 3994.342074] [<ffffffffa05f5470>] ? thrift_binary_protocol_write_double+0x10/0x10 [vrouter]
[ 3994.342489] [<ffffffffa05f5490>] ? thrift_binary_protocol_read_sandesh_end+0x10/0x10 [vrouter]
[ 3994.342896] [<ffffffffa05f54b0>] ? thrift_binary_protocol_read_struct_begin+0x20/0x20 [vrouter]
[ 3994.343295] [<ffffffffa05f5a90>] ? thrift_binary_protocol_read_i16+0x70/0x70 [vrouter]
[ 3994.343693] [<ffffffffa05f54c0>] ? thrift_binary_protocol_read_struct_end+0x10/0x10 [vrouter]
[ 3994.344085] [<ffffffffa05f5c10>] ? thrift_binary_protocol_write_u16+0x60/0x60 [vrouter]
[ 3994.344467] [<ffffffffa05f54d0>] ? thrift_binary_protocol_read_field_end+0x10/0x10 [vrouter]
[ 3994.344847] [<ffffffffa05f5d00>] ? thrift_binary_protocol_read_map_begin+0xf0/0xf0 [vrouter]
[ 3994.345231] [<ffffffffa05f54e0>] ? thrift_binary_protocol_read_map_end+0x10/0x10 [vrouter]
[ 3994.345613] [<ffffffffa05f5dc0>] ? thrift_binary_protocol_read_list_begin+0xc0/0xc0 [vrouter]
[ 3994.345990] [<ffffffffa05f54f0>] ? thrift_binary_protocol_read_list_end+0x10/0x10 [vrouter]
[ 3994.346378] [<ffffffffa05f5500>] ? thrift_binary_protocol_read_set_end+0x10/0x10 [vrouter]
[ 3994.346757] [<ffffffffa05f5570>] ? thrift_binary_protocol_read_bool+0x70/0x70 [vrouter]
[ 3994.347137] [<ffffffffa05f5a20>] ? thrift_binary_protocol_write_i16+0x60/0x60 [vrouter]
[ 3994.347512] [<ffffffffa05f58d0>] ? thrift_binary_protocol_write_u32+0x50/0x50 [vrouter]
[ 3994.347881] [<ffffffffa05f55e0>] ? thrift_binary_protocol_read_byte+0x70/0x70 [vrouter]
[ 3994.348245] [<ffffffffa05f5b40>] ? thrift_binary_protocol_read_field_begin+0xb0/0xb0 [vrouter]
[ 3994.348612] [<ffffffffa05f5940>] ? thrift_binary_protocol_read_i32+0x70/0x70 [vrouter]
[ 3994.348977] [<ffffffffa05f5680>] ? thrift_binary_protocol_read_i64+0xa0/0xa0 [vrouter]
[ 3994.349337] [<ffffffffa05f59b0>] ? thrift_binary_protocol_read_u32+0x70/0x70 [vrouter]
[ 3994.349693] [<ffffffffa05f5720>] ? thrift_binary_protocol_read_u64+0xa0/0xa0 [vrouter]
[ 3994.350163] [<ffffffffa05f5820>] ? thrift_binary_protocol_read_uuid_t+0x30/0x30 [vrouter]
[ 3994.350552] [<ffffffffa05f5dd0>] ? thrift_binary_protocol_read_set_begin+0x10/0x10 [vrouter]
[ 3994.350900] [<ffffffffa05f60c0>] ? thrift_binary_protocol_write_ipaddr+0x100/0x100 [vrouter]
[ 3994.351245] [<ffffffffa05f5dd0>] ? thrift_binary_protocol_read_set_begin+0x10/0x10 [vrouter]
[ 3994.351589] [<ffffffffa05f57f0>] ? thrift_binary_protocol_read_ipaddr+0xd0/0xd0 [vrouter]
[ 3994.351941] [<ffffffffa05f42c0>] ? sandesh_hdr_free+0x10/0x10 [vrouter]
[ 3994.352292] [<ffffffffa05f4596>] sandesh_decode+0x46/0x90 [vrouter]
[ 3994.352645] [<ffffffffa05fe343>] sandesh_proto_decode+0x33/0x50 [vrouter]
[ 3994.353003] [<ffffffffa05fddad>] vr_message_request+0x3d/0x70 [vrouter]
[ 3994.353356] [<ffffffffa05fd323>] netlink_trans_request+0x63/0x1e0 [vrouter]
[ 3994.353706] [<ffffffff81285708>] ? security_capable+0x18/0x20
[ 3994.354055] [<ffffffff81316cc2>] ? nla_parse+0x32/0x120
[ 3994.354405] [<ffffffff8155c0fd>] genl_family_rcv_msg+0x1cd/0x400
[ 3994.354755] [<ffffffff811c106a>] ? kmem_cache_alloc+0x1ba/0x1d0
[ 3994.355104] [<ffffffff8155c330>] ? genl_family_rcv_msg+0x400/0x400
[ 3994.355454] [<ffffffff8155c3c1>] genl_rcv_msg+0x91/0xd0
[ 3994.355807] [<ffffffff8155a329>] netlink_rcv_skb+0xa9/0xc0
[ 3994.356159] [<ffffffff8155a858>] genl_rcv+0x28/0x40
[ 3994.356507] [<ffffffff8155993d>] netlink_unicast+0xed/0x1b0
[ 3994.356864] [<ffffffff81559d30>] netlink_sendmsg+0x330/0x770
[ 3994.357216] [<ffffffff81288935>] ? sock_has_perm+0x75/0x90
[ 3994.357568] [<ffffffff81510dd0>] sock_sendmsg+0xb0/0xf0
[ 3994.357917] [<ffffffff8151149f>] ? sock_recvmsg+0xbf/0x100
[ 3994.358266] [<ffffffff81511209>] ___sys_sendmsg+0x3a9/0x3c0
[ 3994.358618] [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[ 3994.358970] [<ffffffff8163a398>] ? __schedule+0x2d8/0x900
[ 3994.359320] [<ffffffff8130c4bd>] ? list_del+0xd/0x30
[ 3994.359668] [<ffffffff8122bc32>] ? eventfd_ctx_read+0x102/0x210
[ 3994.360015] [<ffffffff815120f1>] __sys_sendmsg+0x51/0x90
[ 3994.360370] [<ffffffff81512142>] SyS_sendmsg+0x12/0x20
[ 3994.360726] [<ffffffff81645a49>] system_call_fastpath+0x16/0x1b
[ 3994.361071] Code: 30 48 01 87 b8 00 00 00 85 d2 74 2d 31 c0 0f 1f 84 00 00 00 00 00 4c 8b 46 38 48 8b 8f d8 00 00 00 48 63 d0 83 c0 01 4d 8b 04 d0 <4c> 01 04 d1 8b 15 b1 88 01 00 39 d0 72 dd 89 97 e0 00 00 00 48
[ 3994.361874] RIP [<ffffffffa0605f45>] vr_interface_add_response+0x75/0x170 [vrouter]
[ 3994.362267] RSP <ffff881f9196b5f0>
[ 3994.362651] CR2: ffff883fccdb8000

[Mladen Maric (mmaric)]

Core dumps:
ftp://ftp.juniper.net/pub/sftp/2016-0829-1122/crash.tgz

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24033
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24033
Committed: http://github.org/Juniper/contrail-vrouter/commit/74bc8f26ecfd99bad9dcb6948d038b5efe8dfa41
Submitter: Zuul
Branch: R3.0.2.x

commit 74bc8f26ecfd99bad9dcb6948d038b5efe8dfa41
Author: Anand H. Krishnan <email address hidden>
Date: Fri Sep 9 19:20:27 2016 +0530

Fix memory allocation for interface request

In order to pass per lcore queue input error statistics to the
application that does vif query, we allocate only VR_MAX_CPUS
worth of memory, but we try to copy vr_num_cpus worth of data.
In the case of vr_num_cpus > VR_MAX_CPUS (64), we will hit a snag.
As a fix, allocate memory for vr_num_cpus.

Change-Id: Ifb1060aac20011b8d51e1b31063a363fe268fd3d
Closes-Bug: #1621816

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24126
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24127
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24128
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24126
Committed: http://github.org/Juniper/contrail-vrouter/commit/e891b0137450e5fa57d2d09e9325fa7cee5126e9
Submitter: Zuul
Branch: master

commit e891b0137450e5fa57d2d09e9325fa7cee5126e9
Author: Anand H. Krishnan <email address hidden>
Date: Fri Sep 9 19:20:27 2016 +0530

Fix memory allocation for interface request

In order to pass per lcore queue input error statistics to the
application that does vif query, we allocate only VR_MAX_CPUS
worth of memory, but we try to copy vr_num_cpus worth of data.
In the case of vr_num_cpus > VR_MAX_CPUS (64), we will hit a snag.
As a fix, allocate memory for vr_num_cpus.

Change-Id: Ifb1060aac20011b8d51e1b31063a363fe268fd3d
Closes-Bug: #1621816

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24127
Committed: http://github.org/Juniper/contrail-vrouter/commit/3bfe603839ce6fa6e399b31df4c8b58cb1f3ace1
Submitter: Zuul
Branch: R3.1

commit 3bfe603839ce6fa6e399b31df4c8b58cb1f3ace1
Author: Anand H. Krishnan <email address hidden>
Date: Fri Sep 9 19:20:27 2016 +0530

Fix memory allocation for interface request

In order to pass per lcore queue input error statistics to the
application that does vif query, we allocate only VR_MAX_CPUS
worth of memory, but we try to copy vr_num_cpus worth of data.
In the case of vr_num_cpus > VR_MAX_CPUS (64), we will hit a snag.
As a fix, allocate memory for vr_num_cpus.

Change-Id: Ifb1060aac20011b8d51e1b31063a363fe268fd3d
Closes-Bug: #1621816

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24128
Committed: http://github.org/Juniper/contrail-vrouter/commit/a709ccd317c289e7790e51ccbb2e9e27ddbe80fb
Submitter: Zuul
Branch: R3.0

commit a709ccd317c289e7790e51ccbb2e9e27ddbe80fb
Author: Anand H. Krishnan <email address hidden>
Date: Fri Sep 9 19:20:27 2016 +0530

Fix memory allocation for interface request

In order to pass per lcore queue input error statistics to the
application that does vif query, we allocate only VR_MAX_CPUS
worth of memory, but we try to copy vr_num_cpus worth of data.
In the case of vr_num_cpus > VR_MAX_CPUS (64), we will hit a snag.
As a fix, allocate memory for vr_num_cpus.

Change-Id: Ifb1060aac20011b8d51e1b31063a363fe268fd3d
Closes-Bug: #1621816