vRouter: kernel crash in vr_flow_lookup
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R2.20 |
Fix Committed
|
High
|
Anand H. Krishnan | |||
R2.21.x |
Fix Committed
|
High
|
Anand H. Krishnan | |||
R2.22.x |
Fix Committed
|
High
|
Anand H. Krishnan | |||
R3.0 |
Fix Committed
|
High
|
Anand H. Krishnan | |||
R3.1 |
Fix Committed
|
High
|
Anand H. Krishnan | |||
Trunk |
Fix Committed
|
High
|
Anand H. Krishnan |
Bug Description
One of the compute nodes went down with the trace that is pasted towards the end of the report. On analysis, it was found that vRouter was trying to form a flow for an ICMP error packet that was generated for another ICMP error. vRouter doesn't form flow keys for such packets, expecting that the flow calls upstream will drop such packets. However, no error is returned to the upstream calls to indicate that packet should be dropped and hence the flow module tries to allocate a new flow entry with uninitialized key resulting in memory corruption (key length value can be high).
[162923.869234] general protection fault: 0000 [#1] SMP
[162923.874937] Modules linked in: ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack veth ipmi_si act_police cls_u32 sch_ingress cls_fw sch_sfq sch_htb mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables dell_rbu nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_
[162923.955481] macvlan kvm_intel fscache kvm bonding btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear bnx2x ahci mdio megaraid_sas libahci libcrc32c wmi [last unloaded: ipmi_si]
[162923.979042] CPU: 48 PID: 5427 Comm: contrail-vroute Tainted: G OX 3.13.0-77-generic #121-Ubuntu
[162923.989944] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.3.6 06/03/2015
[162923.998458] task: ffff8827959eb000 ti: ffff882797722000 task.ti: ffff882797722000
[162924.006970] RIP: 0010:[<
[162924.017365] RSP: 0000:ffff8827df
[162924.023427] RAX: 8827df703a98ffff RBX: ffff88278f6e0b40 RCX: 000000000007e392
[162924.031552] RDX: 0000000038190037 RSI: ffff88278f6e0b40 RDI: ffff881dba565800
[162924.039676] RBP: ffff8827df7037b0 R08: ffff8827df703784 R09: ffff881bf9559980
[162924.047800] R10: ffff8827df007900 R11: 0000000000000001 R12: ffffffffa0418a00
[162924.055923] R13: ffff8827df703a98 R14: 0000000000000001 R15: ffff881dba565828
[162924.064058] FS: 00007f04277fd70
[162924.073250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[162924.079798] CR2: 00007f0408bd64a8 CR3: 0000004f91cfe000 CR4: 00000000001427e0
[162924.087922] Stack:
[162924.090290] 381900370000b82d 0000b82ddf703868 ffff881dba565828 ffff881c1141006c
[162924.098753] ffff8827df703a98 ffffffffa0418a00 00000000000000fe ffff8827df7037f8
[162924.107216] ffffffffa04079b2 0002000100000000 00003790ffff0000 ffff8827df708827
[162924.115674] Call Trace:
[162924.118527] <IRQ>
[162924.120779]
[162924.122578] [<ffffffffa0407
[162924.128839] [<ffffffffa040b
[162924.136174] [<ffffffffa0400
[162924.143018] [<ffffffffa0408
[162924.150348] [<ffffffffa0407
[162924.157580] [<ffffffffa0408
[162924.164527] [<ffffffffa03ff
[162924.171274] [<ffffffffa0400
[162924.178116] [<ffffffffa0406
[162924.185153] [<ffffffff81629
[162924.192290] [<ffffffffa0408
[162924.199231] [<ffffffffa0403
[162924.206174] [<ffffffffa0403
[162924.213697] [<ffffffffa0404
[162924.220250] [<ffffffffa03fb
[162924.227880] [<ffffffff81207
[162924.234724] [<ffffffffa03fb
[162924.242729] [<ffffffff81627
[162924.250153] [<ffffffff81628
[162924.256896] [<ffffffff81628
[162924.263447] [<ffffffffa03fb
[162924.271065] [<ffffffff81627
[162924.278491] [<ffffffff8101b
[162924.285428] [<ffffffff81628
[162924.292171] [<ffffffff81628
[162924.298720] [<ffffffff81629
[162924.305184] [<ffffffffa00c4
[162924.312420] [<ffffffff810a5
[162924.319068] [<ffffffff8165e
[162924.326491] [<ffffffff810a3
[162924.332750] [<ffffffff8108d
[162924.339883] [<ffffffff81360
[162924.347118] [<ffffffffa00c5
[162924.353861] [<ffffffff81628
[162924.360217] [<ffffffff8106c
[162924.366376] [<ffffffff8106d
[162924.372246] [<ffffffff81738
[162924.377726] [<ffffffff8172d
[162924.384176] <EOI>
[162924.386425]
[162924.388212] [<ffffffff81735
[162924.393691] Code: 5e 41 5f 5d c3 0f 1f 84 00 00 00 00 00 80 43 3a 01 66 83 7b 20 01 75 c4 48 8b 43 18 48 85 c0 0f 84 50 01 00 00 41 be 01 00 00 00 <f0> 44 0f c1 70 04 be 08 00 00 00 41 83 fe 02 0f 86 cd 00 00 00
[162924.415752] RIP [<ffffffffa040b
[162924.423481] RSP <ffff8827df703778>
Changed in juniperopenstack: | |
importance: | Undecided → High |
Review in progress for https:/ /review. opencontrail. org/18358
Submitter: Anand H. Krishnan (<email address hidden>)