Juniper Openstack

pbb_evpn: [4.0.0.0-3044~kilo] vRouter crash observed @free_pipe_info+0x62/0xa0, put_pipe_info+0x44/0x60 with continuous vmi/bd addition/deletion

Bug #1672373 reported by Chandra Sekhar Reddy Mallam
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.0
In Progress
High
Divakar Dharanalakota
R2.1
In Progress
High
Divakar Dharanalakota
R2.20
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r2.23 "r2.23"
R2.21.x
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r2.21.3
R2.22.x
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r2.22.3
R3.0
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r3.0.4.0
R3.1
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r3.1.3.0
R3.2
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r3.2.3.0
Trunk
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

vRouter crash observed @free_pipe_info+0x62/0xa0, put_pipe_info+0x44/0x60 with continuous vmi/bd addition/deletion.

[682731.189946] Call Trace:
[682731.200674] [<ffffffff811cb762>] ? free_pipe_info+0x62/0xa0
[682731.211419] [<ffffffff811cb7e4>] put_pipe_info+0x44/0x60
[682731.221761] [<ffffffff811cb8a0>] pipe_release+0xa0/0xb0
[682731.231997] [<ffffffff811c4044>] __fput+0xe4/0x260
[682731.242247] [<ffffffff811c420e>] ____fput+0xe/0x10
[682731.252178] [<ffffffff8108b97c>] task_work_run+0xac/0xd0
[682731.262296] [<ffffffff81014e77>] do_notify_resume+0x97/0xb0
[682731.271868] [<ffffffff8173e01a>] int_signal+0x12/0x17
[682731.281511] Code: Bad RIP value.
[682731.290846] RIP [< (null)>] (null)
[682731.300142] RSP <ffff8800c97bfe38>
[682731.309062] CR2: 0000000000000000

Core file is copied to:
bhushana@10.204.216.50:/home/bhushana/Documents/technical/bugs/<bug-ID>

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/29831
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/29831
Committed: http://github.org/Juniper/contrail-vrouter/commit/0b81b2bbd7dc50a786b3cc56dc31b150f3948098
Submitter: Zuul (<email address hidden>)
Branch: master

commit 0b81b2bbd7dc50a786b3cc56dc31b150f3948098
Author: Divakar D <email address hidden>
Date: Thu Mar 23 12:06:39 2017 +0530

Increment Nexthop ref count atomically

Right now, the NH's refcount is not incremented atomically as reference
to NH is always taken from a single thread in nelink context. This is
becuase Routes, Labels, and other NH's access an NH only when Agent
performs config changes. But with data path mac learning in place
because of PBB, reference to NH, esp L2 Mutlicast NH, happens from
multiple threads leading to requirement incrementing it atomically. Not
doing it causes the ref count to go bad for that particular NH and
results in double free of NH.

Change-Id: I0afa517b56eeac0371670ee76241e392be762f20
closes-bug: #1672293
closes-bug: #1672373
closes-bug: #1673663

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/30278
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/30280
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/30281
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/30282
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/30283
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.0

Review in progress for https://review.opencontrail.org/30287
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.1

Review in progress for https://review.opencontrail.org/30288
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/30280
Committed: http://github.org/Juniper/contrail-vrouter/commit/0cfe6339fa3d2de927175aa99da2149e9adae090
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 0cfe6339fa3d2de927175aa99da2149e9adae090
Author: Divakar D <email address hidden>
Date: Thu Mar 23 12:06:39 2017 +0530

Increment Nexthop ref count atomically

Right now, the NH's refcount is not incremented atomically as reference
to NH is always taken from a single thread in nelink context. This is
becuase Routes, Labels, and other NH's access an NH only when Agent
performs config changes. But with data path mac learning in place
because of PBB, reference to NH, esp L2 Mutlicast NH, happens from
multiple threads leading to requirement incrementing it atomically. Not
doing it causes the ref count to go bad for that particular NH and
results in double free of NH.

Change-Id: I0afa517b56eeac0371670ee76241e392be762f20
closes-bug: #1672293
closes-bug: #1672373
closes-bug: #1673663

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/30281
Committed: http://github.org/Juniper/contrail-vrouter/commit/9900fb6ecc2534806d769aaf7c95915b2398d122
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit 9900fb6ecc2534806d769aaf7c95915b2398d122
Author: Divakar D <email address hidden>
Date: Thu Mar 23 12:06:39 2017 +0530

Increment Nexthop ref count atomically

Right now, the NH's refcount is not incremented atomically as reference
to NH is always taken from a single thread in nelink context. This is
becuase Routes, Labels, and other NH's access an NH only when Agent
performs config changes. But with data path mac learning in place
because of PBB, reference to NH, esp L2 Mutlicast NH, happens from
multiple threads leading to requirement incrementing it atomically. Not
doing it causes the ref count to go bad for that particular NH and
results in double free of NH.

Change-Id: I0afa517b56eeac0371670ee76241e392be762f20
closes-bug: #1672293
closes-bug: #1672373
closes-bug: #1673663

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/30278
Committed: http://github.org/Juniper/contrail-vrouter/commit/d0eeccc9ec0cbe1dffa3653e1c44c9528a41e22b
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit d0eeccc9ec0cbe1dffa3653e1c44c9528a41e22b
Author: Divakar D <email address hidden>
Date: Thu Mar 23 12:06:39 2017 +0530

Increment Nexthop ref count atomically

Right now, the NH's refcount is not incremented atomically as reference
to NH is always taken from a single thread in nelink context. This is
becuase Routes, Labels, and other NH's access an NH only when Agent
performs config changes. But with data path mac learning in place
because of PBB, reference to NH, esp L2 Mutlicast NH, happens from
multiple threads leading to requirement incrementing it atomically. Not
doing it causes the ref count to go bad for that particular NH and
results in double free of NH.

Change-Id: I0afa517b56eeac0371670ee76241e392be762f20
closes-bug: #1672293
closes-bug: #1672373
closes-bug: #1673663

Jeba Paulaiyan (jebap)
tags: added: crashes
Ashish Ranjan (aranjan-n)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/30283
Committed: http://github.org/Juniper/contrail-vrouter/commit/e20b49b09e4227f35697c98c87c9da4b258c93fa
Submitter: Zuul (<email address hidden>)
Branch: R2.21.x

commit e20b49b09e4227f35697c98c87c9da4b258c93fa
Author: Divakar D <email address hidden>
Date: Thu Mar 23 12:06:39 2017 +0530

Increment Nexthop ref count atomically

Right now, the NH's refcount is not incremented atomically as reference
to NH is always taken from a single thread in nelink context. This is
becuase Routes, Labels, and other NH's access an NH only when Agent
performs config changes. But with data path mac learning in place
because of PBB, reference to NH, esp L2 Mutlicast NH, happens from
multiple threads leading to requirement incrementing it atomically. Not
doing it causes the ref count to go bad for that particular NH and
results in double free of NH.

Change-Id: I0afa517b56eeac0371670ee76241e392be762f20
closes-bug: #1672293
closes-bug: #1672373
closes-bug: #1673663

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/30417
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/30282
Committed: http://github.com/Juniper/contrail-vrouter/commit/72daf05792561fb518c7161c082ad309c09fa8cf
Submitter: Zuul (<email address hidden>)
Branch: R2.20

commit 72daf05792561fb518c7161c082ad309c09fa8cf
Author: Divakar D <email address hidden>
Date: Thu Mar 23 12:06:39 2017 +0530

Increment Nexthop ref count atomically

Right now, the NH's refcount is not incremented atomically as reference
to NH is always taken from a single thread in nelink context. This is
becuase Routes, Labels, and other NH's access an NH only when Agent
performs config changes. But with data path mac learning in place
because of PBB, reference to NH, esp L2 Mutlicast NH, happens from
multiple threads leading to requirement incrementing it atomically. Not
doing it causes the ref count to go bad for that particular NH and
results in double free of NH.

Change-Id: I0afa517b56eeac0371670ee76241e392be762f20
closes-bug: #1672293
closes-bug: #1672373
closes-bug: #1673663

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/30417
Committed: http://github.com/Juniper/contrail-vrouter/commit/1f01b6f0dab48580f105c5ac624438835531f23f
Submitter: Zuul (<email address hidden>)
Branch: R2.22.x

commit 1f01b6f0dab48580f105c5ac624438835531f23f
Author: Divakar D <email address hidden>
Date: Thu Mar 23 12:06:39 2017 +0530

Increment Nexthop ref count atomically

Right now, the NH's refcount is not incremented atomically as reference
to NH is always taken from a single thread in nelink context. This is
becuase Routes, Labels, and other NH's access an NH only when Agent
performs config changes. But with data path mac learning in place
because of PBB, reference to NH, esp L2 Mutlicast NH, happens from
multiple threads leading to requirement incrementing it atomically. Not
doing it causes the ref count to go bad for that particular NH and
results in double free of NH.

Change-Id: I0afa517b56eeac0371670ee76241e392be762f20
closes-bug: #1672293
closes-bug: #1672373
closes-bug: #1673663

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.