| 2015-01-16 22:06:29 |
Nischal Sheth |
bug |
|
|
added bug |
| 2015-01-16 22:08:05 |
Nischal Sheth |
description |
Control node deadlocked in RibOutUpdates::TailDequeue. This happened
when UpdatePack tried to acquire a lock on the next RouteUpdate in the
attr_set_ in UpdateQueue. This in turn happened since DequeueCommon
already had a lock on the RouteUpdate in question since it was the previous
one in the (temporal) queue_ in UpdateQueue.
Root cause is that calling clock_gettime with CLOCK_MONOTONIC does not
guarantee monotonically increasing timestamps. |
Control node deadlocked in RibOutUpdates::TailDequeue. This happened
when UpdatePack tried to acquire a lock on the next RouteUpdate in the
attr_set_ in UpdateQueue. This in turn happened since TailDequeue
already had a lock on the RouteUpdate in question since it was the previous one in the (temporal) queue_ in UpdateQueue.
Root cause is that calling clock_gettime with CLOCK_MONOTONIC does not
guarantee monotonically increasing timestamps.
(gdb) info threads
Id Target Id Frame
4 Thread 0x7f86bd323700 (LWP 25182) "contrail-contro" 0x00007f86c4ea3f2c in __lll_lock_wait ()
from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f86bcf22700 (LWP 25183) "contrail-contro" 0x00007f86c4ea3f2c in __lll_lock_wait ()
from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f86bcb21700 (LWP 25184) "contrail-contro" 0x00007f86c4ea1414 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
* 1 Thread 0x7f86c69487c0 (LWP 25177) "contrail-contro" 0x00007f86c3f6e5fa in epoll_ctl () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 is the lock owner
(gdb) frame 5
#5 0x00000000007622df in RibUpdateMonitor::GetAttrNext (this=0x7f86b0096b50, queue_id=1, current_uinfo=0x7f86b0093510,
next_uinfo_p=0x7f86bd322770) at controller/src/bgp/bgp_update_monitor.cc:973
973 RouteUpdatePtr update(mp, rt_update, &mutex_, &cond_var_);
Next queue element by attribute:
(gdb) p mp
$1 = (tbb::mutex *) 0x7f86b02eedb0
(gdb) p rt_update
$2 = (RouteUpdate *) 0x7f86b02eed90
#8 0x000000000071f95d in RibOutUpdates::TailDequeue (this=0x7f86b0096780, queue_id=1, msync=..., blocked=0x7f86bd322af0)
at controller/src/bgp/bgp_ribout_updates.cc:187
187 if (!DequeueCommon(start_marker, update.get(), blocked)) {
The previously processed by Tail dequeue (which is still locked is):
(gdb) p next_update
$3 = {entry_mutexp_ = 0x7f86b02eedb0, rt_update_ = 0x7f86b02eed90, monitor_mutexp_ = 0x7f86b0096b50, cond_var_ = 0x7f86b0096b78}
Previous and current elements have the same timestamp:
(gdb) p update.rt_update_->tstamp_
$6 = 851624330824509
(gdb) p next_update.rt_update_->tstamp_
$7 = 851624330824509
This causes the order of the attribute queue to dependent on the UpdateInfo pointer. |
|
| 2015-01-16 22:09:13 |
Nischal Sheth |
opencontrail: status |
New |
In Progress |
|
| 2015-01-16 22:09:31 |
Nischal Sheth |
bug task added |
|
juniperopenstack |
|
| 2015-01-16 22:09:52 |
Nischal Sheth |
juniperopenstack: importance |
Undecided |
Critical |
|
| 2015-01-16 22:09:55 |
Nischal Sheth |
juniperopenstack: assignee |
|
Nischal Sheth (nsheth) |
|
| 2015-01-16 22:09:58 |
Nischal Sheth |
juniperopenstack: status |
New |
In Progress |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
nominated for series |
|
juniperopenstack/trunk |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
bug task added |
|
juniperopenstack/trunk |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
nominated for series |
|
juniperopenstack/r2.1 |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
bug task added |
|
juniperopenstack/r2.1 |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
nominated for series |
|
juniperopenstack/r1.1 |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
bug task added |
|
juniperopenstack/r1.1 |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
nominated for series |
|
juniperopenstack/r2.0 |
|
| 2015-01-16 22:10:32 |
Nischal Sheth |
bug task added |
|
juniperopenstack/r2.0 |
|
| 2015-01-16 22:10:40 |
Nischal Sheth |
juniperopenstack/r1.1: importance |
Undecided |
Critical |
|
| 2015-01-16 22:10:42 |
Nischal Sheth |
juniperopenstack/r2.0: importance |
Undecided |
Critical |
|
| 2015-01-16 22:10:44 |
Nischal Sheth |
juniperopenstack/r2.1: importance |
Undecided |
Critical |
|
| 2015-01-16 22:10:49 |
Nischal Sheth |
juniperopenstack/r1.1: status |
New |
In Progress |
|
| 2015-01-16 22:10:51 |
Nischal Sheth |
juniperopenstack/r2.0: status |
New |
In Progress |
|
| 2015-01-16 22:10:53 |
Nischal Sheth |
juniperopenstack/r2.1: status |
New |
In Progress |
|
| 2015-01-16 22:10:57 |
Nischal Sheth |
juniperopenstack/r1.1: assignee |
|
Nischal Sheth (nsheth) |
|
| 2015-01-16 22:11:00 |
Nischal Sheth |
juniperopenstack/r2.0: assignee |
|
Nischal Sheth (nsheth) |
|
| 2015-01-16 22:11:02 |
Nischal Sheth |
juniperopenstack/r2.1: assignee |
|
Nischal Sheth (nsheth) |
|
| 2015-01-19 03:53:09 |
Nagabhushana R |
tags |
contrail-control |
blocker contrail-control |
|
| 2015-01-26 17:46:24 |
Nischal Sheth |
juniperopenstack/r1.1: status |
In Progress |
Fix Committed |
|
| 2015-01-26 17:46:27 |
Nischal Sheth |
juniperopenstack/r2.1: status |
In Progress |
Fix Committed |
|
| 2015-01-26 17:46:30 |
Nischal Sheth |
juniperopenstack/trunk: status |
In Progress |
Fix Committed |
|
| 2015-01-26 17:46:34 |
Nischal Sheth |
opencontrail: status |
In Progress |
Fix Committed |
|
| 2015-01-26 17:46:54 |
Nischal Sheth |
juniperopenstack/trunk: milestone |
|
r2.1-fcs |
|
| 2015-01-26 21:14:53 |
Nischal Sheth |
juniperopenstack/r2.0: status |
In Progress |
Fix Committed |
|
| 2015-01-31 00:50:58 |
Nischal Sheth |
juniperopenstack/r2.1: milestone |
|
r2.1-fcs |
|
| 2015-02-23 17:31:35 |
Nischal Sheth |
juniperopenstack/r2.0: milestone |
|
r2.01 |
|
| 2015-02-23 18:02:24 |
Nischal Sheth |
juniperopenstack/trunk: milestone |
r2.1-fcs |
r2.20-fcs |
|
| 2015-03-04 23:16:34 |
Nischal Sheth |
juniperopenstack/r1.1: milestone |
|
r1.22 |
|
| 2015-03-06 08:24:19 |
Nischal Sheth |
juniperopenstack/r2.1: status |
Fix Committed |
Fix Released |
|
| 2015-03-06 08:28:33 |
Nischal Sheth |
opencontrail: status |
Fix Committed |
Fix Released |
|
| 2015-03-11 16:30:57 |
Nischal Sheth |
juniperopenstack/r2.0: status |
Fix Committed |
Fix Released |
|
