UpdateQueue can get wedged in pathological case

Bug #1468499 reported by Nischal Sheth
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.0
Fix Committed
High
Nischal Sheth
R2.1
Fix Released
High
Nischal Sheth
R2.20
Fix Released
High
Nischal Sheth
Trunk
Fix Committed
High
Nischal Sheth

Bug Description

Consider the following sequence of events:

- A single peer which has joined a ribout and hence it's update queues
- Multiple routes are added which results in updates on update queue
- Peer gets blocked after some updates have been sent to it
- Since the peer is the only one in the queue it's marker is not split out
- IOW the tail marker is somewhere in the middle of the update queue
- New peer joins the ribout and and hence the update queues
- Bit for the new peer is added to the tail marker
- New routes get added to table and are enqueued at the end of the queue
- Tail dequeue for ribout is not kicked since tail marker is not at end
- First peer leaves and it's bit in the updates is cleaned up

The new peer doesn't receive the newly added routes and any subsequent
routes that get enqueued to the update queue for the ribout i.e it's wedged.

Nischal Sheth (nsheth)
description: updated
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/12017
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/12018
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.0

Review in progress for https://review.opencontrail.org/12019
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.1

Review in progress for https://review.opencontrail.org/12020
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/12017
Committed: http://github.org/Juniper/contrail-controller/commit/a519451728943779d57a1a7f5c3c4520e6315ec1
Submitter: Zuul
Branch: master

commit a519451728943779d57a1a7f5c3c4520e6315ec1
Author: Nischal Sheth <email address hidden>
Date: Wed Jun 24 12:56:59 2015 -0700

Fix corner case in RibOut tail dequeue scheduling logic

An UpdateQueue can wedged in a pathological case where a new peer joins
when all existing peers are currently blocked, there are pending updates
on the queue and all blocked peers leave the RibOut prior to unblocking.

Fix by triggering tail dequeue for the RibOut if the tail marker is not
at the end of the UpdateQueue when a new peer joins the queue. This may
trigger some unnecessary tail dequeues for the RibOut but that does not
cost much.

Change-Id: I08182dd883bfef7caa753cf6bca8d2a9e8e5eed9
Closes-Bug: 1468499

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/12019
Committed: http://github.org/Juniper/contrail-controller/commit/721883954784765ba3c812327eb52415b2330e67
Submitter: Zuul
Branch: R2.0

commit 721883954784765ba3c812327eb52415b2330e67
Author: Nischal Sheth <email address hidden>
Date: Wed Jun 24 12:56:59 2015 -0700

Fix corner case in RibOut tail dequeue scheduling logic

An UpdateQueue can wedged in a pathological case where a new peer joins
when all existing peers are currently blocked, there are pending updates
on the queue and all blocked peers leave the RibOut prior to unblocking.

Fix by triggering tail dequeue for the RibOut if the tail marker is not
at the end of the UpdateQueue when a new peer joins the queue. This may
trigger some unnecessary tail dequeues for the RibOut but that does not
cost much.

Change-Id: I08182dd883bfef7caa753cf6bca8d2a9e8e5eed9
Closes-Bug: 1468499

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/12018
Committed: http://github.org/Juniper/contrail-controller/commit/9f4a2c4245dd2e47830976e8fe2e580651da73bc
Submitter: Zuul
Branch: R2.20

commit 9f4a2c4245dd2e47830976e8fe2e580651da73bc
Author: Nischal Sheth <email address hidden>
Date: Wed Jun 24 12:56:59 2015 -0700

Fix corner case in RibOut tail dequeue scheduling logic

An UpdateQueue can wedged in a pathological case where a new peer joins
when all existing peers are currently blocked, there are pending updates
on the queue and all blocked peers leave the RibOut prior to unblocking.

Fix by triggering tail dequeue for the RibOut if the tail marker is not
at the end of the UpdateQueue when a new peer joins the queue. This may
trigger some unnecessary tail dequeues for the RibOut but that does not
cost much.

Change-Id: I08182dd883bfef7caa753cf6bca8d2a9e8e5eed9
Closes-Bug: 1468499

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/12020
Committed: http://github.org/Juniper/contrail-controller/commit/bfad13e669b0ba9b243c30e0e1efe181361011d5
Submitter: Zuul
Branch: R2.1

commit bfad13e669b0ba9b243c30e0e1efe181361011d5
Author: Nischal Sheth <email address hidden>
Date: Wed Jun 24 12:56:59 2015 -0700

Fix corner case in RibOut tail dequeue scheduling logic

An UpdateQueue can wedged in a pathological case where a new peer joins
when all existing peers are currently blocked, there are pending updates
on the queue and all blocked peers leave the RibOut prior to unblocking.

Fix by triggering tail dequeue for the RibOut if the tail marker is not
at the end of the UpdateQueue when a new peer joins the queue. This may
trigger some unnecessary tail dequeues for the RibOut but that does not
cost much.

Change-Id: I08182dd883bfef7caa753cf6bca8d2a9e8e5eed9
Closes-Bug: 1468499

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.