When schema transformer/svc-monitor/device-manager are restarted, it can take a long time to elect new master

Bug #1503407 reported by Sachin Bansal
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
Undecided
Sachin Bansal
R2.21.x
Fix Committed
Undecided
Sachin Bansal
Trunk
Fix Committed
Undecided
Sachin Bansal

Bug Description

When schema transformer/svc-monitor/device-manager are restarted, it can take a long time to elect new master even in a single node setup. This happens because supervisord kills the process using SIGKILL. This leaves the orphan entry in zookeeper until zookeeper is able to detect that the connection is gone.

The solution is to use SIGTERM to kill the processes and handle the signal to clean up kombu greenlets.

Tags: config
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/14268
Submitter: Sachin Bansal (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/14268
Committed: http://github.org/Juniper/contrail-controller/commit/b5bd298348df6bb52d31040ae3b7d0a923f41d4f
Submitter: Zuul
Branch: master

commit b5bd298348df6bb52d31040ae3b7d0a923f41d4f
Author: Sachin Bansal <email address hidden>
Date: Tue Oct 6 12:14:43 2015 -0700

Use SIGTERM to kill config daemons and kill all greenlets when SIGTERM is received

Change-Id: Id29300c4ed64ea237829f02f691dd18fff14ebe5
Closes-Bug: 1503407

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/17155
Submitter: Sachin Bansal (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/17155
Submitter: Suresh Balineni (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17155
Committed: http://github.org/Juniper/contrail-controller/commit/7cc26b9325e4e87b17b057a5f7bb730c0267dc33
Submitter: Zuul
Branch: R2.20

commit 7cc26b9325e4e87b17b057a5f7bb730c0267dc33
Author: Sachin Bansal <email address hidden>
Date: Tue Oct 6 12:14:43 2015 -0700

Use SIGTERM to kill config daemons and kill all greenlets when SIGTERM is received

Change-Id: Id29300c4ed64ea237829f02f691dd18fff14ebe5
Closes-Bug: 1503407
(cherry picked from commit b5bd298348df6bb52d31040ae3b7d0a923f41d4f)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/28222
Submitter: Bruno Fernando (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/28222
Committed: http://github.org/Juniper/contrail-controller/commit/5ba4a6040e6a504732ebbaba17889b7e38a2191d
Submitter: Zuul (<email address hidden>)
Branch: R2.21.x

commit 5ba4a6040e6a504732ebbaba17889b7e38a2191d
Author: Sachin Bansal <email address hidden>
Date: Tue Oct 6 12:14:43 2015 -0700

Use SIGTERM to kill config daemons and kill all greenlets when SIGTERM is received

Change-Id: Id29300c4ed64ea237829f02f691dd18fff14ebe5
Closes-Bug: 1503407
(cherry picked from commit b5bd298348df6bb52d31040ae3b7d0a923f41d4f)
(cherry picked from commit 7cc26b9325e4e87b17b057a5f7bb730c0267dc33)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/29899
Submitter: Sachin Bansal (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/29899
Committed: http://github.org/Juniper/contrail-controller/commit/33f671e7e32da5a32d168f44c583ce38ede5ddc8
Submitter: Zuul (<email address hidden>)
Branch: R2.21.x

commit 33f671e7e32da5a32d168f44c583ce38ede5ddc8
Author: Sachin Bansal <email address hidden>
Date: Mon Mar 27 10:53:53 2017 -0700

Define a new signal handler for schema transformer

Since schema transformer does not use rabbitmq before R3.0, the signal
handler defined in VncKombuClient is not used. We should define a new
signal handler for schema transformer.

Change-Id: I30bad3cb383afb973bf92a7d6312e1d3de5e1950
Closes-Bug: 1503407

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.