Multiple nodetool repair are seen concurrently in busy controller/database node

Bug #1513658 reported by Megh Bhatt
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Megh Bhatt
R2.21.x
Fix Committed
High
Megh Bhatt
Trunk
Fix Committed
High
Megh Bhatt

Bug Description

In 2.2X multiple nodetool repair processes are seen running concurrently on a busy database/controller node. Changes from https://bugs.launchpad.net/juniperopenstack/+bug/1484297 need to be back-ported to 2.2X so that this is not seen

Tags: analytics
Megh Bhatt (meghb)
Changed in juniperopenstack:
milestone: none → r2.22
assignee: nobody → Megh Bhatt (meghb)
importance: Undecided → High
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/14913
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/14913
Committed: http://github.org/Juniper/contrail-controller/commit/a5293fd28ec5a16f0a15db24ebcd2d5a61784cdb
Submitter: Zuul
Branch: R2.20

commit a5293fd28ec5a16f0a15db24ebcd2d5a61784cdb
Author: Megh Bhatt <email address hidden>
Date: Thu Nov 5 15:26:33 2015 -0800

Add contrail-cassandra-repair script that will call repair on the
keyspaces that need repair and make sure that previous repair is not
yet running. Backported from changes made for Bug #1484297.
Related-Bug: #1484297
Partial-Bug: #1513658

Change-Id: If735a64bb46bd6b1a188a8880014c5683b85c276

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/14943
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/14943
Committed: http://github.org/Juniper/contrail-packages/commit/4d249c55740af37826649b9808378bb72cd006a4
Submitter: Zuul
Branch: R2.20

commit 4d249c55740af37826649b9808378bb72cd006a4
Author: Megh Bhatt <email address hidden>
Date: Fri Nov 6 11:06:29 2015 -0800

Packages contrail-cassandra-repair script as part of contrail-utils
package for RPM and deb
Partial-Bug: #1513658
Related-Bug: #1484297

Change-Id: Ifaa98bab305551038c39658601ff70c278c5bc8d

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/14990
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/14990
Committed: http://github.org/Juniper/contrail-controller/commit/38c8f77bbd8328e0bc8f78d72241267120af9f27
Submitter: Zuul
Branch: R2.20

commit 38c8f77bbd8328e0bc8f78d72241267120af9f27
Author: Megh Bhatt <email address hidden>
Date: Mon Nov 9 13:48:57 2015 -0800

Replace calls to nodetool repair directly from database nodemgr with
contrail-cassandra-repair since it handles preventing multiple instances
of nodetool repair per keyspace running concurrently.
Closes-Bug: #1513658

Change-Id: I63c4a4123489eea2bf0597f8cc2c7dbc1ec4cb58

Megh Bhatt (meghb)
Changed in juniperopenstack:
status: New → Invalid
milestone: r2.22 → r3.0-fcs
milestone: r3.0-fcs → none
no longer affects: juniperopenstack
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/17057
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17057
Committed: http://github.org/Juniper/contrail-controller/commit/c0d0116cf26866ccda381316cca56d19fdfb9f7d
Submitter: Zuul
Branch: R2.21.x

commit c0d0116cf26866ccda381316cca56d19fdfb9f7d
Author: Megh Bhatt <email address hidden>
Date: Thu Nov 5 15:26:33 2015 -0800

Add contrail-cassandra-repair script that will call repair on the
keyspaces that need repair and make sure that previous repair is not
yet running. Backported from changes made for Bug #1484297.
Related-Bug: #1484297
Partial-Bug: #1513658

Change-Id: If735a64bb46bd6b1a188a8880014c5683b85c276
(cherry picked from commit a5293fd28ec5a16f0a15db24ebcd2d5a61784cdb)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/17200
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17200
Committed: http://github.org/Juniper/contrail-packages/commit/fce4581cebe7680b159db48845b185bdaf894aa1
Submitter: Zuul
Branch: R2.21.x

commit fce4581cebe7680b159db48845b185bdaf894aa1
Author: Megh Bhatt <email address hidden>
Date: Fri Nov 6 11:06:29 2015 -0800

Packages contrail-cassandra-repair script as part of contrail-utils
package for RPM and deb
Partial-Bug: #1513658
Related-Bug: #1484297

Change-Id: Ifaa98bab305551038c39658601ff70c278c5bc8d
(cherry picked from commit 4d249c55740af37826649b9808378bb72cd006a4)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/17297
Submitter: Megh Bhatt (<email address hidden>)

Raj Reddy (rajreddy)
Changed in juniperopenstack:
importance: Undecided → High
assignee: nobody → Megh Bhatt (meghb)
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17297
Committed: http://github.org/Juniper/contrail-controller/commit/7af84d73212aab9161a7d99558b2555698102bab
Submitter: Zuul
Branch: R2.21.x

commit 7af84d73212aab9161a7d99558b2555698102bab
Author: Megh Bhatt <email address hidden>
Date: Mon Nov 9 13:48:57 2015 -0800

Replace calls to nodetool repair directly from database nodemgr with
contrail-cassandra-repair since it handles preventing multiple instances
of nodetool repair per keyspace running concurrently.
Closes-Bug: #1513658

Change-Id: I63c4a4123489eea2bf0597f8cc2c7dbc1ec4cb58
(cherry picked from commit 38c8f77bbd8328e0bc8f78d72241267120af9f27)

Megh Bhatt (meghb)
Changed in juniperopenstack:
milestone: none → r3.0-fcs
status: New → Fix Committed
information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.