Juniper Openstack

contrail-status of database says inactive , where as the Cassandra up and running fine

Bug #1690660 reported by Vinoth Kannan Ganapathy
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
Medium
Sundaresan Rajangam
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"
Trunk
Fix Committed
Medium
Sundaresan Rajangam
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

In the contrail-HA system , contrail-status shows contrail-database as inactive ,
where the node status says up and normal.

root@5b8s31(analyticsdb):/#
root@5b8s31(analyticsdb):/# contrail-status
== Contrail Database ==
contrail-database: inactive

contrail-database-nodemgr active
kafka active

root@5b8s31(analyticsdb):/# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.17.90.2 2.49 GB 256 ? a478fbed-ba2e-4cc3-92e2-ec29d16d1c2d rack1
UN 172.17.90.3 2.58 GB 256 ? 5779a62d-ae63-4930-95bc-aefdad8e9746 rack1
UN 172.17.90.1 2.35 GB 256 ? 5de991d6-aa72-43ea-854e-fd64ff3e68e2 rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
root@5b8s31(analyticsdb):/#
root@5b8s31(analyticsdb):/# ifconfig | grep 172.17
          inet addr:172.17.90.3 Bcast:172.17.90.255 Mask:255.255.255.0
root@5b8s31(analyticsdb):/#

Revision history for this message
Megh Bhatt (meghb) wrote : Re: [Bug 1690660] [NEW] contrail-status of database says inactive , where as the Cassandra up and running fine
Download full text (3.4 KiB)

What build? Which OS?

Please provide all these information in the bug report to avoid going back and forth.

Also provide the following:

ps aux | grep cass

netstat -lnp | grep <pid> of the cassandra process

nodetool statusbinary

Thanks

Megh

> On May 14, 2017, at 11:08 AM, Vinoth Kannan Ganapathy <email address hidden> wrote:
>
> Private bug reported:
>
>
> In the contrail-HA system , contrail-status shows contrail-database as inactive ,
> where the node status says up and normal.
>
> root@5b8s31(analyticsdb):/#
> root@5b8s31(analyticsdb):/# contrail-status
> == Contrail Database ==
> contrail-database: inactive
>
> contrail-database-nodemgr active
> kafka active
>
> root@5b8s31(analyticsdb):/# nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> UN 172.17.90.2 2.49 GB 256 ? a478fbed-ba2e-4cc3-92e2-ec29d16d1c2d rack1
> UN 172.17.90.3 2.58 GB 256 ? 5779a62d-ae63-4930-95bc-aefdad8e9746 rack1
> UN 172.17.90.1 2.35 GB 256 ? 5de991d6-aa72-43ea-854e-fd64ff3e68e2 rack1
>
> Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
> root@5b8s31(analyticsdb):/#
> root@5b8s31(analyticsdb):/# ifconfig | grep 172.17
> inet addr:172.17.90.3 Bcast:172.17.90.255 Mask:255.255.255.0
> root@5b8s31(analyticsdb):/#
>
> ** Affects: juniperopenstack
> Importance: Medium
> Assignee: Sundaresan Rajangam (srajanga)
> Status: New
>
>
> ** Tags: analytics
>
> --
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://bugs.launchpad.net/bugs/1690660
>
> Title:
> contrail-status of database says inactive , where as the Cassandra up
> and running fine
>
> Status in Juniper Openstack:
> New
>
> Bug description:
>
> In the contrail-HA system , contrail-status shows contrail-database as inactive ,
> where the node status says up and normal.
>
> root@5b8s31(analyticsdb):/#
> root@5b8s31(analyticsdb):/# contrail-status
> == Contrail Database ==
> contrail-database: inactive
>
> contrail-database-nodemgr active
> kafka active
>
> root@5b8s31(analyticsdb):/# nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> UN 172.17.90.2 2.49 GB 256 ? a478fbed-ba2e-4cc3-92e2-ec29d16d1c2d rack1
> UN 172.17.90.3 2.58 GB 256 ? 5779a62d-ae63-4930-95bc-aefdad8e9746 rack1
> UN 172.17.90.1 2.35 GB 256 ? 5de991d6-aa72-43ea-854e-fd64ff3e68e2 rack1
>
> Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
> root@5b8s31(analyticsdb):/#
> root@5b8s31(analyticsdb):/# ifconfig | grep 172....

Read more...

Revision history for this message
Vinoth Kannan Ganapathy (vganapathy) wrote :
Download full text (9.1 KiB)

Hi Megh,

contrail-version 4.0 build 2 , SKU Mitaka.
Don't see any functionality , hence filed it as medium..

root@5b8s31(analyticsdb):/# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-database-common 4.0.0.0-2 2
contrail-nodemgr 4.0.0.0-2 2
contrail-openstack-database 4.0.0.0-2 2
contrail-setup 4.0.0.0-2 2
contrail-utils 4.0.0.0-2 2
python-contrail 4.0.0.0-2 2
root@5b8s31(analyticsdb):/# ps aux | grep cass
cassand+ 22566 9.5 4.0 12798724 10813988 ? SLl May11 385:41 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+PerfDisableSharedMem -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=10000 -XX:+UseCondCardMark -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Djava.net.preferIPv4Stack=true -Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC -Djava.library.path=/usr/share/cassandra/lib/sigar-bin -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/crc32ex-0.1.1.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/...

Read more...

Jeba Paulaiyan (jebap)
tags: added: sanity
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/31800
Submitter: Sundaresan Rajangam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/31801
Submitter: Sundaresan Rajangam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/31802
Submitter: Sundaresan Rajangam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/31803
Submitter: Sundaresan Rajangam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/31803
Committed: http://github.com/Juniper/contrail-packaging/commit/a6dbd0fdb188b46bc740cc1c08c290a63fe848f3
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit a6dbd0fdb188b46bc740cc1c08c290a63fe848f3
Author: Sundaresan Rajangam <email address hidden>
Date: Thu May 18 14:22:22 2017 -0700

contrail-database restart should invoke custom start code

- contrail-database restart|reload should call custom code to start
cassandra instead of calling cassandra start directly.
- added sleep to avoid race condition when contrail-database restart is
invoked multiple times in succession.

Change-Id: Ie059323ee01f0594cc4dc7a8adfc5d20730edf1b
Closes-Bug: #1690660

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31801
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/0ce373379cd2b40e95742dc5018c343e3d55e700
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 0ce373379cd2b40e95742dc5018c343e3d55e700
Author: Sundaresan Rajangam <email address hidden>
Date: Thu May 18 14:11:24 2017 -0700

Cassandra service should be started with contrail-database

Should use /etc/init.d/contrail-database instead of
/etc/init.d/cassandra to start/restart cassandra service

Change-Id: Id26b81c2f9b55dc28004014232632cfa8b442e4e
Partial-Bug: #1690660

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31800
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/cc030035d4e1c5347ce1c50d7d0e8fd55ae2261d
Submitter: Zuul (<email address hidden>)
Branch: master

commit cc030035d4e1c5347ce1c50d7d0e8fd55ae2261d
Author: Sundaresan Rajangam <email address hidden>
Date: Thu May 18 14:11:24 2017 -0700

Cassandra service should be started with contrail-database

Should use /etc/init.d/contrail-database instead of
/etc/init.d/cassandra to start/restart cassandra service

Change-Id: Id26b81c2f9b55dc28004014232632cfa8b442e4e
Partial-Bug: #1690660

Sundaresan Rajangam (srajanga)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31802
Committed: http://github.com/Juniper/contrail-packaging/commit/cec49074498f32ad8091c7b62418ea538c754387
Submitter: Zuul (<email address hidden>)
Branch: master

commit cec49074498f32ad8091c7b62418ea538c754387
Author: Sundaresan Rajangam <email address hidden>
Date: Thu May 18 14:22:22 2017 -0700

contrail-database restart should invoke custom start code

- contrail-database restart|reload should call custom code to start
cassandra instead of calling cassandra start directly.
- added sleep to avoid race condition when contrail-database restart is
invoked multiple times in succession.

Change-Id: Ie059323ee01f0594cc4dc7a8adfc5d20730edf1b
Closes-Bug: #1690660

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.