Juniper Openstack

fab backup_cassandra_db is throwing exception for skip_keyspace=["DISCOVERY_SERVER", "system"] config

Series trunk
Bug #1567067

Bug #1567067 reported by Vivek on 2016-04-06

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.20	Fix Committed	High	aswani kumar	Juniper Openstack r2.23 "r2.23"
R2.21.x	Fix Committed	High	aswani kumar	Juniper Openstack r2.21.2
R2.22.x	Fix Committed	High	aswani kumar	Juniper Openstack r2.22.2
R3.0	Fix Committed	High	aswani kumar	Juniper Openstack r3.0.2.0
Trunk	Fix Committed	High	aswani kumar	Juniper Openstack r3.1.0.0-fcs

Bug Description

Version:2.22.1-8

When system keyspace is skipped, it is stripping the system_trace to _trace and trying to snapshot

In the log we can see:
2016-04-06 13:32:50:200330: [root@99.1.1.4] sudo: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent

The traceback:
2016-04-06 13:32:50:098571: 2016-04-06 13:32:50:098445: [root@99.1.1.4] sudo: ls
2016-04-06 13:32:50:113563: [root@99.1.1.4] out: config_db_uuid dm_keyspace system_traces
2016-04-06 13:32:50:195111: [root@99.1.1.4] out: ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace
2016-04-06 13:32:50:195331: [root@99.1.1.4] out: DISCOVERY_SERVER system useragent
2016-04-06 13:32:50:195455: [root@99.1.1.4] out:
2016-04-06 13:32:50:195787:
2016-04-06 13:32:50:200330: [root@99.1.1.4] sudo: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent
2016-04-06 13:32:50:200525: [root@99.1.1.4] out: Requested creating snapshot for: config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent
2016-04-06 13:32:50:984052: [root@99.1.1.4] out: Exception in thread "main" java.io.IOException: Table _traces does not exist
2016-04-06 13:32:50:999911: [root@99.1.1.4] out: at org.apache.cassandra.service.StorageService.getValidTable(StorageService.java:2267)
2016-04-06 13:32:51:000275: [root@99.1.1.4] out: at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2222)
2016-04-06 13:32:51:000361: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2016-04-06 13:32:51:000563: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2016-04-06 13:32:51:001042: [root@99.1.1.4] out: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2016-04-06 13:32:51:001131: [root@99.1.1.4] out: at java.lang.reflect.Method.invoke(Method.java:606)
2016-04-06 13:32:51:001237: [root@99.1.1.4] out: at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
2016-04-06 13:32:51:001503: [root@99.1.1.4] out: at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
2016-04-06 13:32:51:002036: [root@99.1.1.4] out: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2016-04-06 13:32:51:002699: [root@99.1.1.4] out: at java.lang.reflect.Method.invoke(Method.java:606)
2016-04-06 13:32:51:003032: [root@99.1.1.4] out: at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
2016-04-06 13:32:51:003450: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
2016-04-06 13:32:51:003844: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
2016-04-06 13:32:51:004216: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
2016-04-06 13:32:51:004599: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
2016-04-06 13:32:51:004902: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
2016-04-06 13:32:51:005250: [root@99.1.1.4] out: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
2016-04-06 13:32:51:005631: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
2016-04-06 13:32:51:006986: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1491)
2016-04-06 13:32:51:007237: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
2016-04-06 13:32:51:007637: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1332)
2016-04-06 13:32:51:008104: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1424)
2016-04-06 13:32:51:008568: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:852)
2016-04-06 13:32:51:008640: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2016-04-06 13:32:51:008707: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2016-04-06 13:32:51:008774: [root@99.1.1.4] out: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2016-04-06 13:32:51:008841: [root@99.1.1.4] out: at java.lang.reflect.Method.invoke(Method.java:606)
2016-04-06 13:32:51:008907: [root@99.1.1.4] out: at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
2016-04-06 13:32:51:008972: [root@99.1.1.4] out: at sun.rmi.transport.Transport$2.run(Transport.java:202)
2016-04-06 13:32:51:009038: [root@99.1.1.4] out: at sun.rmi.transport.Transport$2.run(Transport.java:199)
2016-04-06 13:32:51:009103: [root@99.1.1.4] out: at java.security.AccessController.doPrivileged(Native Method)
2016-04-06 13:32:51:009168: [root@99.1.1.4] out: at sun.rmi.transport.Transport.serviceCall(Transport.java:198)
2016-04-06 13:32:51:009234: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)
2016-04-06 13:32:51:009300: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
2016-04-06 13:32:51:009367: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)
2016-04-06 13:32:51:009490: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)
2016-04-06 13:32:51:009587: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)
2016-04-06 13:32:51:009682: [root@99.1.1.4] out: at java.security.AccessController.doPrivileged(Native Method)
2016-04-06 13:32:51:011054: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)
2016-04-06 13:32:51:011204: [root@99.1.1.4] out: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-04-06 13:32:51:011336: [root@99.1.1.4] out: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-04-06 13:32:51:011435: [root@99.1.1.4] out: at java.lang.Thread.run(Thread.java:745)
2016-04-06 13:32:51:011529: [root@99.1.1.4] out:
2016-04-06 13:32:51:027389:

2016-04-06 13:32:51:033116: Fatal error: sudo() received nonzero return code 1 while executing!
2016-04-06 13:32:51:033116:
2016-04-06 13:32:51:033116: Requested: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent
2016-04-06 13:32:51:033116: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent "

See original description

Vivek (vivekgarg) on 2016-04-06

Changed in juniperopenstack:
milestone:	r2.22 → r2.22.1
description:	updated

Revision history for this message

Ignatious Johnson Christopher (ijohnson-x) wrote on 2016-04-07:

Workaround:
==============

workaround is to list the system_traces before system in the skip_keyspace list.

skip_keyspace = ["system_traces", "system"]

Changed in juniperopenstack:
milestone:	r2.22.1 → r3.1.0.0-fcs

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-07: [Review update] master

Review in progress for https://review.opencontrail.org/19128
Submitter: aswani kumar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-07: [Review update] R2.20

Review in progress for https://review.opencontrail.org/19129
Submitter: aswani kumar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-07: [Review update] R3.0

Review in progress for https://review.opencontrail.org/19137
Submitter: aswani kumar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-07: [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/19097
Submitter: aswani kumar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-07: [Review update] R2.22.x

#10

Review in progress for https://review.opencontrail.org/19098
Submitter: aswani kumar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-09: A change has been merged

#12

Reviewed: https://review.opencontrail.org/19098
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/b5453c4bcf9898199a9cec37f2084e1ba4fcd9a7
Submitter: Zuul
Branch: R2.22.x

commit b5453c4bcf9898199a9cec37f2084e1ba4fcd9a7
Author: Aswani Kumar Reddy G <email address hidden>
Date: Wed Apr 6 10:04:20 2016 +0530

Fix is to consistently backup at <hostname>/data/

1)cassandra backup location needs to be consistent even in case of the custom
data_file_directories configured in cassandra.yaml
2)replace_key routine was replacing all the matches of the skip_key
3)fix added to replace only the exact match

Change-Id: If0e90ac0ce248b06b7748a568ede3fcc13ac264e
Closes-Bug: #1566022
Closes-Bug: #1567067

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-09:

#13

Reviewed: https://review.opencontrail.org/19097
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/486ace9fbdd9560af296df0475b2ab122636c0c6
Submitter: Zuul
Branch: R2.21.x

commit 486ace9fbdd9560af296df0475b2ab122636c0c6
Author: Aswani Kumar Reddy G <email address hidden>
Date: Wed Apr 6 10:04:20 2016 +0530

Fix is to consistently backup at <hostname>/data/

Change-Id: If0e90ac0ce248b06b7748a568ede3fcc13ac264e
Closes-Bug: #1566022
Closes-Bug: #1567067

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-11:

#14

Reviewed: https://review.opencontrail.org/19129
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/9d8d45eee5c96009d7a81133ed6fd0a6dad789af
Submitter: Zuul
Branch: R2.20

commit 9d8d45eee5c96009d7a81133ed6fd0a6dad789af
Author: Aswani Kumar Reddy G <email address hidden>
Date: Thu Apr 7 10:28:22 2016 +0530

fix for exception while skipping system keyspace

1)replace_key routine replacing all the matches of the skip_key
2)fix added to replace only the exact match

Change-Id: Id3e067df7d90edd7a3f728b4babe8d1ecd2b3f75
Closes-Bug: #1567067

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-11:

#15

Reviewed: https://review.opencontrail.org/19137
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/5307f5535b845a7a842c81bda68c1bb8b69195ef
Submitter: Zuul
Branch: R3.0

commit 5307f5535b845a7a842c81bda68c1bb8b69195ef
Author: Aswani Kumar Reddy G <email address hidden>
Date: Thu Apr 7 11:33:28 2016 +0530

fix for exception while skipping system keyspace

1)replace_key routine replacing all the matches of the skip_key
2)fix added to replace only the exact match

Change-Id: I7a1e49eb29d81832a17e50ab4f45cb5e033584ca
Closes-Bug: #1567067

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-11:

#17

Reviewed: https://review.opencontrail.org/19128
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/3b5263f0f519d77d31718cb9ce22f954fb8e8bbd
Submitter: Zuul
Branch: master

commit 3b5263f0f519d77d31718cb9ce22f954fb8e8bbd
Author: Aswani Kumar Reddy G <email address hidden>
Date: Thu Apr 7 10:12:33 2016 +0530

fix for exception while skipping system keyspace

1)replace_key routine replacing all the matches of the skip_key
2)fix added to replace only the exact match

Change-Id: Id908f74d6f8e262d4b64c83fd01e7f98b14254fb
Closes-Bug: #1567067

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.