When system keyspace is skipped, it is stripping the system_trace to _trace and trying to snapshot
In the log we can see: 2016-04-06 13:32:50:200330: [root@99.1.1.4] sudo: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent
The traceback: 2016-04-06 13:32:50:098571: 2016-04-06 13:32:50:098445: [root@99.1.1.4] sudo: ls 2016-04-06 13:32:50:113563: [root@99.1.1.4] out: config_db_uuid dm_keyspace system_traces 2016-04-06 13:32:50:195111: [root@99.1.1.4] out: ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace 2016-04-06 13:32:50:195331: [root@99.1.1.4] out: DISCOVERY_SERVER system useragent 2016-04-06 13:32:50:195455: [root@99.1.1.4] out: 2016-04-06 13:32:50:195787: 2016-04-06 13:32:50:200330: [root@99.1.1.4] sudo: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent 2016-04-06 13:32:50:200525: [root@99.1.1.4] out: Requested creating snapshot for: config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent 2016-04-06 13:32:50:984052: [root@99.1.1.4] out: Exception in thread "main" java.io.IOException: Table _traces does not exist 2016-04-06 13:32:50:999911: [root@99.1.1.4] out: at org.apache.cassandra.service.StorageService.getValidTable(StorageService.java:2267) 2016-04-06 13:32:51:000275: [root@99.1.1.4] out: at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2222) 2016-04-06 13:32:51:000361: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2016-04-06 13:32:51:000563: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 2016-04-06 13:32:51:001042: [root@99.1.1.4] out: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2016-04-06 13:32:51:001131: [root@99.1.1.4] out: at java.lang.reflect.Method.invoke(Method.java:606) 2016-04-06 13:32:51:001237: [root@99.1.1.4] out: at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) 2016-04-06 13:32:51:001503: [root@99.1.1.4] out: at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) 2016-04-06 13:32:51:002036: [root@99.1.1.4] out: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2016-04-06 13:32:51:002699: [root@99.1.1.4] out: at java.lang.reflect.Method.invoke(Method.java:606) 2016-04-06 13:32:51:003032: [root@99.1.1.4] out: at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) 2016-04-06 13:32:51:003450: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) 2016-04-06 13:32:51:003844: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) 2016-04-06 13:32:51:004216: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) 2016-04-06 13:32:51:004599: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) 2016-04-06 13:32:51:004902: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) 2016-04-06 13:32:51:005250: [root@99.1.1.4] out: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) 2016-04-06 13:32:51:005631: [root@99.1.1.4] out: at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) 2016-04-06 13:32:51:006986: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1491) 2016-04-06 13:32:51:007237: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) 2016-04-06 13:32:51:007637: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1332) 2016-04-06 13:32:51:008104: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1424) 2016-04-06 13:32:51:008568: [root@99.1.1.4] out: at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:852) 2016-04-06 13:32:51:008640: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2016-04-06 13:32:51:008707: [root@99.1.1.4] out: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 2016-04-06 13:32:51:008774: [root@99.1.1.4] out: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2016-04-06 13:32:51:008841: [root@99.1.1.4] out: at java.lang.reflect.Method.invoke(Method.java:606) 2016-04-06 13:32:51:008907: [root@99.1.1.4] out: at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) 2016-04-06 13:32:51:008972: [root@99.1.1.4] out: at sun.rmi.transport.Transport$2.run(Transport.java:202) 2016-04-06 13:32:51:009038: [root@99.1.1.4] out: at sun.rmi.transport.Transport$2.run(Transport.java:199) 2016-04-06 13:32:51:009103: [root@99.1.1.4] out: at java.security.AccessController.doPrivileged(Native Method) 2016-04-06 13:32:51:009168: [root@99.1.1.4] out: at sun.rmi.transport.Transport.serviceCall(Transport.java:198) 2016-04-06 13:32:51:009234: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567) 2016-04-06 13:32:51:009300: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828) 2016-04-06 13:32:51:009367: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619) 2016-04-06 13:32:51:009490: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684) 2016-04-06 13:32:51:009587: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681) 2016-04-06 13:32:51:009682: [root@99.1.1.4] out: at java.security.AccessController.doPrivileged(Native Method) 2016-04-06 13:32:51:011054: [root@99.1.1.4] out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681) 2016-04-06 13:32:51:011204: [root@99.1.1.4] out: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2016-04-06 13:32:51:011336: [root@99.1.1.4] out: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2016-04-06 13:32:51:011435: [root@99.1.1.4] out: at java.lang.Thread.run(Thread.java:745) 2016-04-06 13:32:51:011529: [root@99.1.1.4] out: 2016-04-06 13:32:51:027389:
2016-04-06 13:32:51:033116: Fatal error: sudo() received nonzero return code 1 while executing! 2016-04-06 13:32:51:033116: 2016-04-06 13:32:51:033116: Requested: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent 2016-04-06 13:32:51:033116: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_keyspace to_bgp_keyspace useragent "
When system keyspace is skipped, it is stripping the system_trace to _trace and trying to snapshot
In the log we can see: keyspace to_bgp_keyspace useragent
2016-04-06 13:32:50:200330: [root@99.1.1.4] sudo: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_
The traceback: keyspace to_bgp_keyspace keyspace to_bgp_keyspace useragent keyspace to_bgp_keyspace useragent IOException: Table _traces does not exist cassandra. service. StorageService. getValidTable( StorageService. java:2267) cassandra. service. StorageService. takeSnapshot( StorageService. java:2222) NativeMethodAcc essorImpl. invoke0( Native Method) NativeMethodAcc essorImpl. invoke( NativeMethodAcc essorImpl. java:57) DelegatingMetho dAccessorImpl. invoke( DelegatingMetho dAccessorImpl. java:43) reflect. Method. invoke( Method. java:606) misc.Trampoline .invoke( MethodUtil. java:75) GeneratedMethod Accessor7. invoke( Unknown Source) DelegatingMetho dAccessorImpl. invoke( DelegatingMetho dAccessorImpl. java:43) reflect. Method. invoke( Method. java:606) misc.MethodUtil .invoke( MethodUtil. java:279) jmx.mbeanserver .StandardMBeanI ntrospector. invokeM2( StandardMBeanIn trospector. java:112) jmx.mbeanserver .StandardMBeanI ntrospector. invokeM2( StandardMBeanIn trospector. java:46) jmx.mbeanserver .MBeanIntrospec tor.invokeM( MBeanIntrospect or.java: 237) jmx.mbeanserver .PerInterface. invoke( PerInterface. java:138) jmx.mbeanserver .MBeanSupport. invoke( MBeanSupport. java:252) jmx.interceptor .DefaultMBeanSe rverInterceptor .invoke( DefaultMBeanSer verInterceptor. java:819) jmx.mbeanserver .JmxMBeanServer .invoke( JmxMBeanServer. java:801) t.remote. rmi.RMIConnecti onImpl. doOperation( RMIConnectionIm pl.java: 1491) t.remote. rmi.RMIConnecti onImpl. access$ 300(RMIConnecti onImpl. java:97) t.remote. rmi.RMIConnecti onImpl$ PrivilegedOpera tion.run( RMIConnectionIm pl.java: 1332) t.remote. rmi.RMIConnecti onImpl. doPrivilegedOpe ration( RMIConnectionIm pl.java: 1424) t.remote. rmi.RMIConnecti onImpl. invoke( RMIConnectionIm pl.java: 852) NativeMethodAcc essorImpl. invoke0( Native Method) NativeMethodAcc essorImpl. invoke( NativeMethodAcc essorImpl. java:57) DelegatingMetho dAccessorImpl. invoke( DelegatingMetho dAccessorImpl. java:43) reflect. Method. invoke( Method. java:606) server. UnicastServerRe f.dispatch( UnicastServerRe f.java: 322) transport. Transport$ 2.run(Transport .java:202) transport. Transport$ 2.run(Transport .java:199) AccessControlle r.doPrivileged( Native Method) transport. Transport. serviceCall( Transport. java:198) transport. tcp.TCPTranspor t.handleMessage s(TCPTransport. java:567) transport. tcp.TCPTranspor t$ConnectionHan dler.run0( TCPTransport. java:828) transport. tcp.TCPTranspor t$ConnectionHan dler.access$ 400(TCPTranspor t.java: 619) transport. tcp.TCPTranspor t$ConnectionHan dler$1. run(TCPTranspor t.java: 684) transport. tcp.TCPTranspor t$ConnectionHan dler$1. run(TCPTranspor t.java: 681) AccessControlle r.doPrivileged( Native Method) transport. tcp.TCPTranspor t$ConnectionHan dler.run( TCPTransport. java:681) concurrent. ThreadPoolExecu tor.runWorker( ThreadPoolExecu tor.java: 1145) concurrent. ThreadPoolExecu tor$Worker. run(ThreadPoolE xecutor. java:615) Thread. run(Thread. java:745)
2016-04-06 13:32:50:098571: 2016-04-06 13:32:50:098445: [root@99.1.1.4] sudo: ls
2016-04-06 13:32:50:113563: [root@99.1.1.4] out: config_db_uuid dm_keyspace system_traces
2016-04-06 13:32:50:195111: [root@99.1.1.4] out: ContrailAnalytics svc_monitor_
2016-04-06 13:32:50:195331: [root@99.1.1.4] out: DISCOVERY_SERVER system useragent
2016-04-06 13:32:50:195455: [root@99.1.1.4] out:
2016-04-06 13:32:50:195787:
2016-04-06 13:32:50:200330: [root@99.1.1.4] sudo: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_
2016-04-06 13:32:50:200525: [root@99.1.1.4] out: Requested creating snapshot for: config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_
2016-04-06 13:32:50:984052: [root@99.1.1.4] out: Exception in thread "main" java.io.
2016-04-06 13:32:50:999911: [root@99.1.1.4] out: at org.apache.
2016-04-06 13:32:51:000275: [root@99.1.1.4] out: at org.apache.
2016-04-06 13:32:51:000361: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:000563: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:001042: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:001131: [root@99.1.1.4] out: at java.lang.
2016-04-06 13:32:51:001237: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:001503: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:002036: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:002699: [root@99.1.1.4] out: at java.lang.
2016-04-06 13:32:51:003032: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:003450: [root@99.1.1.4] out: at com.sun.
2016-04-06 13:32:51:003844: [root@99.1.1.4] out: at com.sun.
2016-04-06 13:32:51:004216: [root@99.1.1.4] out: at com.sun.
2016-04-06 13:32:51:004599: [root@99.1.1.4] out: at com.sun.
2016-04-06 13:32:51:004902: [root@99.1.1.4] out: at com.sun.
2016-04-06 13:32:51:005250: [root@99.1.1.4] out: at com.sun.
2016-04-06 13:32:51:005631: [root@99.1.1.4] out: at com.sun.
2016-04-06 13:32:51:006986: [root@99.1.1.4] out: at javax.managemen
2016-04-06 13:32:51:007237: [root@99.1.1.4] out: at javax.managemen
2016-04-06 13:32:51:007637: [root@99.1.1.4] out: at javax.managemen
2016-04-06 13:32:51:008104: [root@99.1.1.4] out: at javax.managemen
2016-04-06 13:32:51:008568: [root@99.1.1.4] out: at javax.managemen
2016-04-06 13:32:51:008640: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:008707: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:008774: [root@99.1.1.4] out: at sun.reflect.
2016-04-06 13:32:51:008841: [root@99.1.1.4] out: at java.lang.
2016-04-06 13:32:51:008907: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:008972: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009038: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009103: [root@99.1.1.4] out: at java.security.
2016-04-06 13:32:51:009168: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009234: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009300: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009367: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009490: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009587: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:009682: [root@99.1.1.4] out: at java.security.
2016-04-06 13:32:51:011054: [root@99.1.1.4] out: at sun.rmi.
2016-04-06 13:32:51:011204: [root@99.1.1.4] out: at java.util.
2016-04-06 13:32:51:011336: [root@99.1.1.4] out: at java.util.
2016-04-06 13:32:51:011435: [root@99.1.1.4] out: at java.lang.
2016-04-06 13:32:51:011529: [root@99.1.1.4] out:
2016-04-06 13:32:51:027389:
2016-04-06 13:32:51:033116: Fatal error: sudo() received nonzero return code 1 while executing! keyspace to_bgp_keyspace useragent keyspace to_bgp_keyspace useragent "
2016-04-06 13:32:51:033116:
2016-04-06 13:32:51:033116: Requested: nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_
2016-04-06 13:32:51:033116: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "nodetool -h localhost -p 7199 snapshot config_db_uuid dm_keyspace _traces ContrailAnalytics svc_monitor_