Vcenter upgrade from 2.21 to 3.0-2712 takes longer ~20 mins during 'nodetool upgradesstables'

[Sarath]

Vcenter upgrade from 2.21 to 3.0-2712 takes longer ~20 mins during 'nodetool upgradesstables'

Raj knows the problem and opening this PR for tracking.

::
::
2016-02-11 16:39:55:109008: [root@10.87.26.198] out: Configuration file '/etc/cassandra/cassandra-env.sh'
2016-02-11 16:39:55:109068: [root@10.87.26.198] out: ==> Modified (by you or by a script) since installation.
2016-02-11 16:39:55:109240: [root@10.87.26.198] out: ==> Package distributor has shipped an updated version.
2016-02-11 16:39:55:109386: [root@10.87.26.198] out: ==> Using new file as you requested.
2016-02-11 16:39:55:109444: [root@10.87.26.198] out: Installing new version of config file /etc/cassandra/cassandra-env.sh ...
2016-02-11 16:39:55:109494: [root@10.87.26.198] out: Installing new version of config file /etc/cassandra/commitlog_archiving.properties ...
2016-02-11 16:39:55:109773: [root@10.87.26.198] out:
2016-02-11 16:39:55:113064: [root@10.87.26.198] out: Configuration file '/etc/cassandra/cassandra.yaml'
2016-02-11 16:39:55:113124: [root@10.87.26.198] out: ==> Modified (by you or by a script) since installation.
2016-02-11 16:39:55:113174: [root@10.87.26.198] out: ==> Package distributor has shipped an updated version.
2016-02-11 16:39:55:113221: [root@10.87.26.198] out: ==> Using new file as you requested.
2016-02-11 16:39:55:113268: [root@10.87.26.198] out: Installing new version of config file /etc/cassandra/cassandra.yaml ...
2016-02-11 16:39:55:113315: [root@10.87.26.198] out:
2016-02-11 16:39:55:120861: [root@10.87.26.198] out: Configuration file '/etc/init.d/cassandra'
2016-02-11 16:39:55:120937: [root@10.87.26.198] out: ==> Deleted (by you or by a script) since installation.
2016-02-11 16:39:55:120988: [root@10.87.26.198] out: ==> Package distributor has shipped an updated version.
2016-02-11 16:39:55:121036: [root@10.87.26.198] out: ==> Using new file as you requested.
2016-02-11 16:39:55:121118: [root@10.87.26.198] out: Installing new version of config file /etc/init.d/cassandra ...
2016-02-11 16:39:55:121168: [root@10.87.26.198] out: Installing new version of config file /etc/sysctl.d/cassandra.conf ...
2016-02-11 16:39:55:122591: [root@10.87.26.198] out: Installing new version of config file /etc/security/limits.d/cassandra.conf ...
2016-02-11 16:39:55:125853: [root@10.87.26.198] out: vm.max_map_count = 1048575
2016-02-11 16:39:55:133221: [root@10.87.26.198] out: net.ipv4.tcp_keepalive_time = 300
2016-02-11 16:39:55:133281: [root@10.87.26.199] out: [localhost] local: service cassandra stop
2016-02-11 16:39:55:192118: [root@10.87.26.198] out: Processing triggers for python-support (1.0.15) ...
2016-02-11 16:39:55:264977: [root@10.87.26.198] out: Processing triggers for ureadahead (0.100.0-16) ...
2016-02-11 16:39:55:328654: [root@10.87.26.198] out: [localhost] local: service cassandra stop
2016-02-11 16:39:55:432151: [root@10.87.26.197] out: [localhost] local: sleep 5
2016-02-11 16:39:55:975730: [root@10.87.26.199] out: [localhost] local: sudo mkdir -p /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:359357: [root@10.87.26.199] out: [localhost] local: sudo chown -R cassandra: /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:359426: [root@10.87.26.199] out: [localhost] local: df -Pk /var/lib/cassandra/data/ContrailAnalyticsCql | grep % | awk '{print $2}'
2016-02-11 16:39:58:359472: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:359519: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/JVM_OPTS="\$JVM_OPTS -Xss.*"/JVM_OPTS="\$JVM_OPTS -Xss512k"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:367012: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:374420: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:381768: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:385047: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:392408: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:399770: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:403081: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:410593: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/MaxTenuringThreshold=.*"/MaxTenuringThreshold=30"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:417949: [root@10.87.26.199] out: [localhost] local: chown -R cassandra: /var/lib/cassandra/
2016-02-11 16:39:58:425314: [root@10.87.26.199] out: [localhost] local: chown -R cassandra: /var/log/cassandra/
2016-02-11 16:39:58:428823: [root@10.87.26.199] out: [localhost] local: service cassandra start;sleep 5
2016-02-11 16:39:58:428881: [root@10.87.26.198] out: [localhost] local: sudo mkdir -p /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:536029: [root@10.87.26.198] out: [localhost] local: sudo chown -R cassandra: /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:536090: [root@10.87.26.198] out: [localhost] local: df -Pk /var/lib/cassandra/data/ContrailAnalyticsCql | grep % | awk '{print $2}'
2016-02-11 16:39:58:536138: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536185: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/JVM_OPTS="\$JVM_OPTS -Xss.*"/JVM_OPTS="\$JVM_OPTS -Xss512k"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536233: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536279: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536323: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:543908: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:547294: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:554775: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:562272: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:565538: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/MaxTenuringThreshold=.*"/MaxTenuringThreshold=30"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:572846: [root@10.87.26.198] out: [localhost] local: chown -R cassandra: /var/lib/cassandra/
2016-02-11 16:39:58:580303: [root@10.87.26.198] out: [localhost] local: chown -R cassandra: /var/log/cassandra/
2016-02-11 16:39:58:581592: [root@10.87.26.198] out: [localhost] local: service cassandra start;sleep 5
2016-02-11 16:39:58:584923: [root@10.87.26.197] out: [localhost] local: sleep 5
2016-02-11 16:40:01:482948: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:04:035989: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:04:155407: [root@10.87.26.197] out: [localhost] local: sleep 5
2016-02-11 16:40:06:953139: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:09:543044: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:10:278420: [root@10.87.26.197] out: Upgrading database sstables...
2016-02-11 16:40:12:460613: [root@10.87.26.197] out: [localhost] local: nodetool upgradesstables
2016-02-11 16:40:12:460696: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:15:513492: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:15:785901: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:20:984444: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:21:224605: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:26:423437: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:26:695577: [root@10.87.26.199] out: Upgrading database sstables...
2016-02-11 16:40:31:931265: [root@10.87.26.199] out: [localhost] local: nodetool upgradesstables
2016-02-11 16:40:31:931371: [root@10.87.26.198] out: Upgrading database sstables...
2016-02-11 16:40:32:165609: [root@10.87.26.198] out: [localhost] local: nodetool upgradesstables
2016-02-11 16:40:32:165766:

  >>>>>>>>>> this takes long time.. ~20 mins

 [root@10.87.26.197] out: [localhost] local: service cassandra stop
2016-02-11 17:04:55:303319: [root@10.87.26.197] out: [localhost] local: DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install cassandra>=2.1.9
2016-02-11 17:04:57:838124: [root@10.87.26.197] out:
2016-02-11 17:05:00:272621: [root@10.87.26.197] out: Configuration file '/etc/cassandra/cassandra-env.sh'
2016-02-11 17:05:00:272732: [root@10.87.26.197] out: ==> Modified (by you or by a

root@oblocknode04:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0-2712 2712
contrail-config 3.0-2712 2712
contrail-control 3.0-2712 2712
contrail-dns 3.0-2712 2712
contrail-docs 3.0-2712 2712
contrail-f5 3.0-2712 2712
contrail-fabric-utils 3.0-2712 2712
contrail-install-packages 3.0-2712~vcenter 2712
contrail-install-vcenter-plugin 3.0-2712 2712
contrail-lib 3.0-2712 2712
contrail-nodemgr 3.0-2712 2712
contrail-openstack-analytics 3.0-2712 2712
contrail-openstack-control 3.0-2712 2712
contrail-openstack-database 3.0-2712 2712
contrail-setup 3.0-2712 2712
contrail-utils 3.0-2712 2712
contrail-vmware-config 3.0-2712 2712
ifmap-python-client 0.1-2 2712
ifmap-server 0.3.2-1contrail2 2712
python-contrail 3.0-2712 2712
root@oblocknode04:~#

[Sarath (nsarath)]

-bash-4.1$
-bash-4.1$ hostname
centos-build02
-bash-4.1$
-bash-4.1$ pwd
/users/nsarath/PR/PR-1544810
-bash-4.1$
-bash-4.1$ ls -l
total 923344
-rwxrwxrwx 1 nsarath test 941793280 Feb 11 17:47 Ctrl-A-log.tar*
-bash-4.1$

[Raj Reddy (rajreddy)]

It's documented in the upgrade procedure. The users have a choice to drop analytics keyspace before they initiate the upgrade so they can avoid 'long' upgrade times.