vcenter upgrade from 2.21 to 3.0-2712 failing with fatal errors neutron

[Sarath]

This is Vcenter-only-mode and upgrade fails with fatal errors of neutron

Please find below the details,

::
::
2016-02-11 16:39:55:109008: [root@10.87.26.198] out: Configuration file '/etc/cassandra/cassandra-env.sh'
2016-02-11 16:39:55:109068: [root@10.87.26.198] out: ==> Modified (by you or by a script) since installation.
2016-02-11 16:39:55:109240: [root@10.87.26.198] out: ==> Package distributor has shipped an updated version.
2016-02-11 16:39:55:109386: [root@10.87.26.198] out: ==> Using new file as you requested.
2016-02-11 16:39:55:109444: [root@10.87.26.198] out: Installing new version of config file /etc/cassandra/cassandra-env.sh ...
2016-02-11 16:39:55:109494: [root@10.87.26.198] out: Installing new version of config file /etc/cassandra/commitlog_archiving.properties ...
2016-02-11 16:39:55:109773: [root@10.87.26.198] out:
2016-02-11 16:39:55:113064: [root@10.87.26.198] out: Configuration file '/etc/cassandra/cassandra.yaml'
2016-02-11 16:39:55:113124: [root@10.87.26.198] out: ==> Modified (by you or by a script) since installation.
2016-02-11 16:39:55:113174: [root@10.87.26.198] out: ==> Package distributor has shipped an updated version.
2016-02-11 16:39:55:113221: [root@10.87.26.198] out: ==> Using new file as you requested.
2016-02-11 16:39:55:113268: [root@10.87.26.198] out: Installing new version of config file /etc/cassandra/cassandra.yaml ...
2016-02-11 16:39:55:113315: [root@10.87.26.198] out:
2016-02-11 16:39:55:120861: [root@10.87.26.198] out: Configuration file '/etc/init.d/cassandra'
2016-02-11 16:39:55:120937: [root@10.87.26.198] out: ==> Deleted (by you or by a script) since installation.
2016-02-11 16:39:55:120988: [root@10.87.26.198] out: ==> Package distributor has shipped an updated version.
2016-02-11 16:39:55:121036: [root@10.87.26.198] out: ==> Using new file as you requested.
2016-02-11 16:39:55:121118: [root@10.87.26.198] out: Installing new version of config file /etc/init.d/cassandra ...
2016-02-11 16:39:55:121168: [root@10.87.26.198] out: Installing new version of config file /etc/sysctl.d/cassandra.conf ...
2016-02-11 16:39:55:122591: [root@10.87.26.198] out: Installing new version of config file /etc/security/limits.d/cassandra.conf ...
2016-02-11 16:39:55:125853: [root@10.87.26.198] out: vm.max_map_count = 1048575
2016-02-11 16:39:55:133221: [root@10.87.26.198] out: net.ipv4.tcp_keepalive_time = 300
2016-02-11 16:39:55:133281: [root@10.87.26.199] out: [localhost] local: service cassandra stop
2016-02-11 16:39:55:192118: [root@10.87.26.198] out: Processing triggers for python-support (1.0.15) ...
2016-02-11 16:39:55:264977: [root@10.87.26.198] out: Processing triggers for ureadahead (0.100.0-16) ...
2016-02-11 16:39:55:328654: [root@10.87.26.198] out: [localhost] local: service cassandra stop
2016-02-11 16:39:55:432151: [root@10.87.26.197] out: [localhost] local: sleep 5
2016-02-11 16:39:55:975730: [root@10.87.26.199] out: [localhost] local: sudo mkdir -p /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:359357: [root@10.87.26.199] out: [localhost] local: sudo chown -R cassandra: /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:359426: [root@10.87.26.199] out: [localhost] local: df -Pk /var/lib/cassandra/data/ContrailAnalyticsCql | grep % | awk '{print $2}'
2016-02-11 16:39:58:359472: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:359519: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/JVM_OPTS="\$JVM_OPTS -Xss.*"/JVM_OPTS="\$JVM_OPTS -Xss512k"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:367012: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:374420: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:381768: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:385047: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:392408: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:399770: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:403081: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:410593: [root@10.87.26.199] out: [localhost] local: sudo sed -i 's/MaxTenuringThreshold=.*"/MaxTenuringThreshold=30"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:417949: [root@10.87.26.199] out: [localhost] local: chown -R cassandra: /var/lib/cassandra/
2016-02-11 16:39:58:425314: [root@10.87.26.199] out: [localhost] local: chown -R cassandra: /var/log/cassandra/
2016-02-11 16:39:58:428823: [root@10.87.26.199] out: [localhost] local: service cassandra start;sleep 5
2016-02-11 16:39:58:428881: [root@10.87.26.198] out: [localhost] local: sudo mkdir -p /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:536029: [root@10.87.26.198] out: [localhost] local: sudo chown -R cassandra: /var/lib/cassandra/data/ContrailAnalyticsCql
2016-02-11 16:39:58:536090: [root@10.87.26.198] out: [localhost] local: df -Pk /var/lib/cassandra/data/ContrailAnalyticsCql | grep % | awk '{print $2}'
2016-02-11 16:39:58:536138: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536185: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/JVM_OPTS="\$JVM_OPTS -Xss.*"/JVM_OPTS="\$JVM_OPTS -Xss512k"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536233: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536279: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:536323: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:543908: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:547294: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:554775: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:562272: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:565538: [root@10.87.26.198] out: [localhost] local: sudo sed -i 's/MaxTenuringThreshold=.*"/MaxTenuringThreshold=30"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 16:39:58:572846: [root@10.87.26.198] out: [localhost] local: chown -R cassandra: /var/lib/cassandra/
2016-02-11 16:39:58:580303: [root@10.87.26.198] out: [localhost] local: chown -R cassandra: /var/log/cassandra/
2016-02-11 16:39:58:581592: [root@10.87.26.198] out: [localhost] local: service cassandra start;sleep 5
2016-02-11 16:39:58:584923: [root@10.87.26.197] out: [localhost] local: sleep 5
2016-02-11 16:40:01:482948: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:04:035989: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:04:155407: [root@10.87.26.197] out: [localhost] local: sleep 5
2016-02-11 16:40:06:953139: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:09:543044: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:10:278420: [root@10.87.26.197] out: Upgrading database sstables...
2016-02-11 16:40:12:460613: [root@10.87.26.197] out: [localhost] local: nodetool upgradesstables
2016-02-11 16:40:12:460696: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:15:513492: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:15:785901: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:20:984444: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:21:224605: [root@10.87.26.199] out: [localhost] local: sleep 5
2016-02-11 16:40:26:423437: [root@10.87.26.198] out: [localhost] local: sleep 5
2016-02-11 16:40:26:695577: [root@10.87.26.199] out: Upgrading database sstables...
2016-02-11 16:40:31:931265: [root@10.87.26.199] out: [localhost] local: nodetool upgradesstables
2016-02-11 16:40:31:931371: [root@10.87.26.198] out: Upgrading database sstables...
2016-02-11 16:40:32:165609: [root@10.87.26.198] out: [localhost] local: nodetool upgradesstables
2016-02-11 16:40:32:165766:

 << issue-1 here >>

 [root@10.87.26.197] out: [localhost] local: service cassandra stop
2016-02-11 17:04:55:303319: [root@10.87.26.197] out: [localhost] local: DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install cassandra>=2.1.9
2016-02-11 17:04:57:838124: [root@10.87.26.197] out:
2016-02-11 17:05:00:272621: [root@10.87.26.197] out: Configuration file '/etc/cassandra/cassandra-env.sh'
2016-02-11 17:05:00:272732: [root@10.87.26.197] out: ==> Modified (by you or by a script) since installation.
2016-02-11 17:05:00:272830: [root@10.87.26.197] out: ==> Package distributor has shipped an updated version.
2016-02-11 17:05:00:272899: [root@10.87.26.197] out: ==> Using new file as you requested.
2016-02-11 17:05:00:272965: [root@10.87.26.197] out:
2016-02-11 17:05:00:273031: [root@10.87.26.197] out: Configuration file '/etc/cassandra/cassandra.yaml'
2016-02-11 17:05:00:273112: [root@10.87.26.197] out: ==> Modified (by you or by a script) since installation.
2016-02-11 17:05:00:273176: [root@10.87.26.197] out: ==> Package distributor has shipped an updated version.
2016-02-11 17:05:00:273238: [root@10.87.26.197] out: ==> Using new file as you requested.
2016-02-11 17:05:00:273317: [root@10.87.26.197] out: [localhost] local: service cassandra stop
2016-02-11 17:05:01:074736: [root@10.87.26.197] out: [localhost] local: df -Pk /var/lib/cassandra/data/ContrailAnalyticsCql | grep % | awk '{print $2}'
2016-02-11 17:05:01:175067: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDetails"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:175138: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/JVM_OPTS="\$JVM_OPTS -Xss.*"/JVM_OPTS="\$JVM_OPTS -Xss512k"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:175190: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCDateStamps"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:178617: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/JVM_OPTS="\$JVM_OPTS -XX:+PrintHeapAtGC"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:186090: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/JVM_OPTS="\$JVM_OPTS -XX:+PrintTenuringDistribution"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:189421: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/JVM_OPTS="\$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:196877: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/JVM_OPTS="\$JVM_OPTS -XX:+PrintPromotionFailure"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:204598: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/JVM_OPTS="\$JVM_OPTS -XX:PrintFLSStatistics=1"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:207936: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/# JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/JVM_OPTS="\$JVM_OPTS -Xloggc:\/var\/log\/cassandra\/gc-`date +%s`.log"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:215401: [root@10.87.26.197] out: [localhost] local: sudo sed -i 's/MaxTenuringThreshold=.*"/MaxTenuringThreshold=30"/g' /etc/cassandra/cassandra-env.sh
2016-02-11 17:05:01:222860: [root@10.87.26.197] out: [localhost] local: dpkg -s contrail-install-packages | grep Version: | cut -d' ' -f2 | cut -d'-' -f2
2016-02-11 17:05:01:230256: [root@10.87.26.197] out: Backing up /etc/contrail at: /var/tmp/contrail-3.0-2712-upgradesave/etc/contrail
2016-02-11 17:05:01:293973: [root@10.87.26.197] out: [localhost] local: cp -rfp /etc/contrail /var/tmp/contrail-3.0-2712-upgradesave/etc/contrail
2016-02-11 17:05:01:294027: [root@10.87.26.197] out: [localhost] local: DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-database

::
::

2016-02-11 17:06:45:139296: [root@10.87.26.199] sudo: upgrade-vnc-config --self_ip 172.16.80.4 --collector_ip 172.16.80.4 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0
[root@10.87.26.197] out: The following packages were automatically installed and are no longer required:
2016-02-11 17:06:45:169103: [root@10.87.26.197] out: jsvc libcommons-daemon-java libjna-java
2016-02-11 17:06:45:169191: [root@10.87.26.197] out: Use 'apt-get autoremove' to remove them.
2016-02-11 17:06:45:169274: [root@10.87.26.197] out: 0 upgraded, 0 newly installed, 0 to remove and 157 not upgraded.
2016-02-11 17:06:45:184840: [root@10.87.26.197] out:
2016-02-11 17:06:45:185005:
2016-02-11 17:06:45:193695: [root@10.87.26.197] sudo: sudo ls /etc/contrail/service.token
2016-02-11 17:06:45:194302: [root@10.87.26.197] out: /etc/contrail/service.token
2016-02-11 17:06:45:243837: [root@10.87.26.197] out:
2016-02-11 17:06:45:244005:
2016-02-11 17:06:45:248877: [root@10.87.26.197] sudo: sudo cat /etc/contrail/service.token
2016-02-11 17:06:45:125262: [root@10.87.26.198] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-02-11 17:06:45:258587: [root@10.87.26.198] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-02-11 17:06:45:258659: [root@10.87.26.198] out: [localhost] local: dpkg -s contrail-install-packages | grep Version: | cut -d' ' -f2 | cut -d'-' -f2
2016-02-11 17:06:45:258706: [root@10.87.26.198] out: Already the config dir /etc/contrail is backed up at /var/tmp/contrail-3.0-2712-upgradesave/etc/contrail.
2016-02-11 17:06:45:258752: [root@10.87.26.198] out: Backing up /etc/ifmap-server at: /var/tmp/contrail-3.0-2712-upgradesave/etc/ifmap-server
2016-02-11 17:06:45:258797: [root@10.87.26.198] out: [localhost] local: cp -rfp /etc/ifmap-server /var/tmp/contrail-3.0-2712-upgradesave/etc/ifmap-server
2016-02-11 17:06:45:258842: [root@10.87.26.198] out: Backing up /etc/neutron at: /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron
2016-02-11 17:06:45:262347: [root@10.87.26.198] out: [localhost] local: cp -rfp /etc/neutron /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron
2016-02-11 17:06:45:262415: [root@10.87.26.198] out: cp: cannot stat ‘/etc/neutron’: No such file or directory
2016-02-11 17:06:45:262459: 2016-02-11 17:06:45:139401: [root@10.87.26.199] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-02-11 17:06:45:272983: [root@10.87.26.199] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-02-11 17:06:45:273073: [root@10.87.26.199] out: [localhost] local: dpkg -s contrail-install-packages | grep Version: | cut -d' ' -f2 | cut -d'-' -f2
2016-02-11 17:06:45:273142: [root@10.87.26.199] out: Already the config dir /etc/contrail is backed up at /var/tmp/contrail-3.0-2712-upgradesave/etc/contrail.
2016-02-11 17:06:45:273208: [root@10.87.26.199] out: Backing up /etc/ifmap-server at: /var/tmp/contrail-3.0-2712-upgradesave/etc/ifmap-server
2016-02-11 17:06:45:273287: [root@10.87.26.199] out: [localhost] local: cp -rfp /etc/ifmap-server /var/tmp/contrail-3.0-2712-upgradesave/etc/ifmap-server
2016-02-11 17:06:45:273349: [root@10.87.26.199] out: Backing up /etc/neutron at: /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron
2016-02-11 17:06:45:273410: [root@10.87.26.199] out: [localhost] local: cp -rfp /etc/neutron /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron
2016-02-11 17:06:45:273472: [root@10.87.26.199] out: cp: cannot stat ‘/etc/neutron’: No such file or directory
2016-02-11 17:06:45:273534: [root@10.87.26.199] out:
2016-02-11 17:06:45:273595: [root@10.87.26.199] out: Fatal error: local() encountered an error (return code 1) while executing 'cp -rfp /etc/neutron /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron'
2016-02-11 17:06:45:273655: [root@10.87.26.199] out:
2016-02-11 17:06:45:273717: [root@10.87.26.199] out: Aborting.
2016-02-11 17:06:45:273775: [root@10.87.26.199] out:
2016-02-11 17:06:45:277585: [root@10.87.26.198] out:
2016-02-11 17:06:45:277947: [root@10.87.26.198] out: Fatal error: local() encountered an error (return code 1) while executing 'cp -rfp /etc/neutron /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron'
2016-02-11 17:06:45:278006: [root@10.87.26.198] out:
2016-02-11 17:06:45:278056: [root@10.87.26.198] out: Aborting.
2016-02-11 17:06:45:278100: [root@10.87.26.198] out:
2016-02-11 17:06:45:278314:

2016-02-11 17:06:45:279833: Fatal error: sudo() received nonzero return code 1 while executing!
2016-02-11 17:06:45:279833:
2016-02-11 17:06:45:279833: Requested: upgrade-vnc-config --self_ip 172.16.80.3 --collector_ip 172.16.80.3 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0
2016-02-11 17:06:45:279833: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "upgrade-vnc-config --self_ip 172.16.80.3 --collector_ip 172.16.80.3 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0"
2016-02-11 17:06:45:279833:
2016-02-11 17:06:45:279887: Aborting.
2016-02-11 17:06:45:279887: 2016-02-11 17:06:45:279734:

2016-02-11 17:06:45:284711: Fatal error: sudo() received nonzero return code 1 while executing!
2016-02-11 17:06:45:284711:
2016-02-11 17:06:45:284711: Requested: upgrade-vnc-config --self_ip 172.16.80.4 --collector_ip 172.16.80.4 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0
2016-02-11 17:06:45:284711: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "upgrade-vnc-config --self_ip 172.16.80.4 --collector_ip 172.16.80.4 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0"
2016-02-11 17:06:45:284711:
2016-02-11 17:06:45:284775: Aborting.
2016-02-11 17:06:45:284775: 2016-02-11 17:06:45:284597: 2016-02-11 17:06:45:249003: [root@10.87.26.197] out: 6ec903f8d5b019919ae4
2016-02-11 17:06:45:302736: [root@10.87.26.197] sudo: python -c 'from platform import linux_distribution; print linux_distribution()'
2016-02-11 17:06:45:303004: [root@10.87.26.197] out: ('Ubuntu', '14.04', 'trusty')
2016-02-11 17:06:45:368249: [root@10.87.26.197] out:
2016-02-11 17:06:45:368418:
2016-02-11 17:06:45:368569: Dist is ubuntu
2016-02-11 17:06:45:368682: [root@10.87.26.197] sudo: dpkg -s contrail-install-packages | grep Version: | cut -d' ' -f2 | cut -d'-' -f1
2016-02-11 17:06:45:368750: [root@10.87.26.197] out: 3.0
2016-02-11 17:06:45:433701: [root@10.87.26.197] out:
2016-02-11 17:06:45:433875:
2016-02-11 17:06:45:434047: [root@10.87.26.197] sudo: upgrade-vnc-config --self_ip 172.16.80.2 --collector_ip 172.16.80.2 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0
2016-02-11 17:06:45:434161: [root@10.87.26.197] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-02-11 17:06:45:567946: [root@10.87.26.197] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-02-11 17:06:45:568004: [root@10.87.26.197] out: [localhost] local: dpkg -s contrail-install-packages | grep Version: | cut -d' ' -f2 | cut -d'-' -f2
2016-02-11 17:06:45:568049: [root@10.87.26.197] out: Already the config dir /etc/contrail is backed up at /var/tmp/contrail-3.0-2712-upgradesave/etc/contrail.
2016-02-11 17:06:45:568232: [root@10.87.26.197] out: Backing up /etc/ifmap-server at: /var/tmp/contrail-3.0-2712-upgradesave/etc/ifmap-server
2016-02-11 17:06:45:568352: [root@10.87.26.197] out: [localhost] local: cp -rfp /etc/ifmap-server /var/tmp/contrail-3.0-2712-upgradesave/etc/ifmap-server
2016-02-11 17:06:45:568447: [root@10.87.26.197] out: Backing up /etc/neutron at: /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron
2016-02-11 17:06:45:584098: [root@10.87.26.197] out: [localhost] local: cp -rfp /etc/neutron /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron
2016-02-11 17:06:45:584182: [root@10.87.26.197] out: cp: cannot stat ‘/etc/neutron’: No such file or directory
2016-02-11 17:06:45:584248: [root@10.87.26.197] out:
2016-02-11 17:06:45:584312: [root@10.87.26.197] out: Fatal error: local() encountered an error (return code 1) while executing 'cp -rfp /etc/neutron /var/tmp/contrail-3.0-2712-upgradesave/etc/neutron'
2016-02-11 17:06:45:584373: [root@10.87.26.197] out:
2016-02-11 17:06:45:584441: [root@10.87.26.197] out: Aborting.
2016-02-11 17:06:45:584500: [root@10.87.26.197] out:
2016-02-11 17:06:45:592088:

2016-02-11 17:06:45:599879: Fatal error: sudo() received nonzero return code 1 while executing!
2016-02-11 17:06:45:599879:
2016-02-11 17:06:45:599879: Requested: upgrade-vnc-config --self_ip 172.16.80.2 --collector_ip 172.16.80.2 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0
2016-02-11 17:06:45:599879: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "upgrade-vnc-config --self_ip 172.16.80.2 --collector_ip 172.16.80.2 --multi_tenancy --cassandra_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 6ec903f8d5b019919ae4 --amqp_ip_list 172.16.80.2 172.16.80.3 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --manage_neutron no -P contrail-openstack-config -F 2.21 -T 3.0"
2016-02-11 17:06:45:599879:
2016-02-11 17:06:45:599929: Aborting.
2016-02-11 17:06:45:599929: 2016-02-11 17:06:45:599787:
2016-02-11 17:06:45:619228: Fatal error: One or more hosts failed while executing task 'upgrade_config'
2016-02-11 17:06:45:619228:
2016-02-11 17:06:45:619306: Aborting.
2016-02-11 17:06:45:619306: Disconnecting from 10.87.26.197... done.
2016-02-11 17:06:45:671759: Disconnecting from 10.87.26.198... done.
2016-02-11 17:06:45:770175: Disconnecting from 10.87.26.199... done.
2016-02-11 17:06:45:782680: root@oblocknode02:/opt/contrail/utils#
root@oblocknode02:/opt/contrail/utils#

root@oblocknode02:~#
root@oblocknode02:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control initializing (IFMap:IFMapServer, Collector, Discovery:Collector, Discovery:IfmapServer, Discovery:xmpp-server connection down)
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: inactive
unix:///tmp/supervisord_analytics.sockno

== Contrail Config ==
supervisor-config: inactive
unix:///tmp/supervisord_config.sockno

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: inactive
unix:///tmp/supervisord_support_service.sockno

root@oblocknode02:~#
root@oblocknode02:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21-103 2712
contrail-config 2.21-103 2712
contrail-control 2.21-103 2712
contrail-dns 2.21-103 2712
contrail-f5 2.21-103 2712
contrail-fabric-utils 3.0-2712 2712
contrail-install-packages 3.0-2712~vcenter 2712
contrail-install-vcenter-plugin 2.21-103 2712
contrail-lib 2.21-103 2712
contrail-nodemgr 3.0-2712 2712
contrail-openstack-analytics 2.21-103 2712
contrail-openstack-control 2.21-103 2712
contrail-openstack-database 3.0-2712 2712
contrail-openstack-webui 2.21-103 2712
contrail-setup 3.0-2712 2712
contrail-utils 3.0-2712 2712
contrail-vmware-config 2.21-103 2712
contrail-vmware-utils 3.0-2712 2712
contrail-web-controller 2.21-103 2712
contrail-web-core 2.21-103 2712
ifmap-python-client 0.1-2 2712
ifmap-server 0.3.2-1contrail1 2712
python-contrail 3.0-2712 2712
root@oblocknode02:~#
root@oblocknode02:~#
root@oblocknode02:~#

[Sarath (nsarath)]

-bash-4.1$
-bash-4.1$ hostname
centos-build02
-bash-4.1$
-bash-4.1$ pwd
/users/nsarath/PR/PR-1544810
-bash-4.1$
-bash-4.1$ ls -l
total 923344
-rwxrwxrwx 1 nsarath test 941793280 Feb 11 17:47 Ctrl-A-log.tar*
-bash-4.1$

[Raj Reddy (rajreddy)]

the 'nodetool upgradesstables' will take time depending on the amount of data present in the system.
we will document and/or give option to not upgrade analytics data to alleviate this problem.

The other issue needs to be looked at by config/openstack team.

[amudhar (amudha)]

neutron errors are fixed by https://review.opencontrail.org/#/c/17190/.
Please try with the latest build (>2713) and let know if the issue still exists.

[amudhar (amudha)]

vcenter specific upgrade related neutron errors are fixed by https://review.opencontrail.org/#/c/17190/

[Sarath (nsarath)]

Verified build#2723