Vcenter-only: fab upgrade_contrail from build #6 to #13 fails at upgrade_config

[Sarath]

Below given the upgrade procedure,
dpkg -i /root/contrail-install-packages_3.1.0.0-13~vcenter_all.deb
cd /opt/contrail/contrail_packages
./setup.sh
fab upgrade_contrail:3.1,/root/contrail-install-packages_3.1.0.0-13~vcenter_all.deb

2016-08-02 11:16:27:871602: [root@10.87.26.199] out:
2016-08-02 11:16:27:871744:
2016-08-02 11:16:27:765022: [root@10.87.26.199] sudo: upgrade-vnc-config --self_ip 172.16.80.4 --collector_ip 172.16.80.4 --cassandra_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --control_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --quantum_port 9697 --nworkers 1 --service_token 3b9600ee535d599eadd4 --amqp_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_port 5672 --orchestrator vcenter --seed_list 172.16.80.2,172.16.80.13 --data_dir /var/lib/cassandra/data --manage_neutron no -P contrail-vmware-config contrail-database-common -F 3.1 -T 3.1.0.0
2016-08-02 11:16:27:765133: [root@10.87.26.199] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-08-02 11:16:27:898189: [root@10.87.26.199] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-08-02 11:16:27:898330: [root@10.87.26.199] out: [localhost] local: dpkg -s contrail-install-packages | grep Version: | cut -d' ' -f2 | cut -d'-' -f2
2016-08-02 11:16:27:898402: [root@10.87.26.199] out: Already the config dir /etc/contrail is backed up at /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail.
2016-08-02 11:16:27:905858: [root@10.87.26.199] out: Backing up /etc/ifmap-server at: /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/ifmap-server
2016-08-02 11:16:27:905937: [root@10.87.26.199] out: [localhost] local: cp -rfp /etc/ifmap-server /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/ifmap-server
2016-08-02 11:16:27:906009:
2016-08-02 11:16:27:871868: [root@10.87.26.199] sudo: DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes --allow-unauthenticated install libxml-commons-external-java
2016-08-02 11:16:27:872026: [root@10.87.26.199] out:
2016-08-02 11:16:27:936669: [root@10.87.26.199] out: Reading package lists... 0%
2016-08-02 11:16:27:936731: [root@10.87.26.199] out:
2016-08-02 11:16:27:968236: [root@10.87.26.199] out: Reading package lists... 0%
2016-08-02 11:16:27:968310: [root@10.87.26.199] out:
2016-08-02 11:16:27:968362: [root@10.87.26.199] out: Reading package lists... 15%
2016-08-02 11:16:27:968410: [root@10.87.26.199] out: Backing up /etc/init.d/contrail-api at: /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/init.d/contrail-api
2016-08-02 11:16:27:969666: [root@10.87.26.199] out: Backing up /etc/init.d/contrail-discovery at: /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/init.d/contrail-discovery
2016-08-02 11:16:27:969747: [root@10.87.26.199] out: Backing up /etc/sudoers.d/contrail_sudoers at: /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/sudoers.d/contrail_sudoers
2016-08-02 11:16:27:969819: [root@10.87.26.199] out: [localhost] local: DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-vmware-config contrail-database-common
2016-08-02 11:16:27:969891: [root@10.87.26.199] out: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
2016-08-02 11:16:27:969963: [root@10.87.26.199] out: E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
2016-08-02 11:16:27:970032: [root@10.87.26.199] out:
2016-08-02 11:16:27:970101: [root@10.87.26.199] out: Fatal error: local() encountered an error (return code 100) while executing 'DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-vmware-config contrail-database-common'
2016-08-02 11:16:27:970170: [root@10.87.26.199] out:
2016-08-02 11:16:27:970241: [root@10.87.26.199] out: Aborting.
2016-08-02 11:16:27:970306: [root@10.87.26.199] out:

::
::

2016-08-02 11:16:36:247336: [root@10.87.26.197] out: unix:///var/run/supervisord_support_service.sock no such file
2016-08-02 11:16:36:310933: [root@10.87.26.197] out: mv: cannot stat ‘/etc/init/rabbitmq-server.conf’: No such file or directory
2016-08-02 11:16:36:310996: [root@10.87.26.197] out: Postinst for contrail-vmware-config done
2016-08-02 11:16:36:311050: [root@10.87.26.197] out: Processing triggers for libc-bin (2.19-0ubuntu6.5) ...
2016-08-02 11:16:36:311099: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/vnc_api_lib.ini to: /etc/contrail/vnc_api_lib.ini
2016-08-02 11:16:37:243709: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/contrail-svc-monitor.conf to: /etc/contrail/contrail-svc-monitor.conf
2016-08-02 11:16:37:243824: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/contrail-schema.conf to: /etc/contrail/contrail-schema.conf
2016-08-02 11:16:37:243926: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/contrail-api.conf to: /etc/contrail/contrail-api.conf
2016-08-02 11:16:37:244022: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/contrail-discovery.conf to: /etc/contrail/contrail-discovery.conf
2016-08-02 11:16:37:244118: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/supervisord_config_files/contrail-api.ini to: /etc/contrail/supervisord_config_files/contrail-api.ini
2016-08-02 11:16:37:244214: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/supervisord_config_files/contrail-discovery.ini to: /etc/contrail/supervisord_config_files/contrail-discovery.ini
2016-08-02 11:16:37:244329: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/sudoers.d/contrail_sudoers to: /etc/sudoers.d/contrail_sudoers
2016-08-02 11:16:37:244496: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/init.d/contrail-api to: /etc/init.d/contrail-api
2016-08-02 11:16:37:244764: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/init.d/contrail-discovery to: /etc/init.d/contrail-discovery
2016-08-02 11:16:37:244820: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/ifmap-server to: /etc/ifmap-server
2016-08-02 11:16:37:244871: [root@10.87.26.197] out: [localhost] local: cp -rfp /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/ifmap-server/* /etc/ifmap-server
2016-08-02 11:16:37:244921: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/contrail-config-nodemgr.conf to: /etc/contrail/contrail-config-nodemgr.conf
2016-08-02 11:16:37:245541: [root@10.87.26.197] out: Restoring /var/tmp/contrail-3.1.0.0-13-upgradesave/etc/contrail/contrail-device-manager.conf to: /etc/contrail/contrail-device-manager.conf
2016-08-02 11:16:37:245602: [root@10.87.26.197] out:
2016-08-02 11:16:37:253091:
2016-08-02 11:16:37:262318:
2016-08-02 11:16:37:277959: Fatal error: One or more hosts failed while executing task 'upgrade_config'
2016-08-02 11:16:37:277959:
2016-08-02 11:16:37:278030: Aborting.

[Sarath (nsarath)]

Please find below the Fab-logs / logs,

nsarath@ubuntu-build04:/auto/cores/1609095$ pwd
/auto/cores/1609095
nsarath@ubuntu-build04:/auto/cores/1609095$ ls -l
total 2048832
-rwxrwxrwx 1 nsarath test 5273600 Aug 2 12:18 Ctrl-A-FabLogs.tar
-rwxrwxrwx 1 nsarath test 761231360 Aug 2 12:19 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 602695680 Aug 2 12:19 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 574781440 Aug 2 12:18 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 29102080 Aug 2 12:18 Vrtr-0-log.tar
-rwxrwxrwx 1 nsarath test 33402880 Aug 2 12:18 Vrtr-1-log.tar
-rwxrwxrwx 1 nsarath test 27822080 Aug 2 12:18 Vrtr-3-log.tar
-rwxrwxrwx 1 nsarath test 27904000 Aug 2 12:18 Vrtr-5-log.tar
-rwxrwxrwx 1 nsarath test 27504640 Aug 2 12:18 Vrtr-7-log.tar
nsarath@ubuntu-build04:/auto/cores/1609095$

[Sarath (nsarath)]

Workaround : fresh-install works fine

[Sarath (nsarath)]

Workaround: during fab upgrade_contrail, may encounter the failure due to "Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)" and please remove the file /var/lib/dpkg/lock on all controller nodes and re-issue the command fab upgrade_contrail