Per Inputs from Jeba & Sachin, This multi-DC provisioning to work with single nova-compute per DC On 3.1.1#50 and also tried 3.2#19 with mitaka This is tracking bug for the failure Below the upgrade procedure used, dpkg -i /root/contrail-install-packages_3.1.0.0-50~mitaka_all.deb cd /opt/contrail/contrail_packages ./setup.sh cd /opt/contrail/utils/ << /opt/contrail/utils/fabfile/testbeds/testbed.py fab prov_esxi fab install_pkg_all:/root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb ## Failures seen during install_pkg Also copied the fab logs and TB file to bugs root@a6s32:/opt/contrail/utils# fab install_pkg_all:/root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb [root@10.84.13.32] Executing task 'install_pkg_all' 2017-01-11 15:31:26:237923: [root@10.84.13.33] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238020: [root@10.84.13.38] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238083: [root@10.84.13.2] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238145: [root@10.84.20.42] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238203: [root@10.84.13.222] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238273: [root@10.84.13.223] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238329: [root@10.84.20.240] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238384: [root@10.84.20.242] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238440: [root@10.84.20.40] Executing task 'install_pkg_all' 2017-01-11 15:31:26:238494: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' 2017-01-11 15:31:26:243718: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Package:\s+\K.*' 2017-01-11 15:31:26:243054: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:250741: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:252372: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:259293: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:254904: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:248864: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:256033: [root@10.84.20.242] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:253977: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:266904: [root@10.84.13.32] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:252024: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:264564: [root@10.84.20.40] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:257678: [localhost] local: dpkg -I /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb | grep -Po '^\s+Version:\s+\K.*' 2017-01-11 15:31:26:273899: [root@10.84.20.240] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:279213: [root@10.84.13.2] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:275495: [root@10.84.13.33] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:277043: [root@10.84.20.42] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:278284: [root@10.84.13.222] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:284911: [root@10.84.13.223] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:288773: [root@10.84.13.38] sudo: python -c 'from platform import linux_distribution; print linux_distribution()' 2017-01-11 15:31:26:282988: [root@10.84.13.32] out: ('Ubuntu', '14.04', 'trusty') 2017-01-11 15:31:26:492205: [root@10.84.13.32] out: 2017-01-11 15:31:26:492544: 2017-01-11 15:31:26:492721: [root@10.84.13.32] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages 2017-01-11 15:31:26:303432: [root@10.84.13.38] out: ('Ubuntu', '14.04', 'trusty') 2017-01-11 15:31:26:512088: [root@10.84.13.38] out: 2017-01-11 15:31:26:512436: 2017-01-11 15:31:26:543701: [root@10.84.13.38] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages 2017-01-11 15:31:26:492927: [root@10.84.13.32] out: 3.1.1.0-50~mitaka 2017-01-11 15:31:26:557661: [root@10.84.13.32] out: 2017-01-11 15:31:26:557871: 2017-01-11 15:31:26:558002: Package (/root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb) already installed in the node (root@10.84.13.32). 2017-01-11 15:31:26:558079: 2017-01-11 15:31:26:287516: [root@10.84.20.40] out: ('Ubuntu', '14.04', 'trusty') 2017-01-11 15:31:26:559773: [root@10.84.20.40] out: 2017-01-11 15:31:26:560171: 2017-01-11 15:31:26:560329: [root@10.84.20.40] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages 2017-01-11 15:31:26:295418: [root@10.84.13.2] out: ('Ubuntu', '14.04', 'trusty') 2017-01-11 15:31:26:561714: [root@10.84.13.2] out: 2017-01-11 15:31:26:562032: 2017-01-11 15:31:26:562148: [root@10.84.13.2] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages 2017-01-11 15:31:26:560533: [root@10.84.20.40] out: 3.2.0.0-19~mitaka 2017-01-11 15:31:26:642004: [root@10.84.20.40] out: 2017-01-11 15:31:26:642326: 2017-01-11 15:31:26:642494: [root@10.84.20.40] sudo: mkdir -p /tmp/tmpmK2z7R 2017-01-11 15:31:26:295651: [root@10.84.13.33] out: ('Ubuntu', '14.04', 'trusty') 2017-01-11 15:31:26:700372: [root@10.84.13.33] out: 2017-01-11 15:31:26:700929: 2017-01-11 15:31:26:701745: [root@10.84.13.33] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages 2017-01-11 15:31:26:643128: [root@10.84.20.40] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmpmK2z7R/contrail-install-packages_3.1.1.0-50~mitaka_all.deb 2017-01-11 15:31:26:544002: [root@10.84.13.38] out: dpkg-query: no packages found matching contrail-install-packages 2017-01-11 15:31:26:762324: [root@10.84.13.38] out: 2017-01-11 15:31:26:762700: 2017-01-11 15:31:26:763050: Warning: sudo() received nonzero return code 1 while executing 'dpkg-query -W -f='${VERSION}\n' contrail-install-packages'! 2017-01-11 15:31:26:763050: 2017-01-11 15:31:26:763050: 2017-01-11 15:31:26:762907: Package contrail-install-packages not installed. 2017-01-11 15:31:26:763131: [root@10.84.13.38] sudo: mkdir -p /tmp/tmpRx7L7K 2017-01-11 15:31:26:298968: [root@10.84.13.222] out: ('Ubuntu', '14.04', 'trusty') 2017-01-11 15:31:26:764864: [root@10.84.13.222] out: 2017-01-11 15:31:26:765283: 2017-01-11 15:31:26:765479: [root@10.84.13.222] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages 2017-01-11 15:31:26:303249: [root@10.84.13.223] out: ('Ubuntu', '14.04', 'trusty') 2017-01-11 15:31:26:767522: [root@10.84.13.223] out: 2017-01-11 15:31:26:768089: 2017-01-11 15:31:26:768449: [root@10.84.13.223] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages 2017-01-11 15:31:26:562338: [root@10.84.13.2] out: dpkg-query: no packages found matching contrail-install-packages 2017-01-11 15:31:26:811641: [root@10.84.13.2] out: 2017-01-11 15:31:26:811947: 2017-01-11 15:31:26:812324: Warning: sudo() received nonzero return code 1 while executing 'dpkg-query -W -f='${VERSION}\n' contrail-install-packages'! 2017-01-11 15:31:26:812324: 2017-01-11 15:31:26:812324: 2017-01-11 15:31:26:812199: Package contrail-install-packages not installed. 2017-01-11 15:31:26:812401: [root@10.84.13.2] sudo: mkdir -p /tmp/tmplDNqKy 2017-01-11 15:31:26:768830: [root@10.84.13.223] out: dpkg-query: no packages found matching contrail-install-packages 2017-01-11 15:31:26:866260: [root@10.84.13.223] out: 2017-01-11 15:31:26:866499: 2017-01-11 15:31:26:873691: Warning: sudo() received nonzero return code 1 while executing 'dpkg-query -W -f='${VERSION}\n' contrail-install-packages'! 2017-01-11 15:31:26:873691: 2017-01-11 15:31:26:873691: 2017-01-11 15:31:26:873590: Package contrail-install-packages not installed. 2017-01-11 15:31:26:873760: [root@10.84.13.223] sudo: mkdir -p /tmp/tmpuyn1jQ 2017-01-11 15:31:26:765793: [root@10.84.13.222] out: dpkg-query: no packages found matching contrail-install-packages 2017-01-11 15:31:26:884054: [root@10.84.13.222] out: 2017-01-11 15:31:26:884257: 2017-01-11 15:31:26:884468: Warning: sudo() received nonzero return code 1 while executing 'dpkg-query -W -f='${VERSION}\n' contrail-install-packages'! 2017-01-11 15:31:26:884468: 2017-01-11 15:31:26:884468: 2017-01-11 15:31:26:884385: Package contrail-install-packages not installed. 2017-01-11 15:31:26:884533: [root@10.84.13.222] sudo: mkdir -p /tmp/tmpDb8EOQ 2017-01-11 15:31:26:763755: [root@10.84.13.38] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmpRx7L7K/contrail-install-packages_3.1.1.0-50~mitaka_all.deb 2017-01-11 15:31:26:885027: [root@10.84.13.222] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmpDb8EOQ/contrail-install-packages_3.1.1.0-50~mitaka_all.deb 2017-01-11 15:31:26:702038: [root@10.84.13.33] out: dpkg-query: no packages found matching contrail-install-packages 2017-01-11 15:31:26:969035: [root@10.84.13.33] out: 2017-01-11 15:31:26:969337: 2017-01-11 15:31:26:970718: Warning: sudo() received nonzero return code 1 while executing 'dpkg-query -W -f='${VERSION}\n' contrail-install-packages'! 2017-01-11 15:31:26:970718: 2017-01-11 15:31:26:970718: 2017-01-11 15:31:26:970596: Package contrail-install-packages not installed. 2017-01-11 15:31:26:970796: [root@10.84.13.33] sudo: mkdir -p /tmp/tmp8wpAVc 2017-01-11 15:31:26:874289: [root@10.84.13.223] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmpuyn1jQ/contrail-install-packages_3.1.1.0-50~mitaka_all.deb 2017-01-11 15:31:26:813005: [root@10.84.13.2] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmplDNqKy/contrail-install-packages_3.1.1.0-50~mitaka_all.deb 2017-01-11 15:31:26:971363: [root@10.84.13.33] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmp8wpAVc/contrail-install-packages_3.1.1.0-50~mitaka_all.deb 2017-01-11 15:31:28:444810: Fatal error: Needed to prompt for a connection or sudo password (host: root@10.84.20.42), but input would be ambiguous in parallel mode 2017-01-11 15:31:28:444810: 2017-01-11 15:31:28:444990: Aborting. 2017-01-11 15:31:28:444990: 2017-01-11 15:31:26:297048: !!! Parallel execution exception under host 'root@10.84.20.242': 2017-01-11 15:31:36:288417: !!! Parallel execution exception under host 'root@10.84.20.240': 2017-01-11 15:31:36:300187: Process root@10.84.20.242: 2017-01-11 15:31:36:308947: Traceback (most recent call last): 2017-01-11 15:31:36:309163: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap 2017-01-11 15:31:36:309290: self.run() 2017-01-11 15:31:36:309679: File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run 2017-01-11 15:31:36:309807: self._target(*self._args, **self._kwargs) 2017-01-11 15:31:36:309924: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 236, in inner 2017-01-11 15:31:36:310030: submit(task.run(*args, **kwargs)) 2017-01-11 15:31:36:310245: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run 2017-01-11 15:31:36:310357: return self.wrapped(*args, **kwargs) 2017-01-11 15:31:36:310465: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 181, in inner 2017-01-11 15:31:36:310542: return func(*args, **kwargs) 2017-01-11 15:31:36:310705: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator 2017-01-11 15:31:36:310797: return func(*args, **kwargs) 2017-01-11 15:31:36:310891: File "/opt/contrail/utils/fabfile/tasks/install.py", line 50, in install_pkg_all 2017-01-11 15:31:36:310973: execute('install_pkg_node', pkg, env.host_string) 2017-01-11 15:31:36:311333: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 415, in execute 2017-01-11 15:31:36:311436: results[''] = task.run(*args, **new_kwargs) 2017-01-11 15:31:36:311539: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run 2017-01-11 15:31:36:311627: return self.wrapped(*args, **kwargs) 2017-01-11 15:31:36:311730: File "/opt/contrail/utils/fabfile/tasks/install.py", line 123, in install_pkg_node 2017-01-11 15:31:36:311829: versions = get_pkg_version_release(pkgname) 2017-01-11 15:31:36:311932: File "/opt/contrail/utils/fabfile/utils/fabos.py", line 116, in get_pkg_version_release 2017-01-11 15:31:36:312018: dist = detect_ostype() 2017-01-11 15:31:36:312178: File "/opt/contrail/utils/fabfile/utils/fabos.py", line 23, in detect_ostype 2017-01-11 15:31:36:312259: (dist, version, extra) = get_linux_distro() 2017-01-11 15:31:36:312341: File "/opt/contrail/utils/fabfile/utils/fabos.py", line 19, in get_linux_distro 2017-01-11 15:31:36:312424: (dist, version, extra) = ast.literal_eval(sudo(linux_distro)) 2017-01-11 15:31:36:312512: File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 633, in host_prompting_wrapper 2017-01-11 15:31:36:312595: return func(*args, **kwargs) 2017-01-11 15:31:36:312846: File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 1095, in sudo 2017-01-11 15:31:36:312958: stderr=stderr, timeout=timeout, shell_escape=shell_escape, 2017-01-11 15:31:36:313291: File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 909, in _run_command 2017-01-11 15:31:36:313376: channel=default_channel(), command=wrapped_command, pty=pty, 2017-01-11 15:31:36:313478: File "/usr/local/lib/python2.7/dist-packages/fabric/state.py", line 387, in default_channel 2017-01-11 15:31:36:313566: chan = _open_session() 2017-01-11 15:31:36:313741: File "/usr/local/lib/python2.7/dist-packages/fabric/state.py", line 379, in _open_session 2017-01-11 15:31:36:313824: return connections[env.host_string].get_transport().open_session() 2017-01-11 15:31:36:313924: File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 154, in __getitem__ 2017-01-11 15:31:36:314019: self.connect(key) 2017-01-11 15:31:36:314123: File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 146, in connect 2017-01-11 15:31:36:314202: user, host, port, cache=self, seek_gateway=seek_gateway) 2017-01-11 15:31:36:314304: File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 559, in connect 2017-01-11 15:31:36:314383: raise NetworkError(msg, e)