Vcenter-as-compute: multi-DC provisioning fails

[Sarath]

Per Inputs from Jeba & Sachin, This multi-DC provisioning to work with single nova-compute per DC
On 3.1.1#50 and also tried 3.2#19 with mitaka

This is tracking bug for the failure

Below the upgrade procedure used,
dpkg -i /root/contrail-install-packages_3.1.0.0-50~mitaka_all.deb
cd /opt/contrail/contrail_packages
./setup.sh
cd /opt/contrail/utils/
<< /opt/contrail/utils/fabfile/testbeds/testbed.py
fab prov_esxi
fab install_pkg_all:/root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb
  ## Failures seen during install_pkg

Also copied the fab logs and TB file to bugs
2017-01-11 16:24:14:022493: [root@10.84.13.2] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmpolAk1B/contrail-install-packages_3.1.1.0-50~mitaka_all.deb
2017-01-11 16:24:14:028879: [root@10.84.13.33] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmpQu_Z7c/contrail-install-packages_3.1.1.0-50~mitaka_all.deb
2017-01-11 16:24:13:675484: [root@10.84.13.222] out: ('Ubuntu', '14.04', 'trusty')
2017-01-11 16:24:14:177101: [root@10.84.13.222] out:
2017-01-11 16:24:14:177778:
2017-01-11 16:24:14:177956: [root@10.84.13.222] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages
2017-01-11 16:24:13:670931: [root@10.84.13.223] out: ('Ubuntu', '14.04', 'trusty')
2017-01-11 16:24:14:192489: [root@10.84.13.223] out:
2017-01-11 16:24:14:194556:
2017-01-11 16:24:14:209394: [root@10.84.13.223] sudo: dpkg-query -W -f='${VERSION}\n' contrail-install-packages
2017-01-11 16:24:14:178240: [root@10.84.13.222] out: dpkg-query: no packages found matching contrail-install-packages
2017-01-11 16:24:14:282374: [root@10.84.13.222] out:
2017-01-11 16:24:14:282735:

2017-01-11 16:24:14:283030: Warning: sudo() received nonzero return code 1 while executing 'dpkg-query -W -f='${VERSION}\n' contrail-install-packages'!
2017-01-11 16:24:14:283030:
2017-01-11 16:24:14:283030: 2017-01-11 16:24:14:282913: Package contrail-install-packages not installed.
2017-01-11 16:24:14:283211: [root@10.84.13.222] sudo: mkdir -p /tmp/tmpPitTHj
2017-01-11 16:24:14:283840: [root@10.84.13.222] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmpPitTHj/contrail-install-packages_3.1.1.0-50~mitaka_all.deb
2017-01-11 16:24:14:209702: [root@10.84.13.223] out: dpkg-query: no packages found matching contrail-install-packages
2017-01-11 16:24:14:408605: [root@10.84.13.223] out:
2017-01-11 16:24:14:408946:

2017-01-11 16:24:14:409267: Warning: sudo() received nonzero return code 1 while executing 'dpkg-query -W -f='${VERSION}\n' contrail-install-packages'!
2017-01-11 16:24:14:409267:
2017-01-11 16:24:14:409267: 2017-01-11 16:24:14:409151: Package contrail-install-packages not installed.
2017-01-11 16:24:14:409443: [root@10.84.13.223] sudo: mkdir -p /tmp/tmp7SPQMX
2017-01-11 16:24:14:410035: [root@10.84.13.223] put: /root/contrail-install-packages_3.1.1.0-50~mitaka_all.deb -> /tmp/tmp7SPQMX/contrail-install-packages_3.1.1.0-50~mitaka_all.deb

2017-01-11 16:24:15:652046: Fatal error: Needed to prompt for a connection or sudo password (host: root@10.84.20.42), but input would be ambiguous in parallel mode
2017-01-11 16:24:15:652046:
2017-01-11 16:24:15:652237: Aborting.
2017-01-11 16:24:15:652237: 2017-01-11 16:24:13:676199:

!!! Parallel execution exception under host 'root@10.84.14.243':
2017-01-11 16:24:23:672276: !!! Parallel execution exception under host 'root@10.84.14.242':
2017-01-11 16:24:23:676195: Process root@10.84.14.243:
2017-01-11 16:24:23:676778: Traceback (most recent call last):
2017-01-11 16:24:23:676947: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
2017-01-11 16:24:23:677060: self.run()
2017-01-11 16:24:23:677290: File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
2017-01-11 16:24:23:677393: self._target(*self._args, **self._kwargs)
2017-01-11 16:24:23:677497: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 236, in inner
2017-01-11 16:24:23:677594: submit(task.run(*args, **kwargs))
2017-01-11 16:24:23:677807: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-01-11 16:24:23:677894: return self.wrapped(*args, **kwargs)
2017-01-11 16:24:23:677979: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 181, in inner
2017-01-11 16:24:23:678054: return func(*args, **kwargs)
2017-01-11 16:24:23:678208: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator
2017-01-11 16:24:23:678294: return func(*args, **kwargs)

[Sarath (nsarath)]

nsarath@ubuntu-build04:/auto/cores/1655786$ ls -l
total 17408
-rwxrwxrwx 1 nsarath test 15237120 Jan 11 15:59 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 2508800 Jan 11 15:59 Ctrl-A-Utils.tar
nsarath@ubuntu-build04:/auto/cores/1655786$
nsarath@ubuntu-build04:/auto/cores/1655786$ pwd
/auto/cores/1655786

[Rudra Rugge (rrugge)]

Targeted starting 4.0

[Sachchidanand Vaidya (vaidyasd)]

Hi Sarath,
    As discussed the failures are not vcenter specific but more of setup issue. Can you rerun and let me know if this is still an issue.

Thanks
Sachin

[Sarath (nsarath)]

Hi Sachin,

correcting the build parameters, we are hitting issue during fab_setup_vcenter and setup_all as below,

2017-01-26 17:20:36:689591: [root@10.84.13.38] out: [localhost] local: sudo mv /tmp/tmpdrJP5j/vrouter_nodemgr_param /etc/contrail/vrouter_nodemgr_param
2017-01-26 17:20:36:705055: [root@10.84.13.38] out: [localhost] local: netstat -rn | grep ^"0.0.0.0" | head -n 1 | grep p4p1 | awk '{ print $2 }'
2017-01-26 17:20:36:708435: [root@10.84.13.38] out: [localhost] local: sudo sed 's/dev=.*/dev=p4p1/g' /etc/contrail/agent_param.tmpl > agent_param.new
2017-01-26 17:20:36:711756: [root@10.84.13.38] out: sed: can't read /etc/contrail/agent_param.tmpl: No such file or directory
2017-01-26 17:20:36:719200: [root@10.84.13.38] out:
2017-01-26 17:20:36:720603: [root@10.84.13.38] out: Fatal error: local() encountered an error (return code 2) while executing 'sudo sed 's/dev=.*/dev=p4p1/g' /etc/contrail/agent_param.tmpl > agent_param.new'
2017-01-26 17:20:36:720736: [root@10.84.13.38] out:
2017-01-26 17:20:36:720826: [root@10.84.13.38] out: Aborting.
2017-01-26 17:20:36:720906: [root@10.84.13.38] out:
2017-01-26 17:20:36:736685:

2017-01-26 17:20:36:744963: Warning: sudo() received nonzero return code 1 while executing 'setup-vnc-compute --self_ip 10.84.13.38 --cfgm_ip 10.84.13.32 --cfgm_user root --cfgm_passwd c0ntrail123 --ncontrols 2 --amqp_server_ip 10.84.13.32 --service_token 12a12a61806944a88999 --orchestrator openstack --hypervisor libvirt --keystone_ip 10.84.13.32 --keystone_version v2.0 --openstack_mgmt_ip 10.84.13.32 --keystone_auth_protocol http --keystone_auth_port 35357 --quantum_service_protocol http --keystone_admin_user admin --keystone_admin_password contrail123 --nova_password contrail123 --neutron_password contrail123 --service_tenant_name service --region_name RegionOne'!
2017-01-26 17:20:36:744963:
2017-01-26 17:20:36:744963: 2017-01-26 17:20:36:744873: Traceback (most recent call last):
2017-01-26 17:20:36:745159: File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 743, in main
2017-01-26 17:20:36:745182: *args, **kwargs
2017-01-26 17:20:36:745317: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 375, in execute
2017-01-26 17:20:36:745344: multiprocessing
2017-01-26 17:20:36:745424: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 271, in _execute
2017-01-26 17:20:36:745442: return task.run(*args, **kwargs)
2017-01-26 17:20:36:745535: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-01-26 17:20:36:745562: return self.wrapped(*args, **kwargs)
2017-01-26 17:20:36:745631: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator
2017-01-26 17:20:36:745659: return func(*args, **kwargs)
2017-01-26 17:20:36:745717: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 168, in __call__
2017-01-26 17:20:36:745744: return self.run(*args, **kwargs)
2017-01-26 17:20:36:745813: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-01-26 17:20:36:745839: return self.wrapped(*args, **kwargs)
2017-01-26 17:20:36:745911:...

[Sarath (nsarath)]

This bug tracked only for older early < 4.0 release multi-cluster/multi-dc support. Not yet verified new knobs/tests on 4.0 and we will open different bug if required for it..

[amudhar (amudha)]

Fix for multi-cluster support on mitaka committed through #1646680.