Vcenter-only: provisioning broken on latest build #3

[Sarath]

Amudha debugged the setup evening and opening this bug for tracking and this issue seen both when
using "contrail-installer package part of bug fix#1599583 (or) usual contrail-install package

Per initial triaging, looks default vcenter-only mode not picked from testbed.py definitions.
Attached to bug, logs and also Fab cmd Logs

###########
Contrail-installer part of bug-fix #1599583
##########

2016-07-21 15:58:46:935257:
2016-07-21 15:58:46:947855: INFO:
2016-07-21 15:58:46:947855: Openstack and cfgm nodes are same, No need for contrail_internal_vip to be specified in testbed.py.
2016-07-21 15:58:46:947916: [root@10.87.26.197] Executing task 'setup_rabbitmq_cluster'
2016-07-21 15:58:46:948163: Provisioning rabbitq in cfgm nodes
2016-07-21 15:58:46:948283: [root@10.87.26.197] Executing task 'verify_cluster_status'
2016-07-21 15:58:46:948402: [root@10.87.26.197] sudo: service rabbitmq-server status
2016-07-21 15:58:46:948544: [root@10.87.26.197] out: rabbitmq-server RUNNING pid 1877, uptime 0:02:20
2016-07-21 15:58:47:179843: [root@10.87.26.197] out:

::
::

2016-07-21 16:13:55:988804: [root@10.87.26.199] out:
2016-07-21 16:14:26:011303: [root@10.87.26.199] out: DIAGNOSTICS
2016-07-21 16:14:26:011506: [root@10.87.26.199] out: ===========
2016-07-21 16:14:26:011578: [root@10.87.26.199] out:
2016-07-21 16:14:26:011650: [root@10.87.26.199] out: attempted to contact: ['rabbit@oblocknode04-ctrl']
2016-07-21 16:14:26:011711: [root@10.87.26.199] out:
2016-07-21 16:14:26:011790: [root@10.87.26.199] out: rabbit@oblocknode04-ctrl:
2016-07-21 16:14:26:011869: [root@10.87.26.199] out: * unable to connect to epmd (port 4369) on oblocknode04-ctrl: timeout (timed out)
2016-07-21 16:14:26:011954: [root@10.87.26.199] out:
2016-07-21 16:14:26:012023: [root@10.87.26.199] out:
2016-07-21 16:14:26:012095: [root@10.87.26.199] out: current node details:
2016-07-21 16:14:26:012168: [root@10.87.26.199] out: - node name: 'rabbitmq-cli-27908@oblocknode04'
2016-07-21 16:14:26:012229: [root@10.87.26.199] out: - home dir: /var/lib/rabbitmq
2016-07-21 16:14:26:012288: [root@10.87.26.199] out: - cookie hash: ak2LCDpyGddGyEdUhkGghw==
2016-07-21 16:14:26:012348: [root@10.87.26.199] out:
2016-07-21 16:14:26:012407: [root@10.87.26.199] out:
2016-07-21 16:14:26:012928:
2016-07-21 16:14:26:030521: Unable to setup RabbitMQ cluster in role[cfgm]....

###########
Contrail-install
##########

2016-07-21 23:23:13:186807: [root@10.87.26.197] out: Error {'succeeded': True, 'failed': False, 'return_code': 0, 'stderr': ''} in getting status of [contrail-api].
2016-07-21 23:23:13:250537: [root@10.87.26.197] out: [contrail-api and rabbitmq] not yet started by supervisor config, Retrying.
2016-07-21 23:23:13:250639: [root@10.87.26.197] out: [localhost] local: sudo service rabbitmq-server status
2016-07-21 23:23:15:252948: [root@10.87.26.197] out: [rabbitmq-server] started by supervisor config.
2016-07-21 23:23:15:353317: [root@10.87.26.197] out: [localhost] local: sudo service contrail-api status
2016-07-21 23:23:15:353450: [root@10.87.26.197] out: [contrail-api] started by supervisor config.
2016-07-21 23:23:15:417126: [root@10.87.26.197] out: [contrail-api and rabbitmq] started by supervisor config, continue to provision.
2016-07-21 23:23:15:417267: [root@10.87.26.197] out:
2016-07-21 23:23:15:417446:
2016-07-21 23:23:15:417804: [localhost] local: echo '10.87.26.205:172.16.80.110' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:418011: [localhost] local: echo '10.87.26.203:172.16.80.108' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:420256: [localhost] local: echo '10.87.26.201:172.16.80.106' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:421855: [localhost] local: echo '10.87.26.207:172.16.80.112' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:423371: [localhost] local: echo '10.87.26.200:172.16.80.105' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:424992: [root@10.87.26.197] put: /tmp/ESXiToVRouterIp-root@10.87.26.197 -> /etc/contrail/ESXiToVRouterIp.map
2016-07-21 23:23:15:433465: [localhost] local: rm /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:454006: [root@10.87.26.197] sudo: setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only
2016-07-21 23:23:15:456818: [root@10.87.26.197] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-07-21 23:23:15:590154: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/tmpDyMbcj/contrail-vcenter-plugin.conf /etc/contrail/contrail-vcenter-plugin.conf
2016-07-21 23:23:15:593572: [root@10.87.26.197] out: [localhost] local: sudo vcenter-plugin-setup.sh
2016-07-21 23:23:15:601020: [root@10.87.26.197] out: contrail-vcenter-plugin: unrecognized service
2016-07-21 23:23:15:616550: [root@10.87.26.197] out:
2016-07-21 23:23:15:616661: [root@10.87.26.197] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo vcenter-plugin-setup.sh'
2016-07-21 23:23:15:616726: [root@10.87.26.197] out:
2016-07-21 23:23:15:616816: [root@10.87.26.197] out: Aborting.
2016-07-21 23:23:15:616874: [root@10.87.26.197] out:
2016-07-21 23:23:15:618270:
2016-07-21 23:23:15:621729: Disconnecting from 10.87.26.208... done.
2016-07-21 23:23:15:627447: Disconnecting from 10.87.26.197... done.
2016-07-21 23:23:15:718619: Disconnecting from 10.87.26.199... done.
2016-07-21 23:23:15:796082:
2016-07-21 23:22:48:988795: Warning: sudo() received nonzero return code 2 while executing 'sudo ls /etc/contrail/haproxy.token'!
2016-07-21 23:22:48:988795:
2016-07-21 23:22:48:988795:
2016-07-21 23:22:51:061828: Warning: sudo() received nonzero return code 2 while executing 'sudo ls /etc/contrail/service.token'!
2016-07-21 23:22:51:061828:
2016-07-21 23:22:51:061828:
2016-07-21 23:23:15:621833: Fatal error: sudo() received nonzero return code 1 while executing!
2016-07-21 23:23:15:621833:
2016-07-21 23:23:15:621833: Requested: setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only
2016-07-21 23:23:15:621833: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "cd /opt/contrail/bin && setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only"
2016-07-21 23:23:15:621833:

[Sarath (nsarath)]

-bash-4.1$ pwd
/cs-shared/bugs/1605500
-bash-4.1$
-bash-4.1$ ls -l
total 69496
-rwxrwxrwx 1 nsarath test 26009600 Jul 21 21:57 Ctrl-A-log.tar*
-rwxrwxrwx 1 nsarath test 4802560 Jul 21 22:04 Ctrl-A-Utils.tar*
-rwxrwxrwx 1 nsarath test 20121600 Jul 21 21:57 Ctrl-B-log.tar*
-rwxrwxrwx 1 nsarath test 19916800 Jul 21 21:57 Ctrl-C-log.tar*
drwxrwxrwx 4 nsarath test 8192 Jul 22 00:07 utils/

[amudhar (amudha)]

With build 3, I do not see any provisioning failures for vcenter-only mode.
With build 5, setup_all failed due to #1606654.

Fix will be committed through #1606654, closing this bug.

Please reopen the bug if you again see the issue you have reported with build 3.

-Amudha

[amudhar (amudha)]

The issue reported on Sarath's setup with build 3 described in this bug is not due to openstack roles or vcenter-only mode ignored (as in the description).

The error seems to be due to missing setup-vcenter-plugin.
2016-07-21 23:23:15:454006: [root@10.87.26.197] sudo: setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only
2016-07-21 23:23:15:456818: [root@10.87.26.197] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-07-21 23:23:15:590154: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/tmpDyMbcj/contrail-vcenter-plugin.conf /etc/contrail/contrail-vcenter-plugin.conf
2016-07-21 23:23:15:593572: [root@10.87.26.197] out: [localhost] local: sudo vcenter-plugin-setup.sh
2016-07-21 23:23:15:601020: [root@10.87.26.197] out: contrail-vcenter-plugin: unrecognized service
2016-07-21 23:23:15:616550: [root@10.87.26.197] out:
2016-07-21 23:23:15:616661: [root@10.87.26.197] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo vcenter-plugin-setup.sh'

This might be because of the vcenter-plugin not installed on 10.87.26.197. I was not able to see this issue on my single node setup with build 3, will triage if this is reported happening again.

-Amudha