CI Job some times fail in python /opt/contrail/utils/provision_vrouter.py

[Ananth Suryanarayana]

This is supposed already fixed in https://review.opencontrail.org/#/c/4974/
Raising a bug to track it. Many CI jobs have been affected by this issue

https://jenkins.opencontrail.org/job/ci-contrail-web-core-systest-ubuntu-precise-pangolin-havana/321/console

015-01-14 02:36:49 2015-01-14 02:35:17:266397: [root@192.168.255.199] out: [localhost] local: sudo rm /tmp/tmpm7R6GJ/ctrl-details
2015-01-14 02:36:49 2015-01-14 02:35:17:274242: [root@192.168.255.199] out: [localhost] local: sudo service libvirt-bin restart
2015-01-14 02:36:49 2015-01-14 02:35:17:290149: [root@192.168.255.199] out: libvirt-bin stop/waiting
2015-01-14 02:36:49 2015-01-14 02:35:17:323180: [root@192.168.255.199] out: libvirt-bin start/running, process 25123
2015-01-14 02:36:49 2015-01-14 02:35:17:355409: [root@192.168.255.199] out: [localhost] local: sudo compute-server-setup.sh
2015-01-14 02:36:49 2015-01-14 02:35:17:355613: [root@192.168.255.199] out: 2:2013.2.1-0ubuntu1.2contrail
2015-01-14 02:36:54 2015-01-14 02:35:17:419761: [root@192.168.255.199] out: [localhost] local: chkconfig openstack-nova-compute on
2015-01-14 02:36:55 2015-01-14 02:35:22:995964: [root@192.168.255.199] out: [localhost] local: chkconfig supervisor-vrouter on
2015-01-14 02:36:55 2015-01-14 02:35:23:369448: [root@192.168.255.199] out: [root@192.168.255.199] run: python /opt/contrail/utils/provision_vrouter.py --host_name ci-oc-subslave-ubuntu-10-84-35-185-1.localdomain.com --host_ip 192.168.255.199 --api_server_ip 192.168.255.199 --oper add --admin_user admin --admin_password c0ntrail123 --admin_tenant_name admin --openstack_ip 192.168.255.199
2015-01-14 02:37:24 2015-01-14 02:35:23:369702: [root@192.168.255.199] out: [root@192.168.255.199] out: No handlers could be found for logger "vnc_api.vnc_api"
2015-01-14 02:37:24 2015-01-14 02:35:52:872306: [root@192.168.255.199] out: [root@192.168.255.199] out: Traceback (most recent call last):
2015-01-14 02:37:24 2015-01-14 02:35:52:904705: [root@192.168.255.199] out: [root@192.168.255.199] out: File "/opt/contrail/utils/provision_vrouter.py", line 183, in <module>
2015-01-14 02:37:24 2015-01-14 02:35:52:904929: [root@192.168.255.199] out: [root@192.168.255.199] out: main()
2015-01-14 02:37:24 2015-01-14 02:35:52:905083: [root@192.168.255.199] out: [root@192.168.255.199] out: File "/opt/contrail/utils/provision_vrouter.py", line 179, in main
2015-01-14 02:37:24 2015-01-14 02:35:52:905311: [root@192.168.255.199] out: [root@192.168.255.199] out: VrouterProvisioner(args_str)
2015-01-14 02:37:24 2015-01-14 02:35:52:905463: [root@192.168.255.199] out: [root@192.168.255.199] out: File "/opt/contrail/utils/provision_vrouter.py", line 42, in __init__
2015-01-14 02:37:24 2015-01-14 02:35:52:905610: [root@192.168.255.199] out: [root@192.168.255.199] out: fq_name=['default-global-system-config'])
2015-01-14 02:37:24 2015-01-14 02:35:52:905716: [root@192.168.255.199] out: [root@192.168.255.199] out: File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 2122, in global_system_config_read
2015-01-14 02:37:25 2015-01-14 02:35:52:905843: [root@192.168.255.199] out: [root@192.168.255.199] out: (args_ok, result) = self._read_args_to_id('global-system-config', fq_name, fq_name_str, id, ifmap_id)
2015-01-14 02:37:25 2015-01-14 02:35:52:906012: [root@192.168.255.199] out: [root@192.168.255.199] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 329, in _read_args_to_id
2015-01-14 02:37:25 2015-01-14 02:35:52:906176: [root@192.168.255.199] out: [root@192.168.255.199] out: return (True, self.fq_name_to_id(obj_type, fq_name))
2015-01-14 02:37:25 2015-01-14 02:35:52:906294: [root@192.168.255.199] out: [root@192.168.255.199] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 436, in fq_name_to_id
2015-01-14 02:37:25 2015-01-14 02:35:52:906699: [root@192.168.255.199] out: [root@192.168.255.199] out: uri = self._action_uri['name-to-id']
2015-01-14 02:37:25 2015-01-14 02:35:52:906804: [root@192.168.255.199] out: [root@192.168.255.199] out: KeyError: 'name-to-id'
2015-01-14 02:37:25 2015-01-14 02:35:52:906899: [root@192.168.255.199] out: [root@192.168.255.199] out:
2015-01-14 02:37:25 2015-01-14 02:35:52:907063: [root@192.168.255.199] out:

[Nagendra Prasath (npchandran)]

Noticed the similar issue when bring up Rhel70-contrail setup in a VM. Please see the attachment contrail-svc-monitor-stdout.log
contrail-status:

== Contrail Control ==
supervisor-control: active
contrail-control initializing (Number of connections:4, Expected:5)
contrail-control-nodemgr active
contrail-dns active
contrail-named active
...
...
...
== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor initializing (ApiServer:ApiServer connection down)
ifmap active

[Vijay Anand (vijanand)]

I'm running into similar issue, contrail2.0+Juno(Ubuntu14.04 LTS)

2015-02-16 13:29:15:011820: [root@mgmt_ip] out: [root@9.9.9.2] run:
python /opt/contrail/utils/provision_vrouter.py --host_name Contrail-NFVO
--host_ip 9.9.9.2 --api_server_ip 9.9.9.2 --oper add --admin_user admin
--admin_password Embe1mpls --admin_tenant_name admin --openstack_ip 9.9.9.2
2015-02-16 13:29:15:011994: [root@mgmt_ip] out: [root@9.9.9.2] out: No
handlers could be found for logger "vnc_api.vnc_api"
2015-02-16 13:29:44:694673: [root@mgmt_ip] out: [root@9.9.9.2] out:
Traceback (most recent call last):
2015-02-16 13:29:44:694951: [root@mgmt_ip] out: [root@9.9.9.2] out:
File "/opt/contrail/utils/provision_vrouter.py", line 178, in <module>
2015-02-16 13:29:44:695117: [root@mgmt_ip] out: [root@9.9.9.2] out:
main()
2015-02-16 13:29:44:695295: [root@mgmt_ip] out: [root@9.9.9.2] out:
File "/opt/contrail/utils/provision_vrouter.py", line 174, in main
2015-02-16 13:29:44:695469: [root@mgmt_ip] out: [root@9.9.9.2] out:
VrouterProvisioner(args_str)
2015-02-16 13:29:44:695666: [root@mgmt_ip] out: [root@9.9.9.2] out:
File "/opt/contrail/utils/provision_vrouter.py", line 42, in __init__
2015-02-16 13:29:44:695850: [root@mgmt_ip] out: [root@9.9.9.2] out:
fq_name=['default-global-system-config'])
2015-02-16 13:29:44:696015: [root@mgmt_ip] out: [root@9.9.9.2] out:
File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py",
line 2122, in global_system_config_read
2015-02-16 13:29:44:696179: [root@mgmt_ip] out: [root@9.9.9.2] out:
(args_ok, result) = self._read_args_to_id('global-system-config',
fq_name, fq_name_str, id, ifmap_id)
2015-02-16 13:29:44:696359: [root@mgmt_ip] out: [root@9.9.9.2] out:
File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 326, in
_read_args_to_id
2015-02-16 13:29:44:696533: [root@mgmt_ip] out: [root@9.9.9.2] out:
return (True, self.fq_name_to_id(obj_type, fq_name))
2015-02-16 13:29:44:696675: [root@mgmt_ip] out: [root@9.9.9.2] out:
File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 430, in
fq_name_to_id
2015-02-16 13:29:44:696817: [root@mgmt_ip] out: [root@9.9.9.2] out:
uri = self._action_uri['name-to-id']
2015-02-16 13:29:44:696958: [root@mgmt_ip] out: [root@9.9.9.2] out:
KeyError: 'name-to-id'
2015-02-16 13:29:44:697098: [root@mgmt_ip] out: [root@9.9.9.2] out:
2015-02-16 13:29:44:697236: [root@mgmt_ip] out:
2015-02-16 13:29:44:730104: [root@mgmt_ip] out:
2015-02-16 13:29:44:730363: [root@mgmt_ip] out: Fatal error: run()
received nonzero return code 1 while executing!
2015-02-16 13:29:44:730536: [root@mgmt_ip] out:
2015-02-16 13:29:44:730729: [root@mgmt_ip] out: Requested: python
/opt/contrail/utils/provision_vrouter.py --host_name Contrail-NFVO
--host_ip 9.9.9.2 --api_server_ip 9.9.9.2 --oper add --admin_user admin
--admin_password Embe1mpls --admin_tenant_name admin --openstack_ip 9.9.9.2
2015-02-16 13:29:44:730893: [root@mgmt_ip] out: Executed: /bin/bash -l
-c "python /opt/contrail/utils/provision_vrouter.py --host_name
Contrail-NFVO --host_ip 9.9.9.2 --api_server_ip 9.9.9.2 --oper add
--admin_user admin --admin_password Embe1mpls --admin_tenant_name admin
--openstack_ip 9.9.9.2"
2015-02-16 13:29:44:731106: [roo...

[Vijay Anand (vijanand)]

root@Contrail-NFVO:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

== Contrail Control ==
supervisor-control: active
contrail-control initializing (Number of connections:4, Expected: 5)
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active

== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (IFMap:IfMap connection down)
contrail-config-nodemgr active
contrail-discovery:0 active
contrail-schema initializing
contrail-svc-monitor initializing
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
supervisor-database: active
contrail-database active
contrail-database-nodemgr active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@Contrail-NFVO:~#

[Vijay Anand (vijanand)]

Any workaround?

[Ignatious Johnson Christopher (ijohnson-x)]

Fixed in R2.1 with commit https://github.com/Juniper/contrail-controller/commit/06a0df648e2d9c19e983d11cbbe1a19beab5076d#diff-7c86bfdc416a185870c50db9633701a0

Fixed in trunk with commit https://github.com/Juniper/contrail-controller/commit/d4bfdf6e6e6d24ee1329a7d60ae2623d1983fde8