R3.0-54-liberty: provision vrouter failed. Rbac with keystone v3 enabled setup

[shajuvk]

Please check the node a5s8 for testbed - 10.84.14.8
=====================================

2016-08-25 11:47:52:501441: [root@10.84.14.10] out: [localhost] local: sudo mv /tmp/tmpqjmmGl/contrail-lbaas-auth.conf /etc/contrail/contrail-lbaas-auth.conf
2016-08-25 11:47:52:509053: [root@10.84.14.10] out: [localhost] local: python /opt/contrail/utils/provision_vrouter.py --host_name a5s10 --host_ip 192.168.10.5 --api_server_ip 192.168.10.3 --oper add --admin_user admin --admin_password contrail123 --admin_tenant_name admin --openstack_ip 192.168.10.4
2016-08-25 11:47:52:516665: [root@10.84.14.10] out: Traceback (most recent call last):
2016-08-25 11:47:52:949131: [root@10.84.14.10] out: File "/opt/contrail/utils/provision_vrouter.py", line 185, in <module>
2016-08-25 11:47:52:949250: [root@10.84.14.10] out: main()
2016-08-25 11:47:52:949353: [root@10.84.14.10] out: File "/opt/contrail/utils/provision_vrouter.py", line 181, in main
2016-08-25 11:47:52:949526: [root@10.84.14.10] out: VrouterProvisioner(args_str)
2016-08-25 11:47:52:949605: [root@10.84.14.10] out: File "/opt/contrail/utils/provision_vrouter.py", line 42, in __init__
2016-08-25 11:47:52:949666: [root@10.84.14.10] out: fq_name=['default-global-system-config'])
2016-08-25 11:47:52:949727: [root@10.84.14.10] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 40, in wrapper
2016-08-25 11:47:52:949785: [root@10.84.14.10] out: return func(self, *args, **kwargs)
2016-08-25 11:47:52:949844: [root@10.84.14.10] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 403, in _object_read
2016-08-25 11:47:52:949904: [root@10.84.14.10] out: res_type, fq_name, fq_name_str, id, ifmap_id)
2016-08-25 11:47:52:949965: [root@10.84.14.10] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 699, in _read_args_to_id
2016-08-25 11:47:52:950026: [root@10.84.14.10] out: return (True, self.fq_name_to_id(obj_type, fq_name))
2016-08-25 11:47:52:950087: [root@10.84.14.10] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 40, in wrapper
2016-08-25 11:47:52:950146: [root@10.84.14.10] out: return func(self, *args, **kwargs)
2016-08-25 11:47:52:950220: [root@10.84.14.10] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 912, in fq_name_to_id
2016-08-25 11:47:52:950276: [root@10.84.14.10] out: raise he
2016-08-25 11:47:52:950331: [root@10.84.14.10] out: cfgm_common.exceptions.HttpError: HTTP Status: 401 Content: Authentication required
2016-08-25 11:47:52:950388: [root@10.84.14.10] out:
2016-08-25 11:47:52:966802: [root@10.84.14.10] out: Fatal error: local() encountered an error (return code 1) while executing 'python /opt/contrail/utils/provision_vrouter.py --host_name a5s10 --host_ip 192.168.10.5 --api_server_ip 192.168.10.3 --oper add --admin_user admin --admin_password contrail123 --admin_tenant_name admin --openstack_ip 192.168.10.4'
2016-08-25 11:47:52:966895: [root@10.84.14.10] out:
2016-08-25 11:47:52:966961: [root@10.84.14.10] out: Aborting.
2016-08-25 11:47:52:967024: [root@10.84.14.10] out:
2016-08-25 11:47:52:982648:

2016-08-25 11:47:52:992207: Fatal error: sudo() received nonzero return code 1 while executing!
2016-08-25 11:47:52:992207:
2016-08-25 11:47:52:992207: Requested: setup-vnc-compute --self_ip 192.168.10.5 --cfgm_ip 192.168.10.3 --cfgm_user root --cfgm_passwd c0ntrail123 --ncontrols 1 --amqp_server_ip 192.168.10.3 --service_token 5c3040509203cd19b723 --orchestrator openstack --hypervisor libvirt --non_mgmt_ip 192.168.10.5 --non_mgmt_gw 192.168.10.254 --keystone_ip 192.168.10.4 --keystone_version v3 --openstack_mgmt_ip 10.84.14.9 --keystone_auth_protocol http --keystone_auth_port 35357 --quantum_service_protocol http --keystone_admin_user admin --keystone_admin_password contrail123 --nova_password contrail123 --neutron_password contrail123 --service_tenant_name service
2016-08-25 11:47:52:992207: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "cd /opt/contrail/bin && setup-vnc-compute --self_ip 192.168.10.5 --cfgm_ip 192.168.10.3 --cfgm_user root --cfgm_passwd c0ntrail123 --ncontrols 1 --amqp_server_ip 192.168.10.3 --service_token 5c3040509203cd19b723 --orchestrator openstack --hypervisor libvirt --non_mgmt_ip 192.168.10.5 --non_mgmt_gw 192.168.10.254 --keystone_ip 192.168.10.4 --keystone_version v3 --openstack_mgmt_ip 10.84.14.9 --keystone_auth_protocol http --keystone_auth_port 35357 --quantum_service_protocol http --keystone_admin_user admin --keystone_admin_password contrail123 --nova_password contrail123 --neutron_password contrail123 --service_tenant_name service"
2016-08-25 11:47:52:992207:
2016-08-25 11:47:52:992239: Aborting.

[shajuvk (shajuvk)]

This is not an rbac issue, deepinder debugged the issue and identified python-memcache is missing from cfgm.

[shajuvk (shajuvk)]

Ashish,

Please approve https://review.opencontrail.org/#/c/23636/.

Thanks,
Amudha

From: Amudha R <email address hidden>
Date: Friday, August 26, 2016 at 9:07 AM
To: Ashish Ranjan <email address hidden>, "Shaju V.K" <email address hidden>
Cc: Rudra Rugge <email address hidden>, Deepinder Singh Setia <email address hidden>, Sachin Bansal <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Provision router failed- R3.0-Build-54-Liberty

Raised https://review.opencontrail.org/#/c/23636/ for R3.0

-Amudha

From: Ashish Ranjan <email address hidden>
Date: Friday, August 26, 2016 at 8:33 AM
To: Amudha R <email address hidden>, "Shaju V.K" <email address hidden>
Cc: Rudra Rugge <email address hidden>, Deepinder Singh Setia <email address hidden>, Sachin Bansal <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Provision router failed- R3.0-Build-54-Liberty

Can we commit this asap. So is this an issue on 3.0.2 fcs build as well (build 51).

Ashish

From: Amudha R
Sent: Thursday, August 25, 2016 9:53:02 PM
To: Shaju V.K
Cc: Deepinder Singh Setia; Sachin Bansal; Ashish Ranjan; Jeba Paulaiyan; Rudra Rugge
Subject: Re: Provision router failed- R3.0-Build-54-Liberty

It does seem to be the same issue, and I had not committed the fix into R3.0.
The issue happenswhen openstack node is different than the config node.
Shaju, can you install the memcache packages on your node and re-run provision_vrouter.py,
that should confirm if the issue is fixed.

Thanks,
Amudha

From: "Shaju V.K" <email address hidden>
Date: Thursday, August 25, 2016 at 11:07 PM
To: Amudha R <email address hidden>
Cc: Deepinder Singh Setia <email address hidden>, Sachin Bansal <email address hidden>, Ashish Ranjan <email address hidden>, Jeba Paulaiyan <email address hidden>, Rudra Rugge <email address hidden>
Subject: Re: Provision router failed- R3.0-Build-54-Liberty

+ Amudha

Amudha,

Looks like this bug is similar to the issue we are seeing now. Commit for this bug is available only on 3.1 and Trunk not on R3.0. Could you please confirm it?

https://bugs.launchpad.net/juniperopenstack/+bug/1602618

Thanks,
Shaju

From: Ashish Ranjan <email address hidden>
Date: Thursday, August 25, 2016 at 8:24 PM
To: "Shaju V.K" <email address hidden>, Jeba Paulaiyan <email address hidden>
Cc: Deepinder Singh Setia <email address hidden>, Sachin Bansal <email address hidden>
Subject: Re: Provision router failed- R3.0-Build-54-Liberty

Deepinder Sachin can you help here..

From: Shaju V.K
Sent: Thursday, August 25, 2016 5:58:38 PM
To: Jeba Paulaiyan
Cc: Ashish Ranjan
Subject: Provision router failed- R3.0-Build-54-Liberty

Hi Jeba,

The below bug is not related the rbac. The provisioning failed due to missing python-memcache from cfgm. I doubt some check-in is missing in R3.0 branch

https://bugs.launchpad.net/juniperopenstack/+bug/1617034

Thanks,
Shaju

[shajuvk (shajuvk)]

Summarizing to avoid confusion:

Bug 1617034 (Reported by Shaju) -> Decided to be duplicate of 1602618. However, the same topology is there in Sanity and it passed in 3.0.2.0-51. So bug 1617034 is wrongly stated as duplicate of 1602618 (OR) it is a breakage after 3.0.2.0-51.
Bug 1616178 (Reported by Vivek) -> This is there in both 3.0.2.0-51 and now. In future, this case will be taken care when we add separate openstack and contrail HA nodes in Sanity.

Sanity testbed: (No Contrail-HA; No Openstack-HA; Multiple cfgm nodes and one of them is openstack node) - Bug 1602618

Config Nodes : [u'nodea35', u'nodea34']
Control Nodes : [u'nodea35', u'nodea34', u'nodec53']
Compute Nodes : [u'nodec54', u'nodec55', u'nodec56']
Openstack Node : nodea34
WebUI Node : nodec53
Analytics Nodes : [u'nodea35', u'nodec53’]

Shaju’s testbed: (No Contrail-HA; No Openstack-HA; cfgm and openstack on different nodes) - Bug 1617034

env.roledefs = {
    'all': [host1, host2, host3],
    'cfgm': [host1],
    'openstack': [host2],
    'control': [host1],
    'compute': [host3],
    'collector': [host1],
    'webui': [host2],
    'database': [host1],
    'build': [host_build],
}

Vivek’s testbed: (Contrail-HA and Openstack node separate ) - Bug 1616178

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6 ],
    'cfgm': [host1, host2, host3],
    'openstack': [host1],
    'control': [host1, host2, host3],
    'compute': [host4, host5, host6],
    'collector': [host1, host2, host3],
    'webui': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

Thanks,
Jeba

From: Vivek Garg <email address hidden>
Date: Friday, August 26, 2016 at 2:30 PM
To: Jeba Paulaiyan <email address hidden>, Ashish Ranjan <email address hidden>, Amudha R <email address hidden>, "Shaju V.K" <email address hidden>
Cc: Sachin Bansal <email address hidden>, Rudra Rugge <email address hidden>, Deepinder Singh Setia <email address hidden>, Sudheendra Rao <email address hidden>, Ignatious Johnson <email address hidden>
Subject: Re: Provision router failed- R3.0-Build-54-Liberty

Hi Jeba,

I see the following bug : https://bugs.launchpad.net/juniperopenstack/+bug/1616178

Version: R3.0-51 Liberty
Roles:
    'all': [host1, host2, host3, host4, host6 ],
    'cfgm': [host1, host2, host3],
    'openstack': [host4],
    'control': [host1, host2, host3],
    'compute': [ host6],
    'collector': [host1, host2, host3],
    'webui': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],

Thanks,
Vivek

From: Jeba Paulaiyan
Date: Friday, August 26, 2016 at 9:25 AM
To: Ashish Ranjan, Amudha R, "Shaju V.K", Vivek Garg
Cc: Sachin Bansal, Rudra Rugge, Deepinder Singh Setia, Sudheendra Rao
Subject: Re: Provision router failed- R3.0-Build-54-Liberty

Reviewed the Sanity results. All jobs has openstack and config role on same node. We have separated openstack and control, database, but config and openstack are shared.

Taking these two AIs:

Will confirm if the FCS build had this issue. (Vivek is trying it out)
Will change the Ubuntu HA testbed from 3 x (All in one role)+Computes —to—> 3 x (Openstack HA) + 3 x (Rest of roles and contrai...

[shajuvk (shajuvk)]

fixed in R3.0 build 56

[Vineet Gupta (vineetrf)]

Please close for all scopes based on your analysis.