During tempest run on centos/icehouse, adding interface to router failed

[Vedamurthy Joshi]

R1.10 Build 38

Was seen when running below tempest cases (test_floating_ip.py) on Centos 6.5 running icehouse.

http://anamika.englab.juniper.net:8080/job/Contrail_Tempest/206/testReport/

2014-09-24 10:57:49.080 903 INFO tempest.common.rest_client [-] Request (FloatingIPTestJSON:setUpClass): 200 POST http://10.204.217.146:5000/v2.0/tokens
2014-09-24 10:57:49.242 903 INFO tempest.common.rest_client [-] Request (FloatingIPTestJSON:setUpClass): 200 POST http://10.204.217.146:5000/v2.0/tokens
2014-09-24 10:57:49.269 903 INFO tempest.common.rest_client [-] Request (FloatingIPTestJSON:setUpClass): 200 POST http://10.204.217.146:35357/v2.0/tenants 0.024s
2014-09-24 10:57:49.353 903 INFO tempest.common.rest_client [-] Request (FloatingIPTestJSON:setUpClass): 200 POST http://10.204.217.146:35357/v2.0/users 0.082s
2014-09-24 10:57:49.466 903 INFO tempest.common.rest_client [-] Request (FloatingIPTestJSON:setUpClass): 200 POST http://10.204.217.146:5000/v2.0/tokens
2014-09-24 10:57:49.468 903 INFO tempest.common.isolated_creds [-] Acquired isolated creds:
 credentials: {'username': u'FloatingIPTestJSON-602729787', 'tenant_id': u'74a38a3ec73943a5add0c6d7181c2e06', 'password': 'pass', 'user_id': u'c00dc920daa84f6d957d9998ef89b9da', 'tenant_name': u'FloatingIPTestJSON-1508800324'}
2014-09-24 10:57:49.469 903 INFO tempest.common.isolated_creds [-] Created isolated network resources for :
 credentials: {'username': u'FloatingIPTestJSON-602729787', 'tenant_id': u'74a38a3ec73943a5add0c6d7181c2e06', 'password': 'pass', 'user_id': u'c00dc920daa84f6d957d9998ef89b9da', 'tenant_name': u'FloatingIPTestJSON-1508800324'}
2014-09-24 10:57:49.586 903 INFO tempest.common.rest_client [-] Request (FloatingIPTestJSON:setUpClass): 200 POST http://10.204.217.146:5000/v2.0/tokens
2014-09-24 10:57:50.420 903 INFO tempest.common.rest_client [req-8b8f53cc-9000-4317-9159-1100876c05e8 None] Request (FloatingIPTestJSON:setUpClass): 201 POST http://10.204.217.147:9696/v2.0/networks 0.832s
2014-09-24 10:57:51.014 903 INFO tempest.common.rest_client [req-695eefbe-8ac4-431e-9648-a39b72b36430 None] Request (FloatingIPTestJSON:setUpClass): 201 POST http://10.204.217.147:9696/v2.0/subnets 0.590s
2014-09-24 10:57:51.562 903 INFO tempest.common.rest_client [req-1363cadb-4652-4c42-af4b-99b423821aa3 None] Request (FloatingIPTestJSON:setUpClass): 201 POST http://10.204.217.147:9696/v2.0/routers 0.545s
2014-09-24 10:57:51.614 903 INFO tempest.common.rest_client [req-188ef51b-5f12-46ba-865a-4bd9a8f21b8b None] Request (FloatingIPTestJSON:setUpClass): 500 PUT http://10.204.217.147:9696/v2.0/routers/456358cf-e38e-4d97-bcb1-a7f0e3d70528/add_router_interface 0.049s
2014-09-24 10:57:51.615 903 ERROR tempest.test [-] setUpClass failed: Got server fault
Details: {"NeutronError": "Request Failed: internal server error while processing your request."}
2014-09-24 10:57:51.615 903 TRACE tempest.test Traceback (most recent call last):
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/test.py", line 76, in decorator
2014-09-24 10:57:51.615 903 TRACE tempest.test f(cls)
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/api/network/test_floating_ips.py", line 62, in setUpClass
2014-09-24 10:57:51.615 903 TRACE tempest.test cls.create_router_interface(cls.router['id'], cls.subnet['id'])
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/api/network/base.py", line 300, in create_router_interface
2014-09-24 10:57:51.615 903 TRACE tempest.test router_id, subnet_id)
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/services/network/json/network_client.py", line 116, in add_router_interface_with_subnet_id
2014-09-24 10:57:51.615 903 TRACE tempest.test resp, body = self.put(uri, update_body)
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/services/network/network_client_base.py", line 74, in put
2014-09-24 10:57:51.615 903 TRACE tempest.test return self.rest_client.put(uri, body, headers)
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/common/rest_client.py", line 221, in put
2014-09-24 10:57:51.615 903 TRACE tempest.test return self.request('PUT', url, extra_headers, headers, body)
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/common/rest_client.py", line 419, in request
2014-09-24 10:57:51.615 903 TRACE tempest.test resp, resp_body)
2014-09-24 10:57:51.615 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/common/rest_client.py", line 515, in _error_checker
2014-09-24 10:57:51.615 903 TRACE tempest.test raise exceptions.ServerFault(message)
2014-09-24 10:57:51.615 903 TRACE tempest.test ServerFault: Got server fault
2014-09-24 10:57:51.615 903 TRACE tempest.test Details: {"NeutronError": "Request Failed: internal server error while processing your request."}
2014-09-24 10:57:51.615 903 TRACE tempest.test
2014-09-24 10:57:51.679 903 INFO tempest.common.rest_client [req-5351f359-df60-4990-9afb-4f013c3de481 None] Request (FloatingIPTestJSON:tearDownClass): 500 GET http://10.204.217.147:9696/v2.0/ports?device_id=456358cf-e38e-4d97-bcb1-a7f0e3d70528 0.058s
2014-09-24 10:57:51.679 903 ERROR tempest.test [-] tearDownClass failed: Got server fault
Details: {"NeutronError": "Request Failed: internal server error while processing your request."}
2014-09-24 10:57:51.679 903 TRACE tempest.test Traceback (most recent call last):
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/test.py", line 81, in decorator
2014-09-24 10:57:51.679 903 TRACE tempest.test cls.tearDownClass()
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/api/network/base.py", line 108, in tearDownClass
2014-09-24 10:57:51.679 903 TRACE tempest.test cls.delete_router(router)
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/api/network/base.py", line 343, in delete_router
2014-09-24 10:57:51.679 903 TRACE tempest.test resp, body = cls.client.list_router_interfaces(router['id'])
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/services/network/json/network_client.py", line 170, in list_router_interfaces
2014-09-24 10:57:51.679 903 TRACE tempest.test resp, body = self.get(uri)
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/services/network/network_client_base.py", line 77, in get
2014-09-24 10:57:51.679 903 TRACE tempest.test return self.rest_client.get(uri, headers)
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/common/rest_client.py", line 212, in get
2014-09-24 10:57:51.679 903 TRACE tempest.test return self.request('GET', url, extra_headers, headers)
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/common/rest_client.py", line 419, in request
2014-09-24 10:57:51.679 903 TRACE tempest.test resp, resp_body)
2014-09-24 10:57:51.679 903 TRACE tempest.test File "/home/ubuntu/jenkins/workspace/Contrail_Tempest/tempest/common/rest_client.py", line 515, in _error_checker
2014-09-24 10:57:51.679 903 TRACE tempest.test raise exceptions.ServerFault(message)
2014-09-24 10:57:51.679 903 TRACE tempest.test ServerFault: Got server fault
2014-09-24 10:57:51.679 903 TRACE tempest.test Details: {"NeutronError": "Request Failed: internal server error while processing your request."}
2014-09-24 10:57:51.679 903 TRACE tempest.test

contrail-api-0.log :
=================
10.204.217.146 - - [2014-09-23 22:28:33] "GET /service-instance/0512dd36-d319-4f6c-8fd5-abad43d02efd?exclude_back_refs=True&exclude_children=True HTTP/1.1" 200 1770 0.018822
ERROR:vnc_cfg_api_server.vnc_cfg_api_server:Exception in REST api handler:
<type 'exceptions.TypeError'>
Python 2.6.6: /usr/bin/python
Tue Sep 23 22:28:33 2014

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in handler_trap_exception(*args=(), **kwargs={'id': '456358cf-e38e-4d97-bcb1-a7f0e3d70528'})
  458 trace = self._generate_rest_api_request_trace()
  459 try:
  460 response = handler(*args, **kwargs)
  461 self._generate_rest_api_response_trace(trace, response)
  462 return response
response undefined
handler = <bound method VncApiServer.logical_router_http_g...i_server.vnc_cfg_api_server.VncApiServer object>>
args = ()
kwargs = {'id': '456358cf-e38e-4d97-bcb1-a7f0e3d70528'}

 /usr/lib/python2.6/site-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py in logical_router_http_get(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, id='456358cf-e38e-4d97-bcb1-a7f0e3d70528')
18277 abort(404, 'No logical-router object found for id %s' %(id))
18278 # common handling for all resource get
18279 (ok, result) = self._get_common(request, id)
18280 if not ok:
18281 (code, msg) = result
ok undefined
result undefined
self = <vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>
self._get_common = <bound method VncApiServer._http_get_common of <...i_server.vnc_cfg_api_server.VncApiServer object>>
global request = <LocalRequest: GET http://0.0.0.0:9100/logical-r...528?exclude_back_refs=True&exclude_children=True>
id = '456358cf-e38e-4d97-bcb1-a7f0e3d70528'

 /usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in _http_get_common(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, request=<LocalRequest: GET http://0.0.0.0:9100/logical-r...528?exclude_back_refs=True&exclude_children=True>, uuid='456358cf-e38e-4d97-bcb1-a7f0e3d70528')
 1166 return ok, result
 1167 else:
 1168 return self._permissions.check_perms_read(request, uuid)
 1169
 1170 return (True, '')
self = <vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>
self._permissions = <vnc_cfg_api_server.vnc_perms.VncPermissions object>
self._permissions.check_perms_read = <bound method VncPermissions.check_perms_read of..._cfg_api_server.vnc_perms.VncPermissions object>>
request = <LocalRequest: GET http://0.0.0.0:9100/logical-r...528?exclude_back_refs=True&exclude_children=True>
uuid = '456358cf-e38e-4d97-bcb1-a7f0e3d70528'

 /usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_perms.py in check_perms_read(self=<vnc_cfg_api_server.vnc_perms.VncPermissions object>, request=<LocalRequest: GET http://0.0.0.0:9100/logical-r...528?exclude_back_refs=True&exclude_children=True>, id='456358cf-e38e-4d97-bcb1-a7f0e3d70528')
  103 return (True, '')
  104
  105 return self.validate_perms(request, id, PERMS_R)
  106 # end check_perms_read
  107
self = <vnc_cfg_api_server.vnc_perms.VncPermissions object>
self.validate_perms = <bound method VncPermissions.validate_perms of <vnc_cfg_api_server.vnc_perms.VncPermissions object>>
request = <LocalRequest: GET http://0.0.0.0:9100/logical-r...528?exclude_back_refs=True&exclude_children=True>
id = '456358cf-e38e-4d97-bcb1-a7f0e3d70528'
global PERMS_R = 4

 /usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_perms.py in validate_perms(self=<vnc_cfg_api_server.vnc_perms.VncPermissions object>, request=<LocalRequest: GET http://0.0.0.0:9100/logical-r...528?exclude_back_refs=True&exclude_children=True>, uuid='456358cf-e38e-4d97-bcb1-a7f0e3d70528', mode=4)
   34 is_admin = 'admin' in [x.lower() for x in roles]
   35
   36 owner = id_perms['permissions']['owner']
   37 group = id_perms['permissions']['group']
   38 perms = id_perms['permissions']['owner_access'] << 6 | \
owner undefined
id_perms = {u'created': None, u'description': None, u'enable': False, u'last_modified': u'2014-09-24T05:28:33.781559', u'permissions': None, u'uuid': None}
<type 'exceptions.TypeError'>: 'NoneType' object is unsubscriptable
    __class__ = <type 'exceptions.TypeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.TypeError object>
    __dict__ = {}
    __doc__ = 'Inappropriate argument type.'
    __format__ = <built-in method __format__ of exceptions.TypeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.TypeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.TypeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.TypeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.TypeError object>
    __init__ = <method-wrapper '__init__' of exceptions.TypeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.TypeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.TypeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.TypeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.TypeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.TypeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.TypeError object>
    __str__ = <method-wrapper '__str__' of exceptions.TypeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.TypeError object>
    args = ("'NoneType' object is unsubscriptable",)
    message = "'NoneType' object is unsubscriptable"

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 460, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 18279, in logical_router_http_get
    (ok, result) = self._get_common(request, id)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1168, in _http_get_common
    return self._permissions.check_perms_read(request, uuid)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_perms.py", line 105, in check_perms_read
    return self.validate_perms(request, id, PERMS_R)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_perms.py", line 36, in validate_perms
    owner = id_perms['permissions']['owner']
TypeError: 'NoneType' object is unsubscriptable

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bottle.py", line 763, in _handle
    return route.call(**args)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1624, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1574, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 460, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 18279, in logical_router_http_get
    (ok, result) = self._get_common(request, id)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1168, in _http_get_common
    return self._permissions.check_perms_read(request, uuid)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_perms.py", line 105, in check_perms_read
    return self.validate_perms(request, id, PERMS_R)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_perms.py", line 36, in validate_perms
    owner = id_perms['permissions']['owner']
TypeError: 'NoneType' object is unsubscriptable
127.0.0.1 - - [2014-09-23 22:28:33] "GET /logical-router/456358cf-e38e-4d97-bcb1-a7f0e3d70528?exclude_back_refs=True&exclude_children=True HTTP/1.1" 500 156 0.012552
<class 'cfgm_common.exceptions.HttpError'>
Python 2.6.6: /usr/bin/python
Tue Sep 23 22:28:33 2014

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_interface.py in plugin_get_router(self=<vnc_openstack.neutron_plugin_interface.NeutronPluginInterface object>, context={u'is_admin': False, u'operation': u'READ', u'roles': [u'_member_'], u'tenant': u'74a38a3ec73943a5add0c6d7181c2e06', u'tenant_id': u'74a38a3ec73943a5add0c6d7181c2e06', u'type': u'router', u'user_id': u'c00dc920daa84f6d957d9998ef89b9da'}, router={u'fields': None, u'filters': None, u'id': u'456358cf-e38e-4d97-bcb1-a7f0e3d70528'})
  754 try:
  755 cfgdb = self._get_user_cfgdb(context)
  756 router_info = cfgdb.router_read(router['id'])
  757 return router_info
  758 except Exception as e:
router_info undefined
cfgdb = <vnc_openstack.neutron_plugin_db.DBInterface object>
cfgdb.router_read = <bound method DBInterface.router_read of <vnc_openstack.neutron_plugin_db.DBInterface object>>
router = {u'fields': None, u'filters': None, u'id': u'456358cf-e38e-4d97-bcb1-a7f0e3d70528'}

 /usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_db.py in router_read(self=<vnc_openstack.neutron_plugin_db.DBInterface object>, rtr_uuid=u'456358cf-e38e-4d97-bcb1-a7f0e3d70528', fields=None)
 3309
 3310 try:
 3311 rtr_obj = self._logical_router_read(rtr_uuid)
 3312 except NoIdError:
 3313 self._raise_contrail_exception(404, RouterNotFound(router_id=rtr_uuid))
rtr_obj undefined
self = <vnc_openstack.neutron_plugin_db.DBInterface object>
self._logical_router_read = <bound method DBInterface._logical_router_read o..._openstack.neutron_plugin_db.DBInterface object>>
rtr_uuid = u'456358cf-e38e-4d97-bcb1-a7f0e3d70528'

 /usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_db.py in _logical_router_read(self=<vnc_openstack.neutron_plugin_db.DBInterface object>, rtr_id=u'456358cf-e38e-4d97-bcb1-a7f0e3d70528', fq_name=None)
  934 raise KeyError
  935 except KeyError:
  936 rtr_obj = self._vnc_lib.logical_router_read(id=rtr_id)
  937 fq_name_str = json.dumps(rtr_obj.get_fq_name())
  938 self._db_cache['vnc_routers'][rtr_id] = rtr_obj
rtr_obj undefined
self = <vnc_openstack.neutron_plugin_db.DBInterface object>
self._vnc_lib = <vnc_api.vnc_api.VncApi object>
self._vnc_lib.logical_router_read = <bound method VncApi.logical_router_read of <vnc_api.vnc_api.VncApi object>>
builtinid = <built-in function id>
rtr_id = u'456358cf-e38e-4d97-bcb1-a7f0e3d70528'

 /usr/lib/python2.6/site-packages/vnc_api/gen/vnc_api_client_gen.py in logical_router_read(self=<vnc_api.vnc_api.VncApi object>, fq_name=None, fq_name_str=None, id=u'456358cf-e38e-4d97-bcb1-a7f0e3d70528', ifmap_id=None, fields=None)
 5640 query_params = {'exclude_back_refs':True,
 5641 'exclude_children':True,}
 5642 content = self._request_server(rest.OP_GET, uri, query_params)
 5643
 5644 obj_dict = json.loads(content)['logical-router']
content undefined
self = <vnc_api.vnc_api.VncApi object>
self._request_server = <bound method VncApi._request_server of <vnc_api.vnc_api.VncApi object>>
global rest = <module 'cfgm_common.rest' from '/usr/lib/python2.6/site-packages/cfgm_common/rest.pyc'>
rest.OP_GET = 2
uri = u'/logical-router/456358cf-e38e-4d97-bcb1-a7f0e3d70528'
query_params = {'exclude_back_refs': True, 'exclude_children': True}

 /usr/lib/python2.6/site-packages/vnc_api/vnc_api.py in _request_server(self=<vnc_api.vnc_api.VncApi object>, op=2, url=u'/logical-router/456358cf-e38e-4d97-bcb1-a7f0e3d70528', data={'exclude_back_refs': True, 'exclude_children': True}, retry_on_error=True, retry_after_authn=False)
  371 continue
  372 else: # Unknown Error
  373 raise HttpError(status, content)
  374 # end while True
  375
global HttpError = <class 'cfgm_common.exceptions.HttpError'>
status = 500
content = u'Internal Server Error'
<class 'cfgm_common.exceptions.HttpError'>: HTTP Status: 500 Content: Internal Server Error
    __class__ = <class 'cfgm_common.exceptions.HttpError'>
    __delattr__ = <method-wrapper '__delattr__' of HttpError object>
    __dict__ = {'content': u'Internal Server Error', 'status_code': 500}
    __doc__ = None
    __format__ = <built-in method __format__ of HttpError object>
    __getattribute__ = <method-wrapper '__getattribute__' of HttpError object>
    __getitem__ = <method-wrapper '__getitem__' of HttpError object>
    __getslice__ = <method-wrapper '__getslice__' of HttpError object>
    __hash__ = <method-wrapper '__hash__' of HttpError object>
    __init__ = <bound method HttpError.__init__ of HttpError()>
    __module__ = 'cfgm_common.exceptions'
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of HttpError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of HttpError object>
    __repr__ = <method-wrapper '__repr__' of HttpError object>
    __setattr__ = <method-wrapper '__setattr__' of HttpError object>
    __setstate__ = <built-in method __setstate__ of HttpError object>
    __sizeof__ = <built-in method __sizeof__ of HttpError object>
    __str__ = <bound method HttpError.__str__ of HttpError()>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of HttpError object>
    __weakref__ = None
    args = ()
    content = u'Internal Server Error'
    message = ''
    status_code = 500

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_interface.py", line 756, in plugin_get_router
    router_info = cfgdb.router_read(router['id'])
  File "/usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_db.py", line 3311, in router_read
    rtr_obj = self._logical_router_read(rtr_uuid)
    rtr_obj = self._logical_router_read(rtr_uuid)
  File "/usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_db.py", line 936, in _logical_router_read
    rtr_obj = self._vnc_lib.logical_router_read(id=rtr_id)
  File "/usr/lib/python2.6/site-packages/vnc_api/gen/vnc_api_client_gen.py", line 5642, in logical_router_read
    content = self._request_server(rest.OP_GET, uri, query_params)
  File "/usr/lib/python2.6/site-packages/vnc_api/vnc_api.py", line 373, in _request_server
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bottle.py", line 763, in _handle
    return route.call(**args)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1624, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1574, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_interface.py", line 880, in plugin_http_post_router
    return self.plugin_get_router(context, router)
  File "/usr/lib/python2.6/site-packages/vnc_openstack/neutron_plugin_interface.py", line 760, in plugin_get_router
    raise e
HttpError: HTTP Status: 500 Content: Internal Server Error
10.204.217.147 - - [2014-09-23 22:28:33] "POST /neutron/router HTTP/1.1" 500 156 0.029974
+++ read 5d9a646e-6115-4321-a5ed-f881a08023ab admin=yes, mode=444 mask=770 admin/"admin,KeystoneServiceAdmin,KeystoneAdmin", perms=777 (admin/admin)

-==========
neutron/server.log:

2014-09-23 22:28:33.791 INFO [neutron.wsgi] 10.204.219.237,10.204.217.147 - - [23/Sep/2014 22:28:33] "POST /v2.0/routers HTTP/1.1" 201 568 0.539409

2014-09-23 22:28:33.799 INFO [neutron.wsgi] (13661) accepted ('10.204.217.147', 55426)

2014-09-23 22:28:33.808 INFO [urllib3.connectionpool] Starting new HTTP connection (1): 10.204.217.147
2014-09-23 22:28:33.841 ERROR [neutron.api.v2.resource] add_router_interface failed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 87, in resource
    result = method(request=request, **args)
  File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 180, in _handle_action
    resource = self._item(request, id, True)
  File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 267, in _item
    obj = obj_getter(request.context, id, **kwargs)
  File "/usr/lib/python2.6/site-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 553, in get_router
    return self._get_resource('router', context, router_id, fields)
  File "/usr/lib/python2.6/site-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 291, in _get_resource
    fields=fields, obj_name=res_type)
  File "/usr/lib/python2.6/site-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 251, in _transform_response
    self._raise_contrail_error(status_code, info, obj_name)
  File "/usr/lib/python2.6/site-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 257, in _raise_contrail_error
    error_class = CONTRAIL_EXCEPTION_MAP[status_code]
KeyError: 500
2014-09-23 22:28:33.843 INFO [neutron.wsgi] 10.204.219.237,10.204.217.147 - - [23/Sep/2014 22:28:33] "PUT /v2.0/routers/456358cf-e38e-4d97-bcb1-a7f0e3d70528/add_router_interface HTTP/1.1" 500 315 0.043026

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/3223
Committed: http://github.org/Juniper/contrail-controller/commit/f1342cfddd940e85c08d763d935db12f7aa66d91
Submitter: Zuul
Branch: R1.10

commit f1342cfddd940e85c08d763d935db12f7aa66d91
Author: Sachin Bansal <email address hidden>
Date: Fri Sep 26 15:48:25 2014 -0700

During router create, if we are also updating external gateway, we were updating the logical_router object without reading it. This could clobber the id_perms on the server. We should read before updating the object. Ideally, we should protect against this in api server.

Change-Id: I529b7f9e0b2ab3b635809af8678addc34467422c
Closes-Bug: 1373257

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/3224
Committed: http://github.org/Juniper/contrail-controller/commit/0f245ac6ea7f607e462478d3a4136f7aef7286ce
Submitter: Zuul
Branch: master

commit 0f245ac6ea7f607e462478d3a4136f7aef7286ce
Author: Sachin Bansal <email address hidden>
Date: Fri Sep 26 15:48:25 2014 -0700

During router create, if we are also updating external gateway, we were updating the logical_router object without reading it. This could clobber the id_perms on the server. We should read before updating the object. Ideally, we should protect against this in api server.

Change-Id: I529b7f9e0b2ab3b635809af8678addc34467422c
Closes-Bug: 1373257
(cherry picked from commit f1342cfddd940e85c08d763d935db12f7aa66d91)