[Mainline-2746] Neutron Exception while resource access

[Suresh]

Topo:
=====
nodel11

 This node has 6 VM named nodel11-vm1 to nodel11-vm6.

Description:
===========
1. Trying out different combinations of roledef in vm1, vm2 & vm3. vm4, vm5 & vm6 are used as compute nodes.

2. Neutron exception seen while resource access. The exception is seen in different places. Once seen when trying to create subnet. Other time seen while resource access after creating subnet.

2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource [req-23f05999-fac9-45d7-9478-92f7a9b133eb 63c90551b3424176a6d5ba1d1976c71c 410ae7591ec345ab85f1e140e7768bb0 - - -] create failed
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 410, in create
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in exit
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource self.force_reraise()
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 521, in _create
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource obj = do_create(body)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 503, in do_create
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in exit
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource self.force_reraise()
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 496, in do_create
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 276, in create_subnet
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource subnet_created = self._create_resource('subnet', context, subnet)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 255, in _create_resource
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource obj_name=res_type)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 237, in _transform_response
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource plugin_base._raise_contrail_error(info, obj_name)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 111, in _raise_contrail_error
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource raise exc.NeutronException(**info)
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource NeutronException: An unknown exception occurred.
2016-10-21 18:26:59.629 27696 ERROR neutron.api.v2.resource

Roledef:
========
env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'compute': [host4, host5, host6],
    'database': [host1, host2, host3],
    'collector': [host1, host2, host3],
    'control': [host2, host3],
    'webui': [host1, host2, host3],
    'openstack': [host3],
    'cfgm': [host1, host2],
    'build': [host_build],
}

[Hampapur Ajay (hajay)]

Can you provide contrail-api*.log in /var/log/contrail from all controllers? Is this observed in latest image too?