Concurrency in sec-group creation with service_instance module

Bug #1342122 reported by Valeriy Ponomaryov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
New
Low
Julia Varlamova

Bug Description

Environment: multibackend (two generic drivers).
Description: There are some problem with locks for service_instance, rarely, but concurrency appears creating security_group, that should be unique.

from tempest job http://logs.openstack.org/11/103911/18/check/gate-manila-tempest-dsvm-neutron-multibackend/4812315/logs/screen-m-shr.txt.gz#_2014-07-15_13_17_20_461 :

2014-07-15 13:17:20.461 3570 ERROR oslo.messaging.rpc.dispatcher [req-7184c69e-e6ef-47c9-9938-f9ae82314eb3 ] Exception during message handling: Security group default already exists for project 37e0dbd0096642c298dff4b1851b9010. (HTTP 400) (Request-ID: req-6dab8966-bcd9-48d2-85be-72331ad61546)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 188, in create_share
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher {'status': 'error'})
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/openstack/common/excutils.py", line 82, in __exit__
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 182, in create_share
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher context, share_network_id, share_id)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 139, in _provide_share_server_for_share
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher return _provide_share_server_for_share()
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/openstack/common/lockutils.py", line 324, in inner
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 128, in _provide_share_server_for_share
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher share_server = self._setup_server(context, share_network)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 414, in _setup_server
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher self.network_api.deallocate_network(context, share_network)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/openstack/common/excutils.py", line 82, in __exit__
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 402, in _setup_server
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher metadata=metadata)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 501, in setup_server
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher network_info['neutron_subnet_id'],
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/service_instance.py", line 282, in set_up_service_instance
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher neutron_subnet_id)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/service_instance.py", line 372, in _create_service_instance
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher nics=[{'port-id': port_id}])
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/compute/nova.py", line 172, in server_create
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher min_count=instance_count, admin_pass=admin_pass)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/python-novaclient/novaclient/v1_1/servers.py", line 882, in create
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher **boot_kwargs)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/python-novaclient/novaclient/v1_1/servers.py", line 545, in _boot
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher return_raw=return_raw, **kwargs)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/python-novaclient/novaclient/base.py", line 100, in _create
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher _resp, body = self.api.client.post(url, body=body)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/python-novaclient/novaclient/client.py", line 485, in post
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher return self._cs_request(url, 'POST', **kwargs)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/python-novaclient/novaclient/client.py", line 459, in _cs_request
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher **kwargs)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/python-novaclient/novaclient/client.py", line 441, in _time_request
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher resp, body = self.request(url, method, **kwargs)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/python-novaclient/novaclient/client.py", line 435, in request
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher raise exceptions.from_response(resp, body, url, method)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher BadRequest: Security group default already exists for project 37e0dbd0096642c298dff4b1851b9010. (HTTP 400) (Request-ID: req-6dab8966-bcd9-48d2-85be-72331ad61546)
2014-07-15 13:17:20.461 3570 TRACE oslo.messaging.rpc.dispatcher
2014-07-15 13:17:20.477 3570 ERROR oslo.messaging._drivers.common [req-7184c69e-e6ef-47c9-9938-f9ae82314eb3 ] Returning exception Security group default already exists for project 37e0dbd0096642c298dff4b1851b9010. (HTTP 400) (Request-ID: req-6dab8966-bcd9-48d2-85be-72331ad61546) to caller
2014-07-15 13:17:20.477 3570 ERROR oslo.messaging._drivers.common [req-7184c69e-e6ef-47c9-9938-f9ae82314eb3 ] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/new/manila/manila/share/manager.py", line 188, in create_share\n {\'status\': \'error\'})\n', ' File "/opt/stack/new/manila/manila/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/new/manila/manila/share/manager.py", line 182, in create_share\n context, share_network_id, share_id)\n', ' File "/opt/stack/new/manila/manila/share/manager.py", line 139, in _provide_share_server_for_share\n return _provide_share_server_for_share()\n', ' File "/opt/stack/new/manila/manila/openstack/common/lockutils.py", line 324, in inner\n return f(*args, **kwargs)\n', ' File "/opt/stack/new/manila/manila/share/manager.py", line 128, in _provide_share_server_for_share\n share_server = self._setup_server(context, share_network)\n', ' File "/opt/stack/new/manila/manila/share/manager.py", line 414, in _setup_server\n self.network_api.deallocate_network(context, share_network)\n', ' File "/opt/stack/new/manila/manila/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/new/manila/manila/share/manager.py", line 402, in _setup_server\n metadata=metadata)\n', ' File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 501, in setup_server\n network_info[\'neutron_subnet_id\'],\n', ' File "/opt/stack/new/manila/manila/share/drivers/service_instance.py", line 282, in set_up_service_instance\n neutron_subnet_id)\n', ' File "/opt/stack/new/manila/manila/share/drivers/service_instance.py", line 372, in _create_service_instance\n nics=[{\'port-id\': port_id}])\n', ' File "/opt/stack/new/manila/manila/compute/nova.py", line 172, in server_create\n min_count=instance_count, admin_pass=admin_pass)\n', ' File "/opt/stack/new/python-novaclient/novaclient/v1_1/servers.py", line 882, in create\n **boot_kwargs)\n', ' File "/opt/stack/new/python-novaclient/novaclient/v1_1/servers.py", line 545, in _boot\n return_raw=return_raw, **kwargs)\n', ' File "/opt/stack/new/python-novaclient/novaclient/base.py", line 100, in _create\n _resp, body = self.api.client.post(url, body=body)\n', ' File "/opt/stack/new/python-novaclient/novaclient/client.py", line 485, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/opt/stack/new/python-novaclient/novaclient/client.py", line 459, in _cs_request\n **kwargs)\n', ' File "/opt/stack/new/python-novaclient/novaclient/client.py", line 441, in _time_request\n resp, body = self.request(url, method, **kwargs)\n', ' File "/opt/stack/new/python-novaclient/novaclient/client.py", line 435, in request\n raise exceptions.from_response(resp, body, url, method)\n', 'BadRequest: Security group default already exists for project 37e0dbd0096642c298dff4b1851b9010. (HTTP 400) (Request-ID: req-6dab8966-bcd9-48d2-85be-72331ad61546)\n']

description: updated
Changed in manila:
importance: Undecided → Medium
Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :

Rare error, setting importance to 'low'.

Changed in manila:
importance: Medium → Low
Changed in manila:
milestone: none → kilo-1
Changed in manila:
assignee: nobody → Julia Varlamova (jvarlamova)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.