"Duplicate default security group was not created" from neutron-server

[Mikhail Chernik]

During NeutronSecGroupPlugin.create_and_delete_secgroups Rally scenario we have found following problem:

Rally log:
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 79, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/rally/plugins/neutron_security_groups.py", line 119, in create_and_delete_secgroups
    security_group_count)
  File "/opt/rally/plugins/neutron_security_groups.py", line 66, in _neutron_create_security_groups
    sec_groups = self.clients("neutron").create_security_group(body)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 716, in create_security_group
    return self.post(self.security_groups_path, body=body)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in post
    headers=headers, params=params)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20
    status_code=status_code)
InternalServerError: Request Failed: internal server error while processing your request.

Neutron log:
http://paste.openstack.org/show/475467/

Environment:
198 computes, 3 controllers

api: '1.0'
astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
auth_required: true
build_id: '301'
build_number: '301'
feature_groups:
- mirantis
fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
release: '7.0'

[Ilya Shakhat (shakhat)]

the root cause - db_base_plugin_v2._create_bulk () contains transaction that is not wrapped by retry_on_request

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/12778

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/12778
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: d29c537c2fafacf7e9e9057298bedd978bc5d65e
Author: Oleg Bondarev <email address hidden>
Date: Tue Oct 13 12:41:47 2015

Wrap create_security_group_bulk with db retry on request

Commit c77c0b40383f205223a50817a750e34d43e56856 added RetryRequest
to _ensure_default_security_group(), however one case where it is
called from within a transaction was missed. This patch fixes it.

Closes-Bug: #1503294
Change-Id: I951222368073b046c394d2c88ce5cb5188edfbea

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13299

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Eugene Nikanorov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/13299
Reason: Since liberty, DB retries are and API level

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/13606

[Mikhail Chernik (mchernik)]

Verified on MOS 7.0 GA (build 301), updated to 7.0-mu-1, fixed.
Test scenario and results: http://paste.openstack.org/show/478516/

[Mikhail Chernik (mchernik)]

Verified in scale lab on 200-nodes env, MOS 8.0 ISO 482, fixed.

Network configuration Neutron VLAN+DVR