[fuel-devops 3.0] AddressPool._safe_create_network does not protect from race condition

Bug #1598141 reported by Anton Studenov
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Medium
Anton Studenov

Bug Description

When multiple environments are being created at the same time, there is a race condition
for AddressPool networks.

Steps to reproduce:
  Run two dos.py create-env simultaneously

Expected result:
  Environments are created properly

Actual result:
  Second environment gets error stacktrace:

File "/home/tohin/.virtualenvs/devops3.0.0/bin/dos.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/tohin/Code/fuel-devops/bin/dos.py", line 22, in <module>
    main()
  File "/home/tohin/Code/fuel-devops/devops/shell.py", line 628, in main
    Shell(args).execute()
  File "/home/tohin/Code/fuel-devops/devops/shell.py", line 55, in execute
    self.commands.get(self.params.command)(self)
  File "/home/tohin/Code/fuel-devops/devops/shell.py", line 219, in do_create_env
    self._create_env_from_config(config)
  File "/home/tohin/Code/fuel-devops/devops/shell.py", line 228, in _create_env_from_config
    self.env = Environment.create_environment(config)
  File "/home/tohin/Code/fuel-devops/devops/models/environment.py", line 314, in create_environment
    environment.add_address_pools(address_pools)
  File "/home/tohin/Code/fuel-devops/devops/models/environment.py", line 121, in add_address_pools
    **data.get('params', {})
  File "/home/tohin/Code/fuel-devops/devops/models/environment.py", line 138, in add_address_pool
    **params
  File "/home/tohin/Code/fuel-devops/devops/models/network.py", line 248, in address_pool_create
    **params
  File "/home/tohin/.virtualenvs/devops3.0.0/local/lib/python2.7/site-packages/Django-1.8.13-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/home/tohin/Code/fuel-devops/devops/models/network.py", line 230, in _safe_create_network
    transaction.rollback()
  File "/home/tohin/.virtualenvs/devops3.0.0/local/lib/python2.7/site-packages/Django-1.8.13-py2.7.egg/django/db/transaction.py", line 49, in rollback
    get_connection(using).rollback()
  File "/home/tohin/.virtualenvs/devops3.0.0/local/lib/python2.7/site-packages/Django-1.8.13-py2.7.egg/django/db/backends/base/base.py", line 182, in rollback
    self.validate_no_atomic_block()
  File "/home/tohin/.virtualenvs/devops3.0.0/local/lib/python2.7/site-packages/Django-1.8.13-py2.7.egg/django/db/backends/base/base.py", line 322, in validate_no_atomic_block
    "This is forbidden when an 'atomic' block is active.")
django.db.transaction.TransactionManagementError: This is forbidden when an 'atomic' block is active.

Workaround:
  Create environments consecutively

Impact:
  May affect CI

Tags: area-qa
Changed in fuel:
assignee: nobody → Anton Studenov (astudenov)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-devops (master)

Fix proposed to branch: master
Review: https://review.openstack.org/336566

Dmitry Klenov (dklenov)
Changed in fuel:
milestone: none → 10.0
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-devops (master)

Reviewed: https://review.openstack.org/336566
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=59742856f8071a3482d013910db7fb1ecfc6bda3
Submitter: Jenkins
Branch: master

commit 59742856f8071a3482d013910db7fb1ecfc6bda3
Author: Anton Studenov <email address hidden>
Date: Fri Jul 1 16:42:36 2016 +0300

    Fix AddressPool._safe_create_network

    - transaction.atomic was used improperly.
      transaction.rollback() will be called at atomic.__exit__
      in case of error
    - Added DevopsError for duplicate names and networks
    - Added unit tests

    Closes-Bug: #1598141
    Closes-Bug: #1597025
    Change-Id: I024c29761048447446a09ac4439cca66e60b11d4

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-devops (release/2.9)

Fix proposed to branch: release/2.9
Review: https://review.openstack.org/350400

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-devops (release/2.9)

Reviewed: https://review.openstack.org/350400
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=fa073f615de9d247960b97f725d5cc543e480aaa
Submitter: Jenkins
Branch: release/2.9

commit fa073f615de9d247960b97f725d5cc543e480aaa
Author: Alexey Stepanov <email address hidden>
Date: Wed Aug 3 10:05:58 2016 +0300

    Fix AddressPool._safe_create_network

    - transaction.atomic was used improperly.
      transaction.rollback() will be called at atomic.__exit__
      in case of error
    - Added DevopsError for duplicate names and networks
    - Added unit tests

    Closes-Bug: #1598141
    Closes-Bug: #1597025
    Closes-bug: #1607342
    Change-Id: I024c29761048447446a09ac4439cca66e60b11d4

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.