Race condition on creating docker network
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Zun |
Fix Released
|
High
|
hongbin |
Bug Description
Currently, we lazily create docker network. At runtime, we check if the network is already there. If not, create the network (see _get_or_
This has a race condition:
1. Request A checks for network with name N but couldn't find one
2. Request B checks for network with name N but couldn't find one
3. Request A create a network with name N
4. Request B create another network with name N
Then, there are two duplicated network created. This will cause failure in the system. Such failure occasionally happened in the gate:
To resolve the problem, there are several options. One option is to record each docker network in database and impose a unique constraints on name to avoid duplication. There should be other options available. Choose the optimal solution if any.
Changed in zun: | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in zun: | |
assignee: | nobody → Rajat Sharma (tajar29) |
Changed in zun: | |
assignee: | Rajat Sharma (tajar29) → nobody |
I think, 'oslo_concurren cy.lockutils' could help this problem.
With using the '@lockutils' decorator, we can make a function called once at the time across threads.
(Other threads will be waiting for unlocking)
code could look like this:
from oslo_concurrency import lockutils
@lockutils. synchronized( 'get_or_ create_ docker_ network' ) create_ docker_ network( self, context, network_api,
neutron_ net_id) :
def _get_or_
...