Bug #1382064 “Failure to allocate tunnel id when creating networ...” : Series kilo : Bugs : neutron

Bug Description

When multiple networks are created concurrently, the following trace is observed:

WARNING neutron.plugins.ml2.drivers.helpers [req-34103ce8-b6d0-459b-9707-a24e369cf9de None] Allocate gre segment from pool failed after 10 failed attempts
DEBUG neutron.context [req-2995f877-e3e6-4b32-bdae-da6295e492a1 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /usr/lib/python2.7/dist-packages/neutron/context.py:83
DEBUG neutron.plugins.ml2.drivers.helpers [req-3541998d-44df-468f-b65b-36504e893dfb None] Allocate gre segment from pool, attempt 1 failed with segment {'gre_id': 300L} allocate_partially_specified_segment /usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/helpers.py:138
DEBUG neutron.context [req-6dcfb91d-2c5b-4e4f-9d81-55ba381ad232 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /usr/lib/python2.7/dist-packages/neutron/context.py:83
ERROR neutron.api.v2.resource [req-34103ce8-b6d0-459b-9707-a24e369cf9de None] create failed
TRACE neutron.api.v2.resource Traceback (most recent call last):
TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 87, in resource
TRACE neutron.api.v2.resource result = method(request=request, **args)
TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 448, in create
TRACE neutron.api.v2.resource obj = obj_creator(request.context, **kwargs)
TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 497, in create_network
TRACE neutron.api.v2.resource tenant_id)
TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 160, in create_network_segments
TRACE neutron.api.v2.resource segment = self.allocate_tenant_segment(session)
TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 189, in allocate_tenant_segment
TRACE neutron.api.v2.resource segment = driver.obj.allocate_tenant_segment(session)
TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 115, in allocate_tenant_segment
TRACE neutron.api.v2.resource alloc = self.allocate_partially_specified_segment(session)
TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/helpers.py", line 143, in allocate_partially_specified_segment
TRACE neutron.api.v2.resource raise exc.NoNetworkFoundInMaximumAllowedAttempts()
TRACE neutron.api.v2.resource NoNetworkFoundInMaximumAllowedAttempts: Unable to create the network. No available network found in maximum allowed attempts.
TRACE neutron.api.v2.resource

Additional conditions: multiserver deployment and mysql.

Tags: Edit Tag help

Eugene Nikanorov (enikanorov) on 2014-10-16

Changed in neutron:
assignee:	nobody → Eugene Nikanorov (enikanorov)
importance:	Undecided → High

Eugene Nikanorov (enikanorov) on 2014-10-17

tags:

added: ml2

OpenStack Infra (hudson-openstack) wrote on 2014-10-17: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/129288

Changed in neutron:
status:	New → In Progress

Eugene Nikanorov (enikanorov) on 2014-10-30

tags:

added: juno-backport-potential

Jay Pipes (jaypipes) wrote on 2014-11-19:

Is this using MySQL Galera as the backend database server? And if so, is the Galera setup using only a single writer node?

Eugene Nikanorov (enikanorov) wrote on 2014-11-19:

Yes, it's with galera, single writer node.
But that doesn't matter. The issue would be the same with single mysql backend.

Oleg Bondarev (obondarev) wrote on 2014-12-10:

Seems following patch reveals the problem: https://review.openstack.org/#/c/140493/

OpenStack Infra (hudson-openstack) wrote on 2014-12-12:

Fix proposed to branch: master
Review: https://review.openstack.org/141453

Changed in neutron:
assignee:	Eugene Nikanorov (enikanorov) → Ed Bak (ed-bak2)

OpenStack Infra (hudson-openstack) on 2014-12-15

Changed in neutron:
assignee:	Ed Bak (ed-bak2) → Eugene Nikanorov (enikanorov)

OpenStack Infra (hudson-openstack) on 2015-01-12

Changed in neutron:
assignee:	Eugene Nikanorov (enikanorov) → Russell Bryant (russellb)

OpenStack Infra (hudson-openstack) on 2015-01-14

Changed in neutron:
assignee:	Russell Bryant (russellb) → Eugene Nikanorov (enikanorov)

OpenStack Infra (hudson-openstack) wrote on 2015-01-16: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/129288
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6617f8fccc8d99520a87cd84a598c4f9a1a43761
Submitter: Jenkins
Branch: master

commit 6617f8fccc8d99520a87cd84a598c4f9a1a43761
Author: Eugene Nikanorov <email address hidden>
Date: Mon Nov 17 11:00:49 2014 +0400

Change transaction isolation so retry logic could work properly

    Lower isolation level from REPEATABLE READ to READ COMMITTED for
    transaction that is used to create a network.
    This allows retry logic to see changes done in other connections
    while doing the same query.
    Perform that only for mysql db backend.

Change-Id: I6b9d9212c37fe028566e0df4a3dfa51f284ce6e9
Closes-Bug: #1382064

Changed in neutron:
status:	In Progress → Fix Committed

OpenStack Infra (hudson-openstack) wrote on 2015-01-18: Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/148083

OpenStack Infra (hudson-openstack) wrote on 2015-01-18: Change abandoned on neutron (stable/juno)

Change abandoned by Ryan Tidwell (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/148083
Reason: Cherry-picked change reverted on master

OpenStack Infra (hudson-openstack) wrote on 2015-01-19: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/148339

YAMAMOTO Takashi (yamamoto) wrote on 2015-01-20:

#10

copy-and-paste from https://review.openstack.org/#/c/129288/

enikanorov
12-16 19:51
Patch Set 8:
Ok, I'll add this to the bug.
So my current understanding is the following:
when REPEATABLE READ is used, each distinct query in the transaction creates a snapshot on DB backend side that is used when going along the query or when issuing the same query in that transaction. When READ COMMITTED is used, each fetch reaches table directly and that, IMO, increases possible contention that leads to much more frequent deadlocks. Previous version of the patch (that set isolation level for each connection globally) demonstrated it in the gates. But that's only my guess of technical reasons, I can be wrong here.
Context manager is nice suggestion, thanks.

OpenStack Infra (hudson-openstack) wrote on 2015-01-22:

#11

Fix proposed to branch: master
Review: https://review.openstack.org/149261

OpenStack Infra (hudson-openstack) wrote on 2015-01-28: Change abandoned on neutron (master)

#12

Change abandoned by enikanorov (<email address hidden>) on branch: master
Review: https://review.openstack.org/148339

Thierry Carrez (ttx) on 2015-02-05

Changed in neutron:
milestone:	none → kilo-2
status:	Fix Committed → Fix Released

Eugene Nikanorov (enikanorov) on 2015-02-20

Changed in neutron:
status:	Fix Released → In Progress
milestone:	kilo-2 → kilo-3

OpenStack Infra (hudson-openstack) wrote on 2015-02-24:

#13

Change abandoned by Ed Bak (<email address hidden>) on branch: master
Review: https://review.openstack.org/141453
Reason: Not needed any longer. https://review.openstack.org/#/c/129288 solves this problem.

OpenStack Infra (hudson-openstack) on 2015-03-13

Changed in neutron:
assignee:	Eugene Nikanorov (enikanorov) → Assaf Muller (amuller)

OpenStack Infra (hudson-openstack) wrote on 2015-03-13: Fix merged to neutron (master)

#14

Reviewed: https://review.openstack.org/149261
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5dbb34b56fc42d9c68bf6647910a437a2ad6b29e
Submitter: Jenkins
Branch: master

commit 5dbb34b56fc42d9c68bf6647910a437a2ad6b29e
Author: Eugene Nikanorov <email address hidden>
Date: Thu Jan 22 15:54:29 2015 +0300

Refactor retry mechanism used in some DB operations

    Use oslo_db helper that will allow to restart the whole
    transaction in case it needs a certain operation to be repeated.
    This is a workaround for the REPEATABLE READ problem where
    retrying logic will not work because queries inside a transation
    will not see updates made by other transactions.
    So, run every attempt in a separate transaction.

Change-Id: I68f9ae8019879725df58f5da2c83bb699a548255
Closes-Bug: #1382064

Changed in neutron:
status:	In Progress → Fix Committed

Eugene Nikanorov (enikanorov) on 2015-03-13

Changed in neutron:
assignee:	Assaf Muller (amuller) → Eugene Nikanorov (enikanorov)

Thierry Carrez (ttx) on 2015-03-19

Changed in neutron:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in neutron:
milestone:	kilo-3 → 2015.1.0

OpenStack Infra (hudson-openstack) wrote on 2015-05-14: Related fix proposed to neutron (master)

#15

Related fix proposed to branch: master
Review: https://review.openstack.org/182901

OpenStack Infra (hudson-openstack) wrote on 2015-05-14: Related fix merged to neutron (master)

#16

Reviewed: https://review.openstack.org/182901
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1d9fd2aec00cb85034e5a23cc1beac33c74e0110
Submitter: Jenkins
Branch: master

commit 1d9fd2aec00cb85034e5a23cc1beac33c74e0110
Author: Eugene Nikanorov <email address hidden>
Date: Mon May 11 01:34:35 2015 +0400

Randomize tunnel id query to avoid contention

    When networks are created rapidly, neutron-servers compete
    for segmentation ids which creates too much contention and
    may lead to inability to choose available id in hardcoded amount
    of attempts (11)
    Randomize tunnel id selection so that condition is not hit.

    Change-Id: I7068f90fe4927e6e693f8a62cb704213b2da2920
    Related-Bug: #1382064
    Closes-Bug: #1454434

OpenStack Infra (hudson-openstack) wrote on 2015-05-22: Related fix proposed to neutron (neutron-pecan)

#17

Related fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

OpenStack Infra (hudson-openstack) wrote on 2015-06-06: Related fix proposed to neutron (stable/kilo)

#18

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/188994

OpenStack Infra (hudson-openstack) wrote on 2015-06-09: Related fix merged to neutron (stable/kilo)

#19

Reviewed: https://review.openstack.org/188994
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9bc323316a27676859280c9ee413a791c386ac64
Submitter: Jenkins
Branch: stable/kilo

commit 9bc323316a27676859280c9ee413a791c386ac64
Author: Eugene Nikanorov <email address hidden>
Date: Mon May 11 01:34:35 2015 +0400

Randomize tunnel id query to avoid contention

    Change-Id: I7068f90fe4927e6e693f8a62cb704213b2da2920
    Related-Bug: #1382064
    Closes-Bug: #1454434
    (cherry picked from commit 1d9fd2aec00cb85034e5a23cc1beac33c74e0110)

tags:

added: in-stable-kilo

OpenStack Infra (hudson-openstack) wrote on 2015-06-25: Related fix proposed to neutron (stable/juno)

#20

Related fix proposed to branch: stable/juno
Review: https://review.openstack.org/195792

OpenStack Infra (hudson-openstack) wrote on 2015-06-25: Change abandoned on neutron (stable/juno)

#21

Change abandoned by stephen-ma (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/195792

OpenStack Infra (hudson-openstack) wrote on 2015-06-25: Fix proposed to neutron (stable/juno)

#22

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/195797

OpenStack Infra (hudson-openstack) wrote on 2015-07-10: Fix merged to neutron (stable/juno)

#23

Reviewed: https://review.openstack.org/195797
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4cd1b58c8c4ae2a9da31afc1a87647003d6ac128
Submitter: Jenkins
Branch: stable/juno

commit 4cd1b58c8c4ae2a9da31afc1a87647003d6ac128
Author: Eugene Nikanorov <email address hidden>
Date: Thu Jan 22 15:54:29 2015 +0300

Refactor retry mechanism used in some DB operations

    Conflicts:
            neutron/plugins/ml2/drivers/helpers.py
            neutron/plugins/ml2/plugin.py
            neutron/tests/unit/ml2/test_ml2_plugin.py

    (cherry picked from commit 5dbb34b56fc42d9c68bf6647910a437a2ad6b29e)
    Change-Id: I68f9ae8019879725df58f5da2c83bb699a548255
    Closes-Bug: #1382064

tags:

added: in-stable-juno

OpenStack Infra (hudson-openstack) wrote on 2015-07-13: Related fix merged to neutron (stable/juno)

#24

Reviewed: https://review.openstack.org/195792
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=07d3d4401f34a19b23738296c189122eeef9150f
Submitter: Jenkins
Branch: stable/juno

commit 07d3d4401f34a19b23738296c189122eeef9150f
Author: Eugene Nikanorov <email address hidden>
Date: Mon May 11 01:34:35 2015 +0400

Randomize tunnel id query to avoid contention

    (cherry picked from commit 1d9fd2aec00cb85034e5a23cc1beac33c74e0110)
    Conflicts:
            neutron/plugins/ml2/drivers/helpers.py

    Related-Bug: #1382064
    Closes-Bug: #1454434
    Change-Id: I7068f90fe4927e6e693f8a62cb704213b2da2920

neutron

Failure to allocate tunnel id when creating networks concurrently

Bug Description

Duplicates of this bug

Other bug subscribers