concurrent disassociate floating ip calls leads to 500 error

Bug #1092761 reported by Vish Ishaya
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Vish Ishaya
Folsom
Fix Released
Medium
Vish Ishaya

Bug Description

Disassocating the same floating ip concurrently will cause a 500 error as shown here (stable/folsom):

2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 642, in _disassociate_floating_ip
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp fixed_address = self.db.floating_ip_disassociate(context, address)
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 309, in floating_ip_disassociate
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp return IMPL.floating_ip_disassociate(context, address)
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 129, in wrapper
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 851, in floating_ip_disassociate
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp floating_ip_ref['fixed_ip_id'])
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 129, in wrapper
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in fixed_ip_get
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp raise exception.FixedIpNotFound(id=id)
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp FixedIpNotFound: No fixed IP associated with id None.
2012-12-21 00:59:13 8428 TRACE nova.openstack.common.rpc.amqp

It should just ignore the remaining code if it cannot find the fixed ip.

Changed in nova:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Vish Ishaya (vishvananda)
Revision history for this message
Vish Ishaya (vishvananda) wrote :

This was partially fixed in trunk by:
    https://review.openstack.org/#/c/14928/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/18525

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/18526

Mark McLoughlin (markmc)
tags: added: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/18525
Committed: http://github.com/openstack/nova/commit/881a93473c32a7c7e23a8e6dcede8394053408c6
Submitter: Jenkins
Branch: master

commit 881a93473c32a7c7e23a8e6dcede8394053408c6
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Dec 20 20:13:37 2012 -0800

    Eliminate race conditions in floating association

    This makes associating and disassociating floating ips atomic and
    idempotent. This means multiple concurrent messages will not leave
    behind iptables rules and concurrent request will not cause odd
    failures.

    Fixes bug 1092762 and bug 1092761.

    Change-Id: Idbcad6c1d2a3d4881cf7180b848ed3844fac4054

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Mark McLoughlin (markmc)
tags: removed: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/18526
Committed: http://github.com/openstack/nova/commit/5a6681222999873f0df9816125fe9888498d91c2
Submitter: Jenkins
Branch: stable/folsom

commit 5a6681222999873f0df9816125fe9888498d91c2
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Dec 20 20:13:37 2012 -0800

    Eliminate race conditions in floating association

    This makes associating and disassociating floating ips atomic and
    idempotent. This means multiple concurrent messages will not leave
    behind iptables rules and concurrent request will not cause odd
    failures.

    Fixes bug 1092762 and bug 1092761.

    Change-Id: Idbcad6c1d2a3d4881cf7180b848ed3844fac4054
    (cherry picked from commit 881a93473c32a7c7e23a8e6dcede8394053408c6)

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.