Duplicate forward rules in nova-network

Bug #1133527 reported by Vish Ishaya
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Vish Ishaya

Bug Description

Under heavy load in one of our environments we have seen duplicate forward rules appearing sometimes. It looks like this is due to odd failure conditions, such as a node losing its connection to rabbit or mysql. Even though the duplicates are due to failures, it would be best to ensure that duplicate forward rules cannot be created.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

This is definitely duplicates in the actual in-memory rules. Verified by logging in via eventlet backdoor and dumping the in memory rules:

http://paste.openstack.org/show/32491/

Changed in nova:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/23010

Changed in nova:
assignee: nobody → Vish Ishaya (vishvananda)
status: New → In Progress
Changed in nova:
milestone: none → grizzly-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/23010
Committed: http://github.com/openstack/nova/commit/80e78c45849ffd1b05f386b5e7aff69b083e0d79
Submitter: Jenkins
Branch: master

commit 80e78c45849ffd1b05f386b5e7aff69b083e0d79
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Feb 26 10:41:43 2013 -0800

    Make sure no duplicate forward rules can exist.

    Some failure conditions can cause floating forwards to not be properly
    removed. We know that duplicate floating rules will never work, so
    even if we have had other failures there is no need to create duplicate
    forward rules. This patch fixes the issue by removing all existing
    forwards for a floating ip before adding new ones. It adds a method
    to IptablesTable to remove rules by regex and includes tests for new
    method and the specific case of a duplicate forward.

    Fixes bug 1133527

    Change-Id: Iaf5df8406fc49e92bac328cd81ffda1969b77d68

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.