Utils.is_neutron can create race conditions

Bug #1274992 reported by Sean M. Collins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Russell Bryant

Bug Description

It appears that using the utils.is_neutron() function can create race conditions, if not used carefully. Review #56381 was attempting to check the presence of Neutron, in order to determine if hairpinning should be enabled on the bridge that Nova creates. Only after creating a new configuration flag in Nova.conf and checking the setting, were the unit tests passing and the race condition resolved.

My hunch is that the global variable that utils.is_neutron( ) creates is the root cause of the issue.

https://review.openstack.org/#/c/56381/

Revision history for this message
Matt Riedemann (mriedem) wrote :

Remember that the tests are run in parallel, so that could have something to do with it.

tags: added: network testing
Changed in nova:
status: New → Confirmed
assignee: nobody → Russell Bryant (russellb)
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/72125

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Mark McLoughlin (markmc) wrote :

I'm pretty confident this is just a case where some other test is calling is_neutron() with network_api_class set to neutron before this libvirt test is run

We should just call nova_utils.reset_is_neutron() in the libvirt test

I think is_neutron() can be tweaked to be safer ... but I don't think it was the cause of the problem in this case

Revision history for this message
Mark McLoughlin (markmc) wrote :

Reducing to low because I don't believe the unit test issues were caused by this

Changed in nova:
importance: Medium → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/72125
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2749bb9138f5946d1ef3030c7fe349fbd5920139
Submitter: Jenkins
Branch: master

commit 2749bb9138f5946d1ef3030c7fe349fbd5920139
Author: Russell Bryant <email address hidden>
Date: Sat Feb 8 16:42:08 2014 -0500

    Make is_neutron() thread-safe

    The utils method is_neutron() was not thread safe and could have returned the
    wrong value if two greenthreads called it at the same time. Specifically, a
    greenthread switch could occur when the import is being done. At that point, it
    looks like the check has already been completed, but the result has not yet been
    recorded.

    Closes-bug: #1274992
    Change-Id: I5a6eedaeb4c0deee4b31509946ae5daedf98389d

Changed in nova:
status: In Progress → Fix Committed
Changed in nova:
milestone: none → icehouse-3
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.