[SRU] race between neutron-ovs-cleanup and nova-compute
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
nova (Ubuntu) |
Fix Released
|
High
|
Edward Hope-Morley | ||
Trusty |
Fix Released
|
High
|
Edward Hope-Morley | ||
Utopic |
Fix Released
|
High
|
Edward Hope-Morley | ||
Vivid |
Fix Released
|
High
|
Edward Hope-Morley |
Bug Description
[Impact]
* We run neutron-ovs-cleanup in startup if neutron installed. If
nova-compute does not wait for completion it will try to use
veth/bridge devices that may be in the process of bring deleted.
[Test Case]
* Create neutron (ovs) network and boot an instance with this network
as --nic
* Check that creation was successful and network is functional. Also make
a note corresponding veth and bridge devices (ip a).
* Reboot system, check that expected veth and bridge devices are still
there and that nova-compute is happy e.g. try sshing to your instance.
Also check /var/log/
for ovs-cleanup to finish.
[Regression Potential]
* None
---- ---- ---- ----
There is a race when both neutron-ovs-cleanup and nova-compute trying to do operations on the qvb*** and qvo*** devices. Below is a scenario I recently met,
1. nova-compute was started and creating the veth_pair for VM instances running on the host - https:/
2. neutron-ovs-cleanup was kicked off and deleted all the ports.
3. when nova-compute tried to set the MTU at https:/
As they both operate on the same resources, there needs a way to synchronize the operations the two processes do on those resources.
Related branches
- Ubuntu Server Developers: Pending requested
-
Diff: 56 lines (+18/-2)3 files modifieddebian/changelog (+10/-0)
debian/control (+2/-2)
debian/nova-compute.upstart (+6/-0)
- Ubuntu Server Developers: Pending requested
-
Diff: 31 lines (+13/-0)2 files modifieddebian/changelog (+7/-0)
debian/nova-compute.upstart (+6/-0)
Changed in neutron: | |
status: | New → Confirmed |
tags: | added: cts |
Changed in nova (Ubuntu): | |
importance: | Undecided → High |
assignee: | nobody → Edward Hope-Morley (hopem) |
status: | Confirmed → In Progress |
description: | updated |
description: | updated |
Changed in nova (Ubuntu Vivid): | |
status: | In Progress → Fix Committed |
summary: |
- race between neutron-ovs-cleanup and nova-compute + [SRU] race between neutron-ovs-cleanup and nova-compute |
tags: |
added: verification-done removed: verification-needed |
I think this can be fixed by adding an upstart pre-start rule similar to the one used in neutron- *-agent. upstart e.g.
pre-start script ovs-cleanup; then neutron- ovs-cleanup WAIT_STATE=running WAITER=nova-compute
# Check to see if openvswitch plugin in use by checking
# status of cleanup upstart configuration
if status neutron-
start wait-for-state WAIT_FOR=
fi
end script