concurrent interface detach/attach may accidently unbind port
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Alexandre arents | ||
Ussuri |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Description
===========
When we run multiple concurrent detach interface on the same instance uuid/port-id,
a lot of those requests are accepted (HTTP 202) an processed because info_cache is updated
only when first request finish, so all request after the first one will fail BUT will send
an unbind that may unbind a port that have been just legitly attached to another instance.
Ex of use case: a kube operator that move all interface from one instance to another(rollout).
Steps to reproduce
==================
On devstack master:
INST=$(openstack server create --flavor m1.small --image cirros-
HOST_IP=$(hostname -I | awk '{print $1}')
PORT=$(openstack port create --network private myport | awk '/ id/{print $4}')
openstack server add port $INST $PORT
TOKEN=$(openstack token issue|awk '/ id/{print $4}')
# create detach script with this content:
echo 'curl -g -i -X DELETE http://
# open in another terminal neutron logs filtering on PUT:
sudo journalctl -u <email address hidden> -f | grep -E PUT
# Now send plenty of detach:
for i in $(seq 1 20) ; do . detach ; sleep 0.2 ; done
HTTP/1.1 202 Accepted
...
HTTP/1.1 202 Accepted
...
In neutron logs you will see plenty of unbinding request:
sudo journalctl -u <email address hidden> -f | grep -E PUT
Aug 25 12:15:01 alex-devstack neutron-
^^^ After the first PUT above, port is free to be attach.. if the app attach here the port risk to be accidently unbinded by other unbind PUT below:
Aug 25 12:15:01 alex-devstack neutron-
Aug 25 12:15:02 alex-devstack neutron-
Aug 25 12:15:02 alex-devstack neutron-
Aug 25 12:15:03 alex-devstack neutron-
Aug 25 12:15:04 alex-devstack neutron-
Aug 25 12:15:05 alex-devstack neutron-
Aug 25 12:15:05 alex-devstack neutron-
Aug 25 12:15:06 alex-devstack neutron-
Aug 25 12:15:06 alex-devstack neutron-
Aug 25 12:15:07 alex-devstack neutron-
Aug 25 12:15:10 alex-devstack neutron-
if an attach pass, and is accidently unbind as previously explain,
apps can detect port is free and try to reattach an interface that is already
attached in libvirt and you GOT HTTP 500 with those error message on compute node:
oslo_db.
0deleted'").
Expected result
===============
Only one unbind should be sent.
Actual result
=============
Multiple unbind are sent and one of them can break a port just attached
summary: |
- concurent interface detach/attach may accidently unbind port + concurrent interface detach/attach may accidently unbind port |
tags: | added: api race-condition |
Changed in nova: | |
assignee: | nobody → Alexandre arents (aarents) |
Fix proposed to branch: master /review. opendev. org/747957
Review: https:/