commit 5fe83cbcb5d2acc8af5e05d65345b114f859a359
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri May 31 11:39:39 2019 +0100
Fix metadata agent proxy list updates race condition
When multiple instances are being spawned and/or teared down at the
same time, multiple updates to the "neutron-metadata-proxy-networks"
list may race each other resulting in some networks UUIDs to go missing
in that process.
One of the problems with these updates is that the external_ids column
in the OVSDBs only supports string type values so, the "list" is an
actual comma-separated string. In order to add or remove an item from
from it the code in networking-ovn needs to transform it from and back
to string and commit the whole thing to OVN database when saving it.
When it races, commits may overwrite each other.
In order to fix this problem this patch is adding a file lock on the
method that manipulates this list and commit it to the database. It's
important to note that the metadata agent only updates the Chassis entry
which it's currently running on, therefore we do not need a distributed
lock (the file lock, being local works).
I was able to reproduce this problem fairly consistently by concurrently
spawning 50 instances each time with BrowBeat (see bug ticket for more
information). After this fix was applied I could attest that the problem
was resolved.
Change-Id: Idc7bae88ed6f3ec44541b38f559a0296d2159449
Closes-Bug: #1831224
Signed-off-by: Lucas Alvares Gomes <email address hidden>
(cherry picked from commit d84cb7938a153bd22a6fae5adbfbf8ec3b819bda)
Reviewed: https:/ /review. opendev. org/663314 /git.openstack. org/cgit/ openstack/ networking- ovn/commit/ ?id=5fe83cbcb5d 2acc8af5e05d653 45b114f859a359
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 5fe83cbcb5d2acc 8af5e05d65345b1 14f859a359
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri May 31 11:39:39 2019 +0100
Fix metadata agent proxy list updates race condition
When multiple instances are being spawned and/or teared down at the metadata- proxy-networks"
same time, multiple updates to the "neutron-
list may race each other resulting in some networks UUIDs to go missing
in that process.
One of the problems with these updates is that the external_ids column
in the OVSDBs only supports string type values so, the "list" is an
actual comma-separated string. In order to add or remove an item from
from it the code in networking-ovn needs to transform it from and back
to string and commit the whole thing to OVN database when saving it.
When it races, commits may overwrite each other.
In order to fix this problem this patch is adding a file lock on the
method that manipulates this list and commit it to the database. It's
important to note that the metadata agent only updates the Chassis entry
which it's currently running on, therefore we do not need a distributed
lock (the file lock, being local works).
I was able to reproduce this problem fairly consistently by concurrently
spawning 50 instances each time with BrowBeat (see bug ticket for more
information). After this fix was applied I could attest that the problem
was resolved.
Change-Id: Idc7bae88ed6f3e c44541b38f559a0 296d2159449 22a6fae5adbfbf8 ec3b819bda)
Closes-Bug: #1831224
Signed-off-by: Lucas Alvares Gomes <email address hidden>
(cherry picked from commit d84cb7938a153bd