Raft bug: OVSDB leadership transfers every 10-20 min after initial compaction
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Invalid
|
Undecided
|
Unassigned | ||
Yoga |
Fix Released
|
Undecided
|
Unassigned | ||
openvswitch (Ubuntu) |
Fix Released
|
Undecided
|
Dariusz Gadomski | ||
Focal |
Fix Released
|
Undecided
|
Unassigned | ||
Jammy |
Fix Released
|
Undecided
|
Dariusz Gadomski | ||
Kinetic |
Fix Released
|
Undecided
|
Dariusz Gadomski |
Bug Description
First compaction starts after 24 hours, or earlier after doubling of DB size.
Subsequent compactions will trigger every 10-20 min.
The OVS version hitting this issue:
ovs-vsctl (Open vSwitch) 2.17.2
Commit ID that fixes the issue is: https:/
https:/
Reproducer:
Trigger compactions by using command line tool:
ovs-appctl -t /var/run/
or by creating DB pressure, i.e.:
#!/bin/bash
for i in {1..5000}
do
ovn-nbctl ls-add sw$i
if [[ $? -ne 0 ]] ; then
echo "Failed on ls-add i: $i"
exit 1
fi
for j in {1..2000}
do
if [[ $? -ne 0 ]] ; then
fi
done
done
for i in {1..5000}
do
echo "Delete iteration i: $i"
ovn-nbctl ls-del sw$i
if [[ $? -ne 0 ]] ; then
echo "Failed on ls-del i: $i"
exit 1
fi
done
Check for leadership transfers using:
sudo grep "Transferring leadership" /var/log/ovn/ov* | grep ovsdb-server-sb.log
There should be a new entry every 10-20min.
=== Ubuntu SRU Details ===
[Impact]
Please see above
[Test Case]
* deploy Openstack Yoga
* connect to the NB DB leader and run the script to generate DB pressure. Compaction will occur after the DB doubles its size
* check for subsequent transfers after one hour using the following script:
sudo grep "Transferring leadership" /var/log/ovn/ov* | grep ovsdb-server-sb.log
[Where things could go wrong]
Regression is not expected since it reduces the frequency of transfers.
The fix has also been applied upstream https:/
Changed in openvswitch (Ubuntu Kinetic): | |
status: | New → Fix Released |
description: | updated |
description: | updated |
description: | updated |
Changed in openvswitch (Ubuntu Kinetic): | |
assignee: | nobody → Dariusz Gadomski (dgadomski) |
Changed in openvswitch (Ubuntu Jammy): | |
assignee: | nobody → Dariusz Gadomski (dgadomski) |
tags: | added: sts sts-sponsor-dgadomski |
Changed in openvswitch (Ubuntu): | |
status: | Confirmed → Fix Released |
Changed in cloud-archive: | |
status: | New → Invalid |
Alternatively, the following script can be used to generate DB pressure:
#!/bin/bash
echo "Iteration i: $i and j:$j"
ovn-nbctl lsp-add sw$i sw$i$j
echo "Failed on lsp-add i: $i and j: $j"
exit 1
for i in {1..5000}
do
ovn-nbctl ls-add sw$i
if [[ $? -ne 0 ]] ; then
echo "Failed on ls-add i: $i"
exit 1
fi
for j in {1..2000}
do
if [[ $? -ne 0 ]] ; then
fi
done
done
for i in {1..5000}
do
echo "Delete iteration i: $i"
ovn-nbctl ls-del sw$i
if [[ $? -ne 0 ]] ; then
echo "Failed on ls-del i: $i"
exit 1
fi
done