Ejecting a valid master is sometimes allowed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack DBaaS (Trove) |
New
|
Undecided
|
Unassigned |
Bug Description
There is a test to eject a valid master that is failing more and more often in the scenario tests. Basically the eject will only work if the heartbeat from the current master is more than a minute old. This shouldn't happen in the scenario test run, but it does - quite often.
One theory is that there is clock drift involved, and the comparison of the two timestamps needs to be synchronized (i.e. both timestamps need to originate from the same machine). Typically the database is entrusted with adding timestamps to records to make sure they're all correct relative to each other, however this may or may not be possible in Trove's circumstances (since network or queue delays could mask the 'real' time the heartbeat was generated. It could be that this is not an issue, in which case the heartbeat time delta could be extracted directly from the db in the form of a time diff in seconds.
The current 'eject valid master' test will be turned off until the fix for this lands.
Related fix proposed to branch: master /review. openstack. org/368230
Review: https:/