garbo-daily is blocking itself while the standalone replica is rebuilt

[Steve McInerney]

2010-03-31 19:13:21 INFO Database replication lagged. Sleeping 300 seconds
2010-03-31 19:18:21 INFO Database replication lagged. Sleeping 300 seconds
2010-03-31 19:23:21 INFO Database replication lagged. Sleeping 300 seconds
2010-03-31 19:28:21 INFO Database replication lagged. Sleeping 300 seconds
....

SELECT replication_lag(42) => 13:35:34.767864

All others are below the 5 min threshhold.

Am assuming that blocking on the standalone is a bad thing™??

[Stuart Bishop (stub)]

This is by design. If it takes 10 hours to rebuild a replica, then 10 hours worth of changes end up being spooled before they can be processed. So we block batch jobs to minimize the amount of changes.

If it is a problem we can revisit it. Perhaps not doing the lag check for critical jobs?

[Stuart Bishop (stub)]

Rethinking on this, the problem is that the LoopTuner timeout code does not kick in when the DBLoopTuner is blocking for replication lag. The loops should terminate after the abort time, even if it never got a chance to do anything due to lag or long running transactions.

[Ursula Junque (ursinha)]

Fixed in stable r10646 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10646>

[Stuart Bishop (stub)]

There is a bug in _sleep() - we should sleep the number of seconds passed in the input parameter.

[Ursula Junque (ursinha)]

Fixed in stable r10819 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10819>