translations to branch script failing with InternalError: transaction is read-only

[Diogo Matsubara]

As seen on OOPS-2024TEB15, an InternalError: transaction is read-only was raised running translations-export-to-branch script

[Jeroen T. Vermeulen (jtv)]

The problem seems to happen in _exportToBranch, which calls branchChanged on a translations branch.

The error indicates that branchChanged makes changes to the Branch model object, as loaded from the slave store.

[Jeroen T. Vermeulen (jtv)]

The branchChanged call (which means “register that branch has changed,” not “branch something that has changed” as you might expect) is meant to trigger a rescan of the branch if its last-mirrored timestamp is out of date.

I don't think branchChanged should be called here at all, *except perhaps* in the workaround for bug 375013, which in turn is marked Fix Released in bzr 2.3b5. That workaround unstacks the branch, if it was stacked. Production deployments are well past bzr 2.3b5, so we can safely remove that workaround. But it doesn't look to me as if that was the intention, or the branchChanged call would be right behind the unstacking call.

As far as I can see, the last-mirrored timestamp can be out of date for four reasons:

1. Maybe a branch needs to be re-scanned after being un-stacked (but I don't think so).
2. Branch has been updated but not scanned yet.
3. Branch has been scanned, but the slave database doesn't know it yet (because of replication lag, or because the scanner simply hasn't committed its changes yet).
4. The scan has broken, and hasn't been re-done (yet).

The translations-to-branch export script was designed to deal with cases 2 and 3 by skipping the afflicted branch and hoping for more luck next time. I think that's reasonable.

As for case 4: this looks like it would be a household problem in codehosting, but which we happen to notice when the translations-to-branch script complains. This does seem to be what used to happen.

A minimal change would be to call branchChanged on a master-store version of the branch. But that's not ideal: a stale last-mirrored timestamp may just be a symptom of high load, in which case requesting a re-scan does nothing but increase the load further. We'd want to check whether the master version is out of date as well. But that gets to be a lot of work for a script that just happened to notice the problem!

The right way to deal with broken scans, I think, would be regular autonomous data-scrubbing within codehosting. A garbo job perhaps.

[Robert Collins (lifeless)]

"
A minimal change would be to call branchChanged on a master-store
version of the branch. But that's not ideal: a stale last-mirrored
timestamp may just be a symptom of high load, in which case requesting a
re-scan does nothing but increase the load further. We'd want to check
whether the master version is out of date as well. But that gets to be
a lot of work for a script that just happened to notice the problem!
"

Scan requests are a depth-1 queue, so requesting one when the branch
is stale due to a backlog won't increase load - it will just coalesce
with the existing request.

-Rob

[Jeroen T. Vermeulen (jtv)]

Robert: it's not just the scan-request queue, but also case 3 as described above.

[Robert Collins (lifeless)]

if we tweak the scan requestor to request as of the last write time
(which the sftp/bzr: servers record) then this should handle case 3
gracefully too.

[Jeroen T. Vermeulen (jtv)]

Then maybe we should implement the update-the-master-branch hack. But in the longer run, I think this script is the wrong place to work around the problem.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r14156 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14156>.