Imports from branches are broken

[Данило Шеган]

It seems that imports from branches have stopped working with the latest release last Wednesday.

Basically, we seem not to be creating any RosettaBranchJobs anymore. The log file for rosetta-branches script looks like:

2010-03-04 12:30:15 INFO creating lockfile
2010-03-04 12:30:33 INFO Ran 0 RosettaBranchJobs.
2010-03-04 12:40:15 INFO creating lockfile
2010-03-04 12:40:41 INFO Ran 1 RosettaBranchJobs.
2010-03-04 12:50:14 INFO creating lockfile
2010-03-04 12:50:33 INFO Ran 2 RosettaBranchJobs.
2010-03-04 13:00:15 INFO creating lockfile
2010-03-04 13:00:37 INFO Ran 4 RosettaBranchJobs.
2010-03-04 13:10:14 INFO creating lockfile
2010-03-04 13:20:15 INFO creating lockfile
2010-03-04 13:30:16 INFO creating lockfile
2010-03-04 13:40:14 INFO creating lockfile
2010-03-04 13:50:15 INFO creating lockfile
2010-03-04 14:00:16 INFO creating lockfile
2010-03-04 14:10:14 INFO creating lockfile
2010-03-04 14:20:16 INFO creating lockfile
2010-03-04 14:30:15 INFO creating lockfile
2010-03-04 14:40:16 INFO creating lockfile
2010-03-04 14:50:14 INFO creating lockfile
2010-03-04 15:00:15 INFO creating lockfile
2010-03-04 15:10:15 INFO creating lockfile
...

(until today only 'creating lockfile' messages appear)

[Данило Шеган (danilo)]

A note from Aaron: he's touching this code on devel/db-devel, so if we need to cherrypick a fix, we may have to do it for both production tree and for the current development tree.

[Jeroen T. Vermeulen (jtv)]

Job 1419818 for a translations-from-branch import, branchjob id 1074074, started on 2010-03-04 13:10:30 UTC. The job had been generated just under 10 minutes before. This was a job for one or our larger and more active branches. At some point it got stuck in a "running" state during a recvfrom call, probably because of the rollout and/or operational problems that coincided with it.

The hung rosetta-branches.py process was killed manually around 2010-03-09 10:50. Almost immediately, and before we could get a manual run going, the next cron run kicked in and processed the 1112 RosettaUploadJobs that had been stuck. It finished without errors less than half an hour later, at 2010-03-09 11:16:41.

While the process was stuck, we found no open database connections that appeared associated with it. We were unable to obtain a backtrace.

The stuck process in "ps": https://pastebin.canonical.com/28900/
Database processes while the process was stuck: https://pastebin.canonical.com/28901/

[Данило Шеган (danilo)]

I can also confirm that they've been running successfully ever since.