Launchpad doesn't generate translation templates

[Данило Шеган]

Checking a random number of entries from the builddmanager logs, I could not identify one project which has its template generated from source code. The projects were either 1) not using LP for translations at all, 2) importing a commited template from the branch, 3) using a very old template.

It seems that something is fundamentally broken with the template generation.

[Henning Eggers (henninge)]

It seems that LightDM has the same problem.

[Henning Eggers (henninge)]

As far as I can tell, no translation templates are being generated at all. This either broke or never worked.

If anybody knows of any project using this feature successfully, please let us know!

[Launchpad QA Bot (lpqabot)]

r13897 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13897) is part of this bug's fix.

[Данило Шеган (danilo)]

It has definitely worked in the past. However, I wonder if the sanitization of the logging/job types was ever completed for template generation jobs? The last I heard no logs were being linked to the appropriate records with them.

[Henning Eggers (henninge)]

Yes, no logs are being stored. I tried to find their librarian ids in production but found only NULLs. :(

It seems that the jobs are never getting to the WAITING state because even after making sure that the end of a job gets logged (see the better-logging branch) there is no output in buildd-manager.log. It looks like the jobs are being started but never finish. A job start is never recorded, either. The buildfarmjob table looks like this: https://pastebin.canonical.com/52308/

[Henning Eggers (henninge)]

When this gets fixed, bug 644464 should get fixed along with it. (I thought that was long done but nm, isn't that a beautiful bug number?)

[Henning Eggers (henninge)]

I have not been able to get to the bottom of this. I tried to run the buildd slave locally (like I did when I developed this feature) but I failed. Due to lack of time I cannot pursue this any more.

If you pick this up, please feel free to contact me about further information.

[Данило Шеган (danilo)]

FWIW, this was broken with the release from 2010-10-14 (that's when we started getting unprocessed translation builds).

[Данило Шеган (danilo)]

So, everything up to build job execution is fine: build-master picks them up correctly, but usually less then 3 minutes later, a PACKAGEFAIL error is returned (well, all but 1 job in October failed with PACKAGEFAIL; that one failed with CHROOTFAIL instead). Something's going fishy on the slaves, so need to focus the effort there.

For reference:
 * log files devpad:/srv/launchpad.net-logs/soyuz/cesium/build-manager.log*
 * look for "Starting templates" and "Template generation job" to find relevant messages, eg.

2011-10-11 20:44:46+0000 [QueryProtocol,client] Starting templates build trunk-4883603 for lp:eog.
...
2011-10-11 20:47:31+0000 [QueryProtocol,client] Templates generation job trunk-4883603 for lp:eog finished with status PACKAGEFAIL.

[Данило Шеган (danilo)]

Ok, with help from William I was able to set up a local instance. The problem is that natty chroot switches default Python to 2.7 and generate-translation-templates hard-codes 2.6 in PYMODULES. That's easy to fix, but since we should have caught this problem much earlier, I am also adding buildlog storage finally (I thought that was supposed to happen by the generic build machinery, but it doesn't if you don't do it yourself, especially if you are not using PackageBuildDerived as the base to derive from).

[Launchpad QA Bot (lpqabot)]

Fixed in stable r14171 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14171>.

[Данило Шеган (danilo)]

Launchpad side is good to roll out (tested on staging which had to be brought up to speed for buildd stuff again), thus marking qa-ok. Actual slave code still needs to be tested, but that gets rolled out independently.