Automatic translation exports fail to commit latest translations

[David Planella]

It's been brought up in the translations list that translations for ubuntu-docs are outdated, even though they were fetched from the daily automatic translations export branch.

Looking at Launchpad for French I can see that:

- The number of untranslated strings at the _upstream project_ [1] is 337, of a total of 2732 translatable strings
- The numbers match exactly for the _Ubuntu source package_ [2]
- When looking at the PO files in the exports branch [3], however, there are 555 untranslated strings (total stays the same)

E.g.

$ bzr branch lp:~ubuntu-core-doc/ubuntu-docs/precise-translations
$ cd precise-translations/ubuntu-help
$ head -n 15 fr.po
# French translation for gnome-user-docs
# Copyright (c) 2011 Rosetta Contributors and Canonical Ltd 2011
# This file is distributed under the same license as the gnome-user-docs package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2011.
#
msgid ""
msgstr ""
"Project-Id-Version: gnome-user-docs\n"
"Report-Msgid-Bugs-To: FULL NAME <EMAIL@ADDRESS>\n"
"POT-Creation-Date: 2012-04-03 22:26-0400\n"
"PO-Revision-Date: 2012-04-08 21:32+0000\n"
"Last-Translator: YannUbuntu <email address hidden>\n"
"Language-Team: French <email address hidden>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"

This means the last change in the PO file seems to have been on 2012-04-08

Which seems to coincide with the latest commit one day later

$ bzr log fr.po
------------------------------------------------------------
revno: 12
committer: Launchpad Translations on behalf of ubuntu-core-doc
branch nick: ubuntu-docs
timestamp: Mon 2012-04-09 05:11:39 +0000
message:
  Launchpad automatic translations update.
------------------------------------------------------------

However, the exports seem to have stuck to that date, and subsequent new translations in Launchpad have not been committed. The number of untranslated strings in the PO file on the branch is 280 more than the untranslated count in the web UI:

$ msgfmt -v -o /dev/null fr.po
2177 translated messages, 555 untranslated messages.

So it seems that:

- Translations sharing in the Lp database is working fine for upstream and downstream
- Up-to-date upstream translations seem to fail getting committed on the exports branch

Could it be that translations after the latest commit date have been made by uploading a PO file instead of using the web UI, and thus changes in translations have not been detected (and not committed)?

[1] https://translations.launchpad.net/ubuntu-docs/precise/+pots/ubuntu-help/fr/+details
[2] https://translations.launchpad.net/ubuntu/precise/+source/ubuntu-docs/+pots/ubuntu-help/fr/+details
[3] https://code.launchpad.net/~ubuntu-core-doc/ubuntu-docs/precise-translations

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jeroen T. Vermeulen (jtv)]

I believe the problem may be that sharing a message does not necessarily update the date_changed on any affected POFiles. It's easy for POTMsgSet._setTranslation to decide to make a message active in other POFiles (in other release series or even between Ubuntu and upstream), but it's harder to figure out exactly which POFiles will see a change because of it.

And so, I suspect that we don't do that at all, failing to set POFile.date_changed, which is exactly what the translations-to-branch exporter looks at to decide whether the POFile needs a new export.

This might be fixable through some offline process, or perhaps the translations-to-branch exporter needs to look at more information than just POFile.date_changed.

[Olivier Dony (Odoo) (odo-openerp)]

I suspect I see a similar problem with translation sharing between series, and the behavior seems to match Jeroen's description.
The Japanese translations for OpenERP Addons have been updated in the trunk series in the past few days, and are properly shared with the 6.1 series. You can see the changed dates of some terms are 20 june in the UI for the 6.1 series [1].
However if you look at the last update date of the PO file itself [2], it's still 12 june, and indeed the last export for the 6.1 series was on june 13 [3]. The PO files are exported correctly in the trunk series however [4].

This is quite a problem, because our workaround would be to "touch" the templates in the 6.1 series every time they are changed in the trunk series, in order to force an export.... possibly requiring hundreds of manual operations every day. Good luck explaining this to the translation teams ...

An offline process to detect this and patch the POFile.date_changed in the database directly would be great.
Is there any chance we can do it ourselves using launchpadlib? It does not seem to expose the actual PO database, only the import queue and translation groups.

Note: the dates I quoted may not be accurate for long as we will have to manually "touch" the templates in order to trigger the missing exports.

[1] https://translations.launchpad.net/openobject-addons/6.1/+pots/purchase/ja/+translate
[2] https://translations.launchpad.net/openobject-addons/6.1/+pots/purchase
[3] http://bazaar.launchpad.net/~openerp/openobject-addons/6.1/view/head:/purchase/i18n/ja.po#L17
[4] http://bazaar.launchpad.net/~openerp/openobject-addons/trunk/view/head:/purchase/i18n/ja.po#L17

[Olivier Dony (Odoo) (odo-openerp)]

For the record, this was reported on OpenERP via bug 1015411, which seems to be an indirect consequence of this (but maybe not strictly a duplicate)

[Dan Wells (dbw2)]

I am still seeing the behavior today. Basically, as noted in comment #2, we think this is related to the language timestamps not updating, even though the underlying strings are. For example, at the time of this writing, compare "tpac" timestamp here:

https://translations.launchpad.net/evergreen/master/+lang/es

vs here:

https://translations.launchpad.net/evergreen/3.0/+lang/es

(Right now they are about an hour apart, but that is just from me fiddling. If we do nothing else, the 3.0 timestamp just gets further and further behind, though the strings inside do reflect the updates.)

[Dan Wells (dbw2)]

(P.S. You can see the exact edit timestamp by hovering over the date display.)