Enable translating selected Ubuntu universe packages in Launchpad

[David Planella]

Binary package hint: pkgbinarymangler

Following a conversation at UDS and subsequent follow-ups with the OEM group, I'd like to request the possibility to enable the translation of selected universe packages in Launchpad.

That would require:
* Changes in pkgbinarymangler to create a whitelisting system to be able to selectively build
* Changes in Launchpad to stop ignoring translation imports coming from universe packages and start importing any translation tarball that pkgbinarymangler provides.

Note that for source packages with upstream translations sharing enabled in Launchpad, this might not be even necessary (is there any check to ignore sharing if the Ubuntu source package comes from universe?), but it will be to import translations for packages not supported by upstream imports sharing (i.e. any project not based on intltool)

Background notes
----------------

(from conversations with pitti and danilo)

In order to import universe translations and ship them in language packs we need:
* A trivial patch to pkgbinarymangler and get its tests updated. Estimated time 15 m.
* A small patch in Launchpad to stop ignoring universe translation tarball uploads.
* Research on how the amount of universe translations compare with those from main. Results:
  - We ship a total of ~1.25 GB (1252512038 bytes) of translation data as the sum of all language packs
  - We ship a total of ~0.85 GB (845977204 bytes) of translation data as the sum of all universe packages
  - This means universe translations have roughly 70% of the size of the translations we ship in language packs
* A decision from the research action above on whether to ship separate language packs or not
* If we decide to ship separate universe language packs, a (not too hard) modification to langpack-o-matic to generate them.

Other notes:
* IIRC, from the previous UDS discussion in Brussels, we'd only want to enable this for selected universe packages, those which are well maintained, since this would introduce a delta with those packages coming from debian (the delta being the .pot building rules).
* The discussion at UDS-O was not about mass-enabling universe translations, but rather the context was about enabling them for some packages in which we'd want to have more flexibility with translations and be able to ship updates in language packs without requiring a main inclusion request for the package. An example is the nanny package, which OEM distributes in their images.
* So as we're not talking about enabling all universe, a start would be to add a way to whitelist selected universe packages and decide to ship them in the regular language packs

Implementation:

 * pkgbinarymangler: Stop building translation tarballs for universe packages, i. e. for packages which it does not strip.
 * pkgbinarymangler: Check "XS-Ubuntu-Use-Langpack: yes" header in debian/control. If present, force stripping/translation tarball building. This can be used for particular universe packages. That way, we don't need a static whitelist in pkgbinarymangler, and it will also work for PPAs, etc.
 * Launchpad: Stop ignoring universe translation tarballs.

[Kyle Nitzsche (knitzsche)]

This is high priority for OEM group.

With this feature there is a much lower chance that customized versions of Ubuntu that OEM creates will have translation regressions in the field.

Without this feature, if we complete translations for a given package (by modifying the package itself) and then if the package is released by upstream in universe (still with incomplete translations), it will pre-empt our version and the translations will revert. There could be significant translation regressions for users.

We are requesting this feature in support Qin (Chinese Ubuntu) project (https://blueprints.launchpad.net/ubuntu/+spec/desktop-o-qin-ubuntu-china).

Thanks,
Kyle

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 100

---------------
pkgbinarymangler (100) oneiric; urgency=low

  * test/run: Add forgotten check_deb_integrity() to documentation symlink
    tests. This catches the wrong md5sums of symlinked doc files.
  * pkgstripfiles: Remove doc files from md5sums which get replaced with a
    symlink. (LP: #795845)
  * test/run: Call check_deb_integrity() in build(), and remove the explicit
    call from all the individual tests. There is no situation where we expect
    broken debs to be created, so this saves us from accidentally forgetting
    to call it in newly created tests.
  * test/run: Drop separate check_translation_tarball() test, and integrate it
    into check_deb_mo(). We now want to move to only building translation
    tarballs for packages which we actually strip, so it does not make sense
    any more to check those two individually.
  * pkgstriptranslations: Do not build a translation tarball if we did not
    strip the packages. With that, Launchpad does not need to check the
    component of the package any more, but can just import everything we throw
    at it. This also allows us to put some selected universe packages under
    langpack support. (LP: #788685, part 1)
  * pkgstriptranslations: Force stripping and translation tarball build if
    debian/control has "X..-Ubuntu-Use-Langpack: yes" header. Add
    corresponding test case. (LP: #788685, part 2)
  * test/run: Fix ResourceWarnings for unclosed files. Python 3, you are
    overly picky!
 -- Martin Pitt <email address hidden> Wed, 15 Jun 2011 09:08:54 +0200

[Данило Шеган (danilo)]

Fix for LP should be quite simple. lib/lp/soyuz/model/queue.py publishRosettaTranslations is where "ignoring" happens. Any LP engineer could take on it, but the priority needs to be raised first: please get that arranged through the stakeholder process for bug escalation.

[Henning Eggers (henninge)]

The actual change maybe as simple as this but I don't know if that is all. Also, we'd want a test for this.

=== modified file 'lib/lp/soyuz/model/queue.py'
--- lib/lp/soyuz/model/queue.py 2011-06-13 18:17:37 +0000
+++ lib/lp/soyuz/model/queue.py 2011-06-15 09:13:41 +0000
@@ -1165,7 +1165,7 @@
         valid_pockets = (
             PackagePublishingPocket.RELEASE, PackagePublishingPocket.SECURITY,
             PackagePublishingPocket.UPDATES, PackagePublishingPocket.PROPOSED)
- valid_component_names = ('main', 'restricted')
+ valid_component_names = ('main', 'restricted', 'universe')
         if (self.packageupload.pocket not in valid_pockets or
             sourcepackagerelease.component.name not in valid_component_names):
             # XXX: CarlosPerelloMarin 2006-02-16 bug=31665:

[Martin Pitt (pitti)]

It might not be that simple, I'm afraid:

 * For oneiric and onwards, I'd just drop the valid_component_names check altogether, so that we can also catch multiverse, partner, and what not (if enabled with X-Ubuntu-Use-Langpacks:).

 * For natty and earlier we need to keep the check, to avoid importing translations for universe SRUs.

[Steve Magoun (smagoun)]

Marked this invalid because oem-priority is not used to track bugs in Launchpad that affect the OEM group, we have a separate escalation process for that. Also, the pkgbinarymangler component of the bug is already FixReleased so there is minimal value in tracking that as oem-priority.

[Francis J. Lacoste (flacoste)]

I forgot to tag this bug as escalated when I raised the priority some weeks ago.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13510 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13510>.

[David Planella (dpm)]

There is still a bit that needs fixing: see bug 1048556 as a follow-up