support shipping verbatim files in the exported tarballs

Bug #123020 reported by Martin Pitt on 2007-06-29
8
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Julian Edwards
langpack-o-matic
Wishlist
Martin Pitt
pkgbinarymangler (Ubuntu)
Wishlist
Martin Pitt

Bug Description

On the live CD, one of the biggest directories is /usr/share/gnome/help/, which contains localized XML and PNG files. It would be nice if pkgstriptranslations would remove those from packages and put them into the translation tarballs. Initially, Rosetta should just ship them verbatim in the exported tarballs for langpack building. Later it could even support generating them dynamically from PO files and ship the updated ones to the langpack builder.

Martin Pitt (pitti) on 2007-06-29
Changed in pkgbinarymangler:
assignee: nobody → pitti
importance: Undecided → Wishlist
Martin Pitt (pitti) wrote :

Currently, /usr/share/gnome/help has 103 MB of language specific files, which gzip -9 into 72 MB.

Some typical sizes of a particular translation (all gzip -9 compressed):
  es: 9.2 MB
  de: 4.4 MB
  fr: 5.5 MB
  ar: 780 KB

Jeroen T. Vermeulen (jtv) wrote :

This would be a lot of work. It would not solve the problem fully (e.g. we couldn't generate those PNG files), and the intermediate step of exporting XML files and such untranslated wouldn't be all that helpful.

For this use case, it should definitely be done outside Launchpad. Where we find a real need to do something like this inside Launchpad, it'd be a blueprint rather than a bug.

Changed in rosetta:
status: New → Won't Fix
Martin Pitt (pitti) wrote :

Rosetta doesn't actually need to interpret any of those files, or change them. The problem is that we need some hook in Launchpad to export the files which we stripped during the build to some place where we can handle them. That could be as easy as making the uploaded _translations.tar.gz files available somewhere (right now they just disappear into a black box).

Martin Pitt (pitti) wrote :

Please close again if even this simple solution is too much work. If so, do you have another idea?

Changed in rosetta:
status: Won't Fix → New
Jeroen T. Vermeulen (jtv) wrote :

I don't even think we ever keep the uploaded tarballs per se in Librarian. A lot relies on us treating the individual translation files inside the tarball as separate files, and we don't have a single other reason to keep the tarball around. This is not a generic file-sharing application!

Danilo points to Soyuz as a possible place to look for the stripped data.

Changed in rosetta:
status: New → Won't Fix
Martin Pitt (pitti) wrote :

Hang on, let's keep this open. I asked Adam and Celso a while ago, but did not get a response so far.

Changed in pkgbinarymangler:
status: New → Invalid
status: Invalid → Confirmed
Martin Pitt (pitti) wrote :

On UDS I had a discussion with Danilo about how to move GNOME help file translations to language packs [1], since they are extremely big.

There doesn't currently seem to be a way to do this through the translations.tar.gz tarball and Rosetta.

So I wondered if it is possible to build another raw-gnome-help or similar tarball during package build and stuff that to the _arch.changes. Would that file land in the librarian somehow, so that langpack-o-matic can pick them up and unpack them into the langpacks? If not, would that be something easy to do? Or am I totally on crack
with the idea?

[1] https://wiki.ubuntu.com/DesktopTeam/Specs/JauntyGnomeHelpLangpacks

Changed in rosetta:
status: Won't Fix → Incomplete
Changed in pkgbinarymangler:
status: Confirmed → Incomplete
Celso Providelo (cprov) wrote :

Pitti,

Yes, implementing a new custom format is fairly easy, the uploaded files will end up in librarian.

You should decide how is the way to expose them. From what I can see they don't need to end up in the repository (as d-i or ddtp-tarballs), so we could provide web UI and/or APIs to facilitate browsing and download.

I'm assuming the changes required in pkgbinarymangler are feasible, I can't really evaluate them.

Hey Celso,

Celso Providelo [2009-01-27 17:00 -0000]:
> Yes, implementing a new custom format is fairly easy, the uploaded files
> will end up in librarian.

Glad to hear it! Assuming the .changes would have a raw-*
tarball, how could I find these tarballs in the librarian?

> You should decide how is the way to expose them. From what I can see
> they don't need to end up in the repository (as d-i or ddtp-tarballs),

Confirmed, I just need to be able to find all of them mechanically,
i. e. the latest version for each source package per distrorelease.

> so we could provide web UI and/or APIs to facilitate browsing and
> download.

That would be awesome.

> I'm assuming the changes required in pkgbinarymangler are feasible, I
> can't really evaluate them.

They are, I just set it to incomplete because I don't know yet how
soyuz needs to get them fed. Would something like this be okay?

  123DEADBEEF 123456 raw-translations-static - gnome-mount_0.8-1ubuntu1_i386_translations-static.tar.gz

Thanks, Martin

Celso Providelo (cprov) wrote :

Pitti,

The .changes snippet suggested looks fine. 'raw-translations-static' identifier is fine too, although it may cause some confusion with the current 'raw-translations' one, but I don't mind as long as it's clear to you.

Regarding the look-up mechanism I'm not entirely sure. Custom uploads are directly related to build records but in most of the cases they are only generated on the i386 attempts (nominatedarchindep), resulting in source->[builds]->[custom] that can be easily aggregated and exposed via the API as:

{{{
>>> ubuntu = launchpad.distributions['ubuntu']
>>> firefox = ubuntu.getSourcePackage('firefox')
>>> for custom_file in firefox.custom_files:
.... print custom_file.filename, custom_file.custom_type
firefox_1.0_i386-translations.tar.gz RAW-TRANSLATION
firefox_1.0_i386-translations-static.tar.gz RAW-TRANSLATIONS-STATIC
...
}}}

Or something similar. Obviously custom file would allow you to get the file contents via the API as well.

On the web UI side, we could simply present 'custom files' everywhere we present the source and binary files, I guess.

Martin Pitt (pitti) wrote :

That launchpadlib workflow looks suitable for me, that would be great.

The solution design seems clear to me now, thus marking as Triaged. Unassigning since the pkgbinarymangler fix is blocked by getting the soyuz feature.

Changed in pkgbinarymangler:
assignee: pitti → nobody
status: Incomplete → Triaged
Changed in soyuz:
status: Incomplete → Confirmed
Celso Providelo (cprov) wrote :

I've some work in the same are related with the debug-symbols support scheduled to 2.2.3. Let's keep this bug around, maybe I can do something yet in this cycle, but no guarantees.

Changed in soyuz:
milestone: none → pending
status: Confirmed → Triaged
Martin Pitt (pitti) wrote :

Any chance we could get this implemented soon? We have to start throwing out some serious features from the desktop CD soon otherwise (killing GIMP, for example).

Celso, let's try and get this done in 2.2.6, we can speak about it in the week.

Changed in soyuz:
milestone: pending → 2.2.6
importance: Undecided → High
tags: added: soyuz-upload
Changed in soyuz:
assignee: nobody → Julian Edwards (julian-edwards)
Changed in soyuz:
milestone: 2.2.6 → 2.2.7
Martin Pitt (pitti) on 2009-06-25
Changed in pkgbinarymangler (Ubuntu):
assignee: nobody → Martin Pitt (pitti)

After talking to pitti, here's what we're going to do:

1. Add a new custom upload file type, RAW-TRANSLATIONS-STATIC
2. Store these files in the librarian, hanging off the usual PackageUploadCustom table entries.
2. Expose a new webservice method: IDistroSeries.getStaticTranslations(since="<date>") which will return a collection of URLs to the files in the librarian.

The files are easily associated with the original package because it's part of the file name.

Optionally, we can add some webservice calls like Celso suggests above.

The buildds also need to be changed to strip these files out and put them in the build, of course.

Job done.

Martin Pitt (pitti) on 2009-06-26
Changed in langpack-o-matic:
assignee: nobody → Martin Pitt (pitti)
importance: Undecided → Wishlist
status: New → Triaged
Changed in soyuz:
status: Triaged → In Progress

Fixed in devel, RF 8772

Changed in soyuz:
status: In Progress → Fix Committed
Changed in soyuz:
status: Fix Committed → Fix Released

Julian Edwards [2009-07-23 10:32 -0000]:
> ** Changed in: soyuz
> Status: Fix Committed => Fix Released

Wohoo, thanks muchly!

So what do I have to do now to make this work? Can pkgbinarymangler
just start to upload RAW-TRANSLATIONS-STATIC files? And how to get
them back through launchpadlib?

Thanks! Martin

Yes, there's still some work on pkgbinarymangler of course, which needs to be
integrated in a new chroot for the buildds.

To retrieve the files on launchpdlib you need to use the getPackageUpload()
method on the DistroSeries object and pass it a param custom_type='raw-
translations-static'. The returned objects have a custom_file_urls list
property that specify librarian URLs where you can get the files.

Martin Pitt (pitti) on 2009-07-23
Changed in pkgbinarymangler (Ubuntu):
status: Triaged → In Progress
Martin Pitt (pitti) wrote :

Julien,

I tested it on dogfood in https://dogfood.launchpad.net/~pitti/+archive/gnome-help-test and gnome-applets built fine:

  pkgstriptranslations: preparing static translation tarball gnome-applets_2.27.3-0ubuntu3test2_i386_translations_static.tar.gz...done (403 files)

But it failed to upload:

2009-08-03 22:02:20 WARNING Upload was rejected:
2009-08-03 22:02:20 WARNING Upload has more architetures than it is supported.

Perhaps it stumbled over the raw-translations-static .changes part? Celso will fish out the .changes.

Martin Pitt (pitti) wrote :

The upload is on https://chinstrap.canonical.com/~cprov/20090803-230207-1064973-2323117/

Reopening the bug for now. Thanks in advance for having a look!

Changed in soyuz:
status: Fix Released → In Progress
Martin Pitt (pitti) wrote :

cprov-afk| pitti: aha, we have a problem in the upload processor
cprov-afk| pitti: we cope with <arch>_translation but not with <arch>_translation_static

Ursula Junque (ursinha) wrote :

As this bug is still in progress, I'm re-targeting it to 2.2.8.

Changed in soyuz:
milestone: 2.2.7 → 2.2.8

Assigning back to 2.2.7 and closing. Please open a new bug with the relevant and specific problems if necessary Martin.

Changed in soyuz:
milestone: 2.2.8 → 2.2.7
status: In Progress → Fix Released
Martin Pitt (pitti) wrote :

Ah, after the n-th iteration of naming schemas it works now. Thanks for your help!

pkgbinarymangler (61) karmic; urgency=low

  * pkgstriptranslations: Extract GNOME help translations into a new tarball
    and add them to the upload as raw-translations-static component. Do not
    strip the files from the .debs just yet, since the upload/download
    infrastructure in Soyuz still needs to be tested, and the language pack
    integration bits need to be written. See LP #123020
  * debian/compat, debian/control: Bump to debhelper compat 5, 4 is
    deprecated.
  * debian/control: Bump Standards-Version to 3.8.2.
  * debian/control: Drop long-obsolete Replaces.

 -- Martin Pitt <email address hidden> Mon, 03 Aug 2009 12:59:52 +0100

From the PPA, I get

$ ./static-translations
upload: https://api.dogfood.launchpad.net/beta/ubuntu/karmic/+upload/1085128
display name: gnome-applets, gnome-applets_2.27.3-0ubuntu3test5_i386_translations.tar.gz, gnome-applets_2.27.3-0ubuntu3test5_static_translations.tar.gz
display version: 2.27.3-0ubuntu3test5
custom files: ['https://librarian.dogfood.launchpad.net/27630559/gnome-applets_2.27.3-0ubuntu3test5_i386_translations.tar.gz', 'https://librarian.dogfood.launchpad.net/27630560/gnome-applets_2.27.3-0ubuntu3test5_static_translations.tar.gz']

which looks fine.

Martin Pitt (pitti) wrote :

pkgbinarymangler (62) karmic; urgency=low

  * pkgstriptranslations:
    - Also store /usr/share/omf/ files in static translation tarball.
    - Remove GNOME help translation files now, and replace them with symlinks
      to /usr/share/omf-langpack/* and usr/share/gnome/help-langpack/*.

 -- Martin Pitt <email address hidden> Sun, 30 Aug 2009 12:39:06 +0200

pkgbinarymangler (61) karmic; urgency=low

  * pkgstriptranslations: Extract GNOME help translations into a new tarball
    and add them to the upload as raw-translations-static component. Do not
    strip the files from the .debs just yet, since the upload/download
    infrastructure in Soyuz still needs to be tested, and the language pack
    integration bits need to be written. See LP #123020
  * debian/compat, debian/control: Bump to debhelper compat 5, 4 is
    deprecated.
  * debian/control: Bump Standards-Version to 3.8.2.
  * debian/control: Drop long-obsolete Replaces.

 -- Martin Pitt <email address hidden> Mon, 03 Aug 2009 12:59:52 +0100

Changed in pkgbinarymangler (Ubuntu):
status: In Progress → Fix Released
Martin Pitt (pitti) wrote :

langpack-o-matic now downloads the tarballs and installs them into the langpacks. Just tested a recent build, about to upload them.

Changed in langpack-o-matic:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers