Expire translations-tarball file contents once they get successfuly processed

Bug #262767 reported by Celso Providelo
4
Affects Status Importance Assigned to Milestone
Launchpad itself
Triaged
Low
Unassigned

Bug Description

In order to reduce the cruft stored in librarian "process-accepted" should expire LibraryFileContents related with translation tarballs once they were successfully processed, at this point all the relevant files will be already stored in the the rosetta queue.

I suggest the LFC expiration approach instead of removing the corresponding PackageUploadCustom record because users might be interested in know when/how/if a particular translation-tarball was uploaded as expected during building time and that's currently visible in the $distroseries/+queue UI.

Revision history for this message
Stuart Bishop (stub) wrote :

Why wait until they are successfully processed? Just create them with an expiry a few months in the future - say 3 months. If they haven't been processed by then, something major has gone wrong and I doubt the files are still relevant.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Danilo, is this possible? Do you use the librarian file that Soyuz uploaded after processing or can it be expired? Should they be created to expire in the future like stub suggests or do you think Rosetta could/should expire them?

Changed in soyuz:
status: New → Incomplete
Revision history for this message
Данило Шеган (danilo) wrote :

Rosetta unpacks the translation tarball right in the addOrUpdateEntriesFromTarball and puts individual files from the tarball into the import queue. Tarball is not used after that and can be immediatelly discarded: we don't keep any LFA or LFC links to it, so we always believed that happened automatically anyway. We did use full tarball access when debugging, so it'd be nice if they lived long enough to be used in debugging.

Also, our script which does "re-uploads" of those tarballs needs them to stay around, but we only use this to fix translations after bug has been fixed without requiring a rebuild of a package. We very rarely used it, so it's not critical for our operation if these tarballs are taking up too much space on librarian.

Changed in soyuz:
status: Incomplete → New
Changed in soyuz:
status: New → Triaged
importance: Undecided → Low
tags: added: trivial
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Soyuz used to create translations tarballs for the upload, but we got rid of those a few years back. Nowadays, attachTranslationFiles feeds the source package release's deb file straight into TranslationsImportQueue.addOrUpdateEntriesFromTarball (after all a deb is also a tarball!), which unpacks the tarball on the fly so that Rosetta never needs the file from the Librarian again.

The only duplication there is that the unpacked translation files do go back into the librarian, but that's only a tiny fraction of disk space usage.

So really this is about sourcepackagerelease files in the librarian, not translations tarballs.

Revision history for this message
Данило Шеган (danilo) wrote :

Jeroen, I think you are mistaken here. Soyuz "custom" upload is an actual upload of the tarball which contains translations (I am not sure if it contains full source code as well, but that's not really relevant). If debs actually included PO files, that'd be very weird, and they'd have to be there without being installed, but would still take a lot of disk space on Ubuntu live CDs eg.

Also, while debs are based on tarballs, they are not really ones. They are tarballs wrapped with some binary data (or something like that), so our code wouldn't even work if this was the case. FWIW, I've tried and even tarfile.is_tarfile() returns False for .deb files.

Revision history for this message
Julian Edwards (julian-edwards) wrote : Re: [Bug 262767] Re: Expire translations-tarball file contents once they get successfuly processed

On Thursday 12 August 2010 15:03:23 Данило Шеган wrote:
> Jeroen, I think you are mistaken here. Soyuz "custom" upload is an
> actual upload of the tarball which contains translations (I am not sure
> if it contains full source code as well, but that's not really
> relevant).

Custom uploads are defined as anything that has a "custom" file attached
(translations fall into that category).

> If debs actually included PO files, that'd be very weird, and
> they'd have to be there without being installed, but would still take a
> lot of disk space on Ubuntu live CDs eg.
>
> Also, while debs are based on tarballs, they are not really ones. They
> are tarballs wrapped with some binary data (or something like that), so
> our code wouldn't even work if this was the case. FWIW, I've tried and
> even tarfile.is_tarfile() returns False for .deb files.

A deb is actually a gnu "ar" archive :)

Try "ar t <file.deb>"

Revision history for this message
Данило Шеган (danilo) wrote :

We can run the following query to expire those which are older than 3 months: https://pastebin.canonical.com/35808/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.