Doing a search in the ddtp-ubuntu project's translations templates times out

[David Planella]

Trying to search for a string to translate in the ddtp-ubuntu seems to always cause a timeout:

https://translations.launchpad.net/ddtp-ubuntu/oneiric/+pots/ddtp-ubuntu-universe/ca/+translate?batch=10&show=all&search=modeller

Here's one of the OOPSes I got: OOPS-9d22d31c670b0b10bbd6b726d2968c5c

[Hendrik Schrieber (hennekn)]

Also seeing this problem. For example Error ID: OOPS-eb5135d8c8ec7f178dc45cbae4be4633. This bug is present for ddtp-ubuntu-main and ddtp-ubuntu-universe.

[Michael Basse (michael-alpha-unix)]

i was adding the launchpad-maintainers on CC, so that they will see this bug

[xatr0z (xatr0z)]

Nightmonkey is totaly unusable for me because of this problem. I can't click any link to a translation (because it uses the searchfield).
Also if I manually search it doesnt work *at all* right now, however in the past it did on other projects (not ubuntu).

Everytime there is a timeout error after 10 secons waiting. The OOPS-ID does change every time.
For example:
https://translations.launchpad.net/ddtp-ubuntu/precise/+pots/ddtp-ubuntu-universe/nl/+translate?batch=10&show=all&search=pcman
Gives (Error ID: OOPS-7ef9f80c46d50ffd9f4b46a071cb8e79)

[Dennis Baudys (thecondordb)]

I'd like to confirm this timeout problem and like to ask what the current status is on this bug?
What exactly causes the timeout? Who can fix this?

As xatr0z already wrote, Nightmonkey is practically unuseable for (at least) searching for package descriptions in the universe repository – though Nightmonkey has been the only tool that could point translators to the strings in a reasonable time and with an overview (important for translation consistence).

That's why I fear we will not see ANY translations of (at least) universe package descriptions in the moment and in the future.

I'm willing to help, if there is anything I can do (PHP, MySQL, xHTML skills), please tell me.

Thanks very much in advance for your hard work, for investigating this critical issue and for hopefully fixing it.

Dennis

[Jeroen T. Vermeulen (jtv)]

The ddtp-ubuntu translations are large enough to be problematic. We've recently upgraded our PostgreSQL server, and we hope to replace our text search indexes with a faster alternative soon. Hopefully that will help.

[Hendrik Schrieber (hennekn)]

This is still an issue. For "universe" its basically not possible to submit translations and "main" isn't fun either. Why is this an issue now? It has worked before (a year ago or so)!

[Karma Dorje (taaroa)]

Timeout error
Sorry, something just went wrong in Launchpad.

We’ve recorded what happened, and we’ll fix it as soon as possible. Apologies for the inconvenience.

Trying again in a couple of minutes might work.

(Error ID: OOPS-996e935c8cdf8f97ae7849649a0c3413)

[William Grant (wgrant)]

I experimented with rewriting the query to use trigram indices for substring matching. The temp_* indices in https://pastebin.canonical.com/74853/ are new, https://pastebin.canonical.com/74853/ is a code diff to use them, and https://pastebin.canonical.com/74852/ is an example of the final query. It works fine (hundreds of milliseconds) for obscure strings like 'pcman', but strings with common trigrams like 'program' end up spending a couple of seconds in the GIN trigram indices. I suspect that proper FTI would be substantially faster.

[William Grant (wgrant)]

http://pastebin.ubuntu.com/1216430/ is the code diff.

[Pierre Slamich (pierre-slamich)]

Would it be dangerous to implement this partial solution ?

[Hendrik Schrieber (hennekn)]

A partial solution is better than none. At the moment, every search in DDTP just gives an OOPS. We are struggeling to get new translators to get a farely complete translation of DDTP but we can't tell anybody to try translating package descriptions if you cannot access them.

Also I wonder, why this problem came up about a year ago and it hasn't happened before? (if it's just a meaning of servers not beeing able to process search requests in time because of a lack of capacity). Was there such a great increase in server usage?

[Redmar (redmar)]

Since this bug has been around for so long, I've created a workaround. It consists of a small python program that sorts the offline ddtp po file based on popcon. This way, translators can simply start working from the beginning of the po file, and no other external tools are needed. (It also takes into account whether or not a package is present in the software center).

You can get the code with the following command, it also contains a readme file:
bzr branch lp:~redmar/+junk/ddtp_popsort

This is my first coding project for ubuntu, so please let me know if there are better way's of doing things!

[gisele perreault (gisele-perreault)]

This timeout error issue really sucks.
Even when we type or validate new suggestions, when clicking on the "save & continue" button, we often get a timeout error, and the work done on the current page is lost.

[Hannie Dumoleyn (lafeber-dumoleyn)]

Hi Gisèle,
It seems this problem is never going to be solved. If you subscribe to the ubuntu-translators mailing list, <email address hidden>, you can follow the discussion "Semi-mechanizing the DTTP translations" on this subject.
Hannie
Ubuntu Dutch Translators

[Sylvie Gallet (sylvie-gallet)]

Hi Gisèle and Hannie,

When you get a timeout error after hitting the "save and continue" button, your work is not lost, just go back to the previous page with the backspace key in your web browser, and you'll see the suggestions or translations you've made are there.

Searches in "main" eventually work, after 1 try (but sometimes for very large values of 1 :-) ), but thezy never work in "universe", and even with the semi-automated translation method we've been using for about a month, translating "universe" is a pain...

Sylvie
Ubuntu French Translators

[Hannie Dumoleyn (lafeber-dumoleyn)]

Hey Sylvie,
Thanks for the tip. When searching for something, I usually press F5 after a timeout error, to refresh the page.
Greets,
Hannie
Ubuntu Dutch Translators

[Pierre Slamich (pierre-slamich)]

Here's an excerpt from #launchpad for a little insight on that bug

teolemon: I wondered whether there was anyone with the rights to increase the timeout limit for Search, such as in this example https://translations.launchpad.net/ddtp-ubuntu/quantal/+pots/ddtp-ubuntu-universe/fr/+translate?batch=10&show=all&search=debugging+symbols+for
like giving it a bit more room to complete the query
czajkowski: wgrant: ^^
wgrant: When that page is slow, it's *really* slow. We can't sensibly increase it enough to make it succeed frequently.
teolemon: even one out of 10 would save the day
wgrant: Sure, but we'd have to at least double or possibly triple the timeout which is not hugely sensible
teolemon: i reckon being more specific increases my chances of having the query complete
wgrant: Mmm, potentially a little.
teolemon: are there any factors, time of the day that may be useful
wgrant: No
wgrant: It's just that the translation system wasn't really designed for the sort of scale that that template requires.
teolemon: In how many parts would we need to split the template to solve the issue to make search somehow work again ?
wgrant: That's not really much of a solution and I don't know a number.

[Dennis Baudys (thecondordb)]

Hi Jeroen and everybody who it concerns,

on Jeroen wrote 2012-05-24 »[…] We've recently upgraded our PostgreSQL server, and we hope to replace our text search indexes with a faster alternative soon. […]«

Did you already replace the text search indexes with a faster solution or what is the current status on that?

Thanks for the update on the progress of this critical bug (that is assigned to nobody).

Dennis

[Redmar (redmar)]

I know this bug has been annoying a lot of people, but I would just like to point out that (hopefully) soon the dttp package descriptions will be sorted based on popcon by default, which means external tools like nightmonkey will not be required anymore. This should make this bug obsolete, even if it is not solved.

[Gabor Kelemen (kelemeng)]

Redmar: that's good news, the guy behind Nightmonkey just chose life anyways :)

[Pascal De Vuyst (pascal-devuyst)]

The ddtp-ubuntu timeout only occurs for main and universe because these have too many strings, it works fine for multiverse and restricted.