RTL languages displayed incorrectly in terminal

[Yaron]

This problem is a known problem with Hebrew and there has been many attempts in the past to fix it and most of them were complete failures

My guess is that this is a kernel related bug

All CLI apps suffer from this problem in Ubuntu (And Debian and all other ditros)

I recommend cancelling the LC_MESSAGES=he_IL.UTF-8 for CLI apps while using the Hebrew locale

Maybe creating some sort of a environmental variable which only affects CLI apps

This is a very annoying bug causing a lot of frustration for the Hebrew Ubuntu users worldwide and also other distros

One of the many attempts that were made are the fribidi package but the rendering is not perfect and each program contains a rendering engine of their own, this way there is a double fix and the text is flipped twice and then the problem reoccurs

My suggestion to remove using an environmental variable which is only CLI related is the best method I could think of...

Best Regards,
Yaron Shahrabani

There's a bounty on this bug at:
http://www.fossfactory.org/project/p189

[Cyril Lavier (davromaniak)]

Thank you for submitting this bug.

I think this bug is related with the source package langpack-locales.

[Tomer Cohen (tomerc)]

What is "Hebrew is displayed incorrectly"? Can you be more specific and a screenshot to demonstrate your problem(s)?

[Yaron (sh-yaron)]

No need to attach a screenshot
this one is very simple
While using Hebrew translated CLI apps (Most of them are done by Eli
Zaretsky) the text is rendered from left to right instead of right to left

What Eli did is is a little trick to bypass this bug by translating the
entire app backwards

Example:
אובונטו
He wrote as
וטנובוא
and this way they are all displayed pretty fine

Same as for aptitude, only that aptitude was translated normally so that all
of 7.10 users have seen (while installing)
...םינותנ סיסב ארוק (...esabataD gnidaeR)
instead of
קורא בסיס נתונים... (Reading Databse...)

Right at that point we decided to revert the changes done to the po and this
way the user did not encounter any Hebrew while using aptitude but at least
there was some readable text on the screen
Our decision to revert the po file to the original English only affected
8.04 users and they can now use apt-get with no Hebrew but with readable
English which is the best thing we can offer right now...

I would love to answer further questions about the problem

[Tomer Cohen (tomerc)]

Can you give examples for applications translated in such silly way?

[Eyal Levin (eyalev)]

Screenshot attached.

Ubuntu 8.10.

[Tomer Cohen (tomerc)]

Is it Ubuntu-specific issue or it comes from the upstream?

I know similar issues and solutions.

[Eyal Levin (eyalev)]

If I'm not mistaken it's coming from Debian.

[Tomer Cohen (tomerc)]

Than it is not Ubuntu-specific. How do you think to solve it? The easiest
way is to use terminal application which support BiDi, such as mlterm.

On Mon, Feb 23, 2009 at 00:39, Eyal Levin <email address hidden> wrote:

> If I'm not mistaken it's coming from Debian.
>
> --
> Hebrew not displayed correctly in terminal
> https://bugs.launchpad.net/bugs/325324
> You received this bug notification because you are a member of Ubuntu
> Hebrew Translators, which is a direct subscriber.
>

--
Tomer Cohen
http://tomercohen.com
Sent from: Haifa Israel.

[Tomer Cohen (tomerc)]

Than it is not Ubuntu-specific. How do you think to solve it? The easiest way is to use terminal application which support BiDi, such as mlterm.

[Yaron (sh-yaron)]

Tomer... the translations are coming from Debian
And you can always suggest a solution if you have one, As you probably
already know Ubuntu is a derivative of Debian but that doesn't mean that
there are no home made solutions, there are many Ubuntu specific solutions
that with a small effort from the original devs can be submitted back to
source (until the code opening of launchpad this July)
Even if everything will be fixed as planned all we have to do is to submit
the changes from the packages to the sources manually, its pretty easy and
its already packed as tar.gz and the diff as well

The fact that Launchpad does not work with upstream doesn't mean that the
sources are not public as well

you have to make a little effort to commit them back but it is totally
possible

[Tomer Cohen (tomerc)]

Yaron - Do you mind telling me why you are talking about upstreams when the bug summary is for Hebrew messages in the console? It is possible I am misunderstanding you, but I am not sure if this is the case.

Lior Kaplan is too lazy to login and add his comment. I hope he is watching this bug. In the meantime, he requested to publish here the following posts from 2005 which talks on this subject -
http://linmagazine.co.il/node/view/8841
http://linmagazine.co.il/node/view/9820

Since Kaplan gave up implementing this in Debian, there is no need to re-implement it in Ubuntu. Since the command line itself is in English, I don't think anyone who don't understand English at all will ever use the command line.

[Diego Iastrubni (elcuco)]

Lets stop the insanity and talk facts:

1) "full screen console", tty[0-9] are handled by linux. Arabeyes were working on a a project which would mirror the display. But such daemon would break applications which handle their own BiDi, for those who don't know, it would mess the cursor position.

2) under X, *term (xterm, konsole, gnome-terminal). There are options for viewing visual hebrew as well. But the same problem would happen. If you display logical hebrew, and then the display gets rendered outside the application (a daemon, or the terminal emulator) the application would get confused and screen is getting garbeled.

If you don't beleave me, under konsole-> Settings -> "Edit current profile" -> "Advanced" -> Enable Bi-Directional text rendering" and play with it.

Please stop - it's been decided that Hebrew does not belong in Console applications. Every other solution is a hack which will bite you in the &%%^ when you play with it. Example:

dpkg --help | grep הסרה

The text we are grepping is in logical form, while dpkg will display visual. And if dpkg displays logical, it will get rendered "reversed" on screen.

Please mark this as invalid, or won't fix and stop wasting our time. We still have to translate a whole desktop, and it's not easy.

[Tomer Cohen (tomerc)]

As Kaplan said - Hebrew in console applications is disabled in Debian, and probably because of configuration error are being shipped in Ubuntu. I suggest creating a new bug for removing Hebrew support from console applications (please select carefully which packages, as some of them we may prefer to keep the language enabled) - I hope it is not too late to get it fixed in 9.04.

[Yaron (sh-yaron)]

Diego, you are my favorite!!!
All I said is that this is a bug, our miniature Israeli community does not
have enough resources to handle such a great mission

Maybe the Persian or the proffessional developers from Canonical can handle
such a mess

The fact is that today as translators we have access to the translation of
CLI apps and CLI strings in partial CLI apps which caused lots of mess in
the past and still does

No matter what the solution is, there should be one
I'm asking all participants of this bug who uses launchpad to view this bug
report to tick the "This bug affects me" on the right top corner of the
thread (right below the bars)

The English solution is temporary, don't forget that most of the Doctor's
queue management is still done on old DOS which handles Hebrew quite
correctly and even can print documents via ink and pinhead printers

No matter how temporary the solution for now will be remember that it is
temporary and that canonical gives us the appropriate resources to handles
this mission if we will try to accomplish it

[Tomer Cohen (tomerc)]

I am sorry, you wrong.

DOS never had Hebrew support and the characters set was limited to Extended ASCII (8 bit). Linux terminal is much more powerful and can support wide range of characters and styles. As for printers, the main reason you see them running on DOS operating system and using pinhead printers is because only there they have good support for Hebrew characters. Trust me, I know what I am talking about, I was administrating the computers for such companies until few years ago.

As for Linux - Please never translate programs in visual order characters, as this will cause everyone troubles when they will try more advanced things, such as showing that text inside software which is capable for showing unicode characters and has BiDi support. It is possible that in case you have dpkg translated, for example, you will see Hebrew text reversed inside update-manager and synaptic. Since you don't want to break GUI programs, you should not use visual order texts.

The previous decision in Debian have been decided by talent people. You should not change this without first asking them for advice, even if you own a distribution, since this will make bad experience for users.

You can, however, create a separate bugs for enabling BiDi support in gnome-terminal and every other console/terminal application, but until most of the application have stable BiDi support you should not place dirty hacks such as reversing text.

[Yaron (sh-yaron)]

Our only problem is lack of resourcesWith enough resources even this can be
solved

And no one talked about reversed order, its not mentioned as a solution even
once so its irrelevant to our discussion

About disabling the ability to show Hebrew translated strings (not
characters) in CLI, if you would have looked further on my report you would
notice that this is what I suggested in the first place, you only suffer
from lack of patience to read everything I type...

We have a good opportunity to solve the BiDi related bugs in terminal via
external resources, avoiding this opportunity may not be a wise decision, we
just have to be more patient and understanding towards the Canonical Crew
Members and eachother

[Diego Iastrubni (elcuco)]

Yaron:

Read again what I read. This is not something worth spending time. Even if you hack the terminal code inside the kernel to display bidi (doable!), applications will have no control over the visual location of the phisical cursor. Ok, lets add new API for that. Now you broke *all* legacy applications.

DOS is exactly VISUAL hebrew, just like the current terminal implementations have.

Now, the real issue is hidden in the huge pile of text here:

<real issue>
Hebrew (Arabic too?) translators should not be displayed POTs or POs of CLI applications.

There should be an option in *rosseta* to mark translations as "non relevant to this team". This way we can disable the command line application translation from the workflow of the translators.
</real issue>

[Yaron (sh-yaron)]

FINALLY!!!!It took us too many time to get to the initial conclusion I
reported in the first place
I would like to thank Tomer in person for making a joke out of me and
putting my ideas out of context, thank you very much... for nothing!

I hope that ends our non efficient discussion so far...

[Shlomil (shlomister)]

Yaron,
Nobody was trying to make a joke out of you. It's just a discussion and sometimes people disagree.

On this matter I have to agree with Diego - CLI applications should not be translated at all.
CLI is an environment for advanced users and those usually don't work with Hebrew translations at all.
Problem solved.

Diego's solution should translate into a bug for Rosetta - enabling you (as you are one of the admins of Ubuntu Hebrew Translators) to lock some files and keep translators away from these files. It's a shame that Rosetta is closed source because we could have fixed it by now.

Other technical solutions exists but they are not perfect:
1. Executing terminal emulators with LC_ALL=C . The downside is that GUI programs executed from CLI will also be English.
2. Creating a package that will divert all unwanted CLI translation files.
Both should be considered temporary solutions until the translation is officially removed from CLI apps.
I'm thinking of implementing #2 for the next Hebrew Remix since I find current status of Hebrew in Ubuntu's console very annoying.

Regards,
Shlomil.

[Tomer Cohen (tomerc)]

I agree with shlomil, but prefer not to install extra package just to fix issues exists because of another package.

By the way, The Firefox translation bugs are fixed in Ubuntu since I last checked? If not, this may require similar work too.

[Yaron (sh-yaron)]

Shlomi, it wasn't a disagreement, my initial notice was ignored, my
suggestion in the first place was the same as what you just wrote, thats not
a disagreement this is a total disrespect towards me
Our fellow here decided that reading the contents of the bug report is
useless, he fed up with the title, this whole discussion is in vain instead
of investing our efforts in promoting our solutions and ideas we invest them
in arguing about stuff that were written in the first place

And of course that I agree with Shlomi, because that's exactly what I said!

[Shahar Or (mightyiam)]

Dear friends,

I love you all.

Please put some more love into this matter as I think our fighting Israeli mentality can use some love.

Let's revive this issue, please.

If I understood correctly, there is agreement that the solution decided upon is:
1. Removing Hebrew translation from CLI programs.
2. Preventing CLI programs from being translated into Hebrew.

Can anyone think of ways to do these things, please?

I'm contacting the Translating Team, the QA Team and LP for help on this.

Many blessings.

[Eyal Levin (eyalev)]

I've recently opened bug 389428 hoping it would be more straightforward.

[Shahar Or (mightyiam)]

Dear friends,

If we are to consider this to be a bug somewhere, and even if we try to work around it for now, in which software is this bug, please?

I think that clearly it isn't a bug in gnome-terminal, right?

This is for the formalities and for really understanding where this comes from and where it should be addressed in the future.

Many blessings.

[Eyal Levin (eyalev)]

Hi Shahar,

Also check out this recent thread on ubuntu-translators list:

https://lists.ubuntu.com/archives/ubuntu-translators/2009-June/002564.html

Eyal

[Shahar Or (mightyiam)]

Dear Eyal,

Thanks.

I know about it. David started the discussion after I sent a message to Ubuntu about it.

Many blessings.

[Adi Roiban (adiroiban)]

Reading the upstream bug and since there is no plan how this problem should be fixed I think we can mark it as wishlisth.

Meanwhile we can talk / implement a workaround by not distributing RTL translations for CLI apps:
 https://bugs.edge.launchpad.net/ubuntu-translations/+bug/389428

[Yaron (sh-yaron)]

I would like to point out another RTL/Hebrew related bug
I suggest the Arab/Persian followers to look at it as well:
https://bugs.edge.launchpad.net/ubuntu/+source/synaptic/+bug/251705

Kind regards,
Yaron Shahrabani

[Jean-Baptiste Lallement (jibel)]

Lets rewind. What is the problem here ? We are talking about an application which does not support RTL languages called VTE. And you're trying to workaround it by removing your work! Since the terminal does not support those languages, it just has to really do so and say 'I do not support this locale, falling back to standard locale "C"' We could add a kind a 'locales blacklist' and fallback to the standard 'C' in case the current locale is part of this list.
I agree it won't fix the problem, but at least we'll have readable english instead of unreadable localized text.

PROS:
- do it once use it everywhere (or once per terminal application)
- You don't have to remove the translations and localized text is available for all the other applications. This way GUI apps calling a CLI app will be fully translated (see bug 251705 where the GUI is partly in english due to some excluded translations from dpkg)
- Relatively easy to do.
CONS:
- GUI executed from CLI will also be English.
- That won't fix the output of commands issued from the terminal e.g the output of 'ls' with filenames in an RTL language, the filenames will be mirrored.

What's your opinion? Is it worth trying to write a patch ?