RTL (right to left) support in terminal (BiDi)

[Usama Akkad]

Binary package hint: gnome-terminal

The program fails to display bi-directional text correctly. For example,
 the Arabic word Went (راح) should be spelled raa (ر) Alif (ا) haa (ح)
 from right to left. The program displays the Arabic text in the opposite direction.

examples:
ا ب ج
become
ج ب ا

if the problem with numbers (and it's not) it will look like:
123
become
321

that makes many translated terminal applications messages unreadable like dpkg & apt-get

ProblemT
ype: Bug
Architecture: i386
Date: Tue Sep 2 04:44:06 2008
DistroRelease: Ubuntu 8.04
ExecutablePath: /usr/bin/gnome-terminal
Package: gnome-terminal 2.22.1-0ubuntu2
PackageArchitecture: i386
ProcEnviron:
 PATH=/usr/locusernamel/sbin:/usr/locusernamel/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/gusernamemes
 LANG=en_US.UTF-8
 SHELL=/bin/busernamesh
SourcePackage: gnome-terminal
Uname: Linux 2.6.24-19-generic i586

[Pedro Villavicencio (pedro)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. The issue that you reported is one that should be reproducible with the live environment of the Desktop CD of the development release - Intrepid Ibex. It would help us greatly if you could test with it so we can work on getting it fixed in the next release of Ubuntu. You can find out more about the development release at http://www.ubuntu.com/testing/ . Thanks again and we appreciate your help.

[Usama Akkad (damascene)]

Please look at this:
http://bugzilla.gnome.org/show_bug.cgi?id=321490

Sorry I can't test this new release because I have dial-up and can't get it.

[Pedro Villavicencio (pedro)]

Thanks, will link the report and re assign this to vte, thanks again.

[alsadi (alsadi)]

I have submitted a patch to the upstream

http://bugzilla.gnome.org/attachment.cgi?id=114595
my comments starts here http://bugzilla.gnome.org/show_bug.cgi?id=321490#c5

the upstream developer wants a better solution so he goes with BiCon
https://www.redhat.com/archives/fedora-devel-list/2008-October/msg00636.html

till Bicon is mature enough this patch could be applied

[أحمد المحمودي (Ahmed El-Mahmoudy) (aelmahmoudy)]

Hello,

  I have prepared a package for BiCon, it is available on mentors.debian.net. Anyways, the upstream (who the same upstream for vte) told me that BiCon is not being maintained anymore !

[Ali Afshar (aafshar)]

Just to mention that this is a huge problem for me.

[Tomer Cohen (tomerc)]

This issue is also important for Hebrew speakers. This will allow us to use Finch from the terminal.

[Usama Akkad (damascene)]

this is more important than just using Finch. if you "ls" files for example you will get unreadable Arabic text. and similar thing on apt-get messages in localized environment.

[Usama Akkad (damascene)]

from upstream: https://bugzilla.gnome.org/show_bug.cgi?id=321490#c15
###
Hi,

Since the VTE does not support RLT languages, wouldn't it be possible to really
do so and say 'I do not support this locale, falling back to standard locale
"C"' ?
I agree, it is not a fix at all, but at least users would have readable english
instead of unreadable RTL language.

The counterpart would be that all application launched from the VTE will
inherit from this locale.

what's your opinion ?
###

I think this is the best thing to do for now since the upstream are not going to fix it any time soon

[Yaron (sh-yaron)]

I see where you are going but I can't understand why have you marked the dpkg bug as a duplicate, dpkg uses vte but since nothing needs to be launched via the vte of dpkg it can be set to the "C" locale without any problem...

Do you agree Osama?

[Usama Akkad (damascene)]

@Yaron

is dpkg the only program that uses vte or we will have to open a bug for every program using it?
please correct me if I'm wrong

[Yaron (sh-yaron)]

No, we don't have to open a bug for each app that uses vte but the solution with the 'C' locale does not apply to dpkg since you shouldn't launch any apps from there...
There are rare situations when apps launch graphical screens, very rare...

So the solution for the dpkg app is relatively easier than the one suggested here (the problem is that each program launched through Terminal will have the 'C' locale while in dpkg it doesn't really matter)

Agreed?

[Usama Akkad (damascene)]

I will consult the bug squad and get back to you. God willing.

[Usama Akkad (damascene)]

what do you think of mlterm, could it be used as the default for RTL languages? it can show Japanese even.
Ubuntu is for every one, right?

notice: the screen-shot is without enabling anti-aliasing

[Yaron (sh-yaron)]

I know all about mlterm, In Debian for example Lior Kaplan added mlterm as
default when the Hebrew language is installed
Same as for the Hebrew calendar
I guess mlsterm should be also installed in Ubuntu for Hebrew, Arabic and
Persian as default

I also think there should be region specific settings such as if I select my
location to Jerusalem the settings of the calendar should act accordingly
(In Israel the rest days are Friday which is a half workday and Shabbat
which is a full nonworking day)

As far as I remember, the last time I installed Evolution I had to set it
manually

What other specific settings and apps are included in the Muslim Edition of
Ubuntu?
(Besides praying time apps and Quran specific apps, Hijra calendar would be
appropriate I guess, maybe even plan a special calendar app that will
support all the special calendar forms who are not Gregorian so it will take
less space and will require less specific apps)

[Usama Akkad (damascene)]

@Yaron I agree on having the calender based on your location,
for example if some one chosen Jerusalem (Palestine) as a city he should have the Hijri and Gregorian calender installed.
(in the Islamic Hijri calender Friday is a holiday )
but this is not the subject of this bug. we might open another bug for that.

I'm thinking of writing to all RTL locale teams to get their opinion on having mlterm installed by default for the RTL locales

I've heard that bahdad the developer of vte is working on solution named harf-buzz that will allow RTL text in terminal. we should investigate this in the upstream bug.

[Yaron (sh-yaron)]

OK Usama, sounds good!

About mlterm as default in installation, maybe we should work on that non-gregorian app as well because adding 2 features as default for RTL will require twice the work
Adding them both at one shot will be more efficient

Maybe we should consider a further development of mlterm so it would be a little more gnome-terminal compliant (I'm talking about graphical interface), its only a technical work so maybe we can get some support from Canonical or GNOME for that cause

Any other ideas?

[Usama Akkad (damascene)]

there is an issue of embedding, I'm not sure if mlterm supports it.

I've contacted the developers
http://sourceforge.net/mailarchive/forum.php?thread_name=p2g1a23f0371004010418rfdb42efck329cacd33cb1140c%40mail.gmail.com&forum_name=mlterm-dev-en

[Usama Akkad (damascene)]

I've got this respond:

####
Hi,

uahello> I wonder if mlterm support embedding and if it can replace vte.

 I think the simplest way to substitute mlterm for vte is

to support XEmbed protocol and wrap embedded mlterm screen

in a GTK+ widget which provides vte compatible API functions.

 It looks interesting, but not so easy. At present I don't

have a plan to try it.
---
Araki Ken
<email address hidden>
####

http://sourceforge.net/mailarchive/message.php?msg_name=20100404.171742.74728127.arakiken%40users.sf.net

am I working alone on this? someone show some interests!

[Yaron (sh-yaron)]

What do you think we should do next?
Get financial support?
Try to contact some vte developers?
Since its a holiday now in Israel the Israeli guys will remain inactive for the next couple of days...

[Usama Akkad (damascene)]

I think pushing mlterm will fix most of the problems

vte is not going to be fixed any time soon and the developer will not change the LANG in terminal
https://bugzilla.gnome.org/show_bug.cgi?id=321490#c17

so I think we should go with mlterm

[Khaled Hosny (khaledhosny)]

Changing LANG in the terminal will do nothing about Arabic rendering, it might just avoid getting Arabic messages in the terminal but then this isn't vte issue to decide about.

[Sense Egbert Hofstede (sense)]

Can't this issue be fixed in VTE? If not, why not?

[Данило Шеган (danilo)]

Alternative "solution" to this is bug #389428: do not use CLI program translations. It wouldn't really solve the problem, but would be a workaround until the solution is found.

[Usama Akkad (damascene)]

@#26 the vte developer says that he need to rewrite vte code to fix it. and he won't.

to all others, did you hear about mlterm? why we don't support replacing vte by mlterm? it will fix 90% of the problem I guess

[Yaron (sh-yaron)]

Another solution I heard was:
Patching the code so Hebrew, Arabic and Persian won't compile for CLI only
apps
The problem begins when we want a translation for a hybrid app that has both
GUI and CLI

In Hebrew most of the hybrid apps are translated correctly besides the
strings that appear on both CLI and GUI
wget for example shouldn't be translated to any semitic language at all
because it only has CLI, adding a small patch to the Ubuntu code repository
will ignore the Hebrew/Arabic/Persian translation during the compilation
process, this is not the best suggestion if we still insist on using mlterm

The deal with mlterm is that I personally think it should be included by
default uppon the installation of any RTL language but I think it would be a
waste of time not to include the non-gregorian calendar, @Usama - this is
negotiablt of course but we should really think if we want to go through all
this twice...

mlterm is great, not doubt but the fact that it is not versatile enough
(like gnome-terminal with vte) makes it a little hard to use, if the mlterm
developer wants some money for improving this app maybe we should all try
and raise funds but if it's not about money I guess we have a problem...

I would love to hear your comments and I hope I didn't repeat anything said
earlier in the discussion

Kind regards,
Yaron Shahrabani

[Usama Akkad (damascene)]

Mlterm 3.0.0 have been released.

you can find pre-packaged mlterm for Lucid here:
https://launchpad.net/~aelmahmoudy/+archive/ppa/+packages

or you can compile it from source:
http://sourceforge.net/projects/mlterm/

in this version you can select the font you want from the GUI by holding right button and pressing Ctrl
then from the font tab press select and chose fixed width font like monospace

hope mlterm get in 10.10

[Usama Akkad (damascene)]

for people thinking mlterm is the possible fix please subscribe with this bug.
as it have been suggested by joaopinto in #ubuntu-devel to open new bug for it

Selecting an RTL language should install and RTL capable terminal emulator
https://bugs.launchpad.net/ubuntu/+source/language-selector/+bug/562130

[Alexander Sack (asac)]

this is about mlterm? we have a MIR bug 603022 pending for that. can we either invalidate this or the other bug and use the proper package?

[Usama Akkad (damascene)]

no this is not about Mlterm as long as VTE does not support RTL.

at least there should be a warning or suggestion or message inside the terminal stating that gnome-terminal does not support RTL and you should install something else, like Mlterm. that is only when a user installs RTL language Local.

that will be a great help for users before they start wondering why their texts is flipped and chopped

[Alexander Sack (asac)]

i dont see any mir team action here ... please open a dedicated MIR bug with clear info. unsubscribing mir.

[Usama Akkad (damascene)]

There is a new ALPHA (testing) workaround for this bug using MLterm API,

To test get the latest build of MLterm from sf.net, here are some instructions:

you will need this packages build-essential, mercurial (to pull source code), libvte-dev and libfribidi-dev

Start by getting the latest build:
$ hg clone http://mlterm.hg.sourceforge.net:8000/hgroot/mlterm/mlterm

Then building:
$ cd mlterm
$./configure --with-type-engines=xcore,xft --enable-fribidi
$make
$cd gtk/
$make

ATTENTION: don't forget to backup libvte.so.9 (you might use locate to find it) before continue.

Using the library from our build and linking,

$ cp mlterm/gtk/.libs/libvte.so.9.0.0 [where official libvte is installed]
$ ln -sf libvte.so.9.0.0 [where official libvte is installed]/libvte.so.9

http://mlterm.sf.net/gnome-terminal.JPG

Known issues:
* Gnome-terminal dose not start with compiz enabled
* Synaptic crashes when it should display terminal output

Please test ,report and help patching if possible. MLterm mail-list is:
https://lists.sourceforge.net/lists/listinfo/mlterm-dev-en

[Usama Akkad (damascene)]

With the latest mlterm build, gnome-terminal with enabled compiz and Synaptic works well on Ubuntu 10.04 after doing the steps above

[Shahar Or (mightyiam)]

Dear ones,

Should this be considered as default setup in RTL environments?

Why not replace original libvte altogether?

Blessings,
Shahar

[Shlomil (shlomister)]

If it works as a libvte replacement (and reasonably stable) then I think it should be default in RTL environments.
However, IMO removal of libvte upon installation is not required. I think both can live together using the Debian Alternatives System. libvte should be available with a lower priority in the Alternatives list, which will make MLTerm default when installed.

Shlomil

[uwe (maabdulhaq)]

It would be great if there is a blueprint for what the plan is regarding this issue, so we can start testing it maybe

[BenginM (sary)]

I like gnome really! but what's the point of translating Gnome to Arabic if one of the basics piece of software doesn't support the language! like really does tha make sense!

[BenginM (sary)]

+1 to uwe suggestion .

[BenginM (sary)]

KDE Konsole , to my knowledge .. seems to be the only Console that suupport RTL+BiDi .

[uwe (maabdulhaq)]

mlterm also supports rtl+bidi i guess

[bar (barhofesh)]

This helps a little : http://askubuntu.com/questions/77657/how-to-enable-arabic-support-in-gnome-terminal

[Usama Akkad (damascene)]

Tested on Ubuntu Xenial Xerus (development branch) 16.04 Gnome Terminal Ubuntu 3.18.2-1ubuntu2.

See also: bug #1537064

A related bug is bug #1537064 to fix the situation when apt and possibly other software show garbled translated scripts in terminal.

Example when running: "apt-cache policy gnome-terminal" will returned garbled text from translated script on RTL local

[Diego Iastrubni (elcuco)]

Usama,

The reason for bug #1537064 is me nagging Yaron. The problem of BiDi cannot be solved in a terminal vt100 environment. The reason - is that the application does not telling the VTE the boundaries of the paragraph.

Imagine "mc" opeenning a dialog to rename a file. Lets check the options;

1) A line contains RTL - apply BiDi to it and display line in RTL. Funny things will happen, as the line bellow will also be reverted. The file name of the right panel - will be on the left.
2) Even funnier - when you type lam and aliph - those will be displayed as a single glyph on screen and the app counted two unicode chars (4 bytes in UTF8) and now the left margin is off by one (the frames of the dialog are also text, remember - the app needs to pad spaces until the "|" char will be displayed).
3) OK - instead of reverting a whole line - lets revers each word - this is broken since you now have proper words in LTR order (lets ignore #2)
4) OK - lets detect the word boundary and de-order "n" words - when happens when the first word of the next paragraph starts with an english word? the direction will be broken.

I assume we can find other problems. Lets find solutions:
1) Do the BiDi in apps, while assuming the ternimal is dumb: OK, it will work on xterm, konsole, gnome-terminal, and VT. Problem - we need to fix *ALL* terminal apps.
2) Do the BiDi in terminals - in order to fix all above problems, we need a way to report to the terminal about the regions that will contain text that he may modify. Then code a reference implementation, issue an RFC and port to all other terminals.

TL.DR. Ternimals are broken by design and cannot support RTL/BIDI. Just move on and use X11/Wayland for Arabic/Farsi/Hebrew/Whatever and don't translate those interfaces to the native languages on terminals.

[Yaron (sh-yaron)]

So enforcing LC_MESSAGES=C or en_US.UTF-8 as a global override will ignore this issue completely.

[uwe (maabdulhaq)]

Um; It would, as we know, mostly ignore RTL reading users.

now RTL works in kde/konsole beautifully, with shaping and everything, so I'm not sure how problems mentioned in #46 are handeled there, and I'm sure its not perfect either, but works better than BiCon for example but if softwares have been using functions to count characters to know width, maybe *that* is broken, and language specific functions should be replaced with language agnostic functions ...

[Egmont Koblinger (egmont-gmail)]

Re Diego:

You are basically right. I'd like to add a couple of points.

As for complex apps such as mc, your approach (2) cannot work. This is because the BiDi algorithm would not just need to shuffle around characters that are visible in the terminal; it might need to pull offscreen chars onscreen and vice versa. E.g. think of mcview's unwrapped display mode, and long lines so that you need to scroll horizontally to read the file. In order to properly display the BiDi text, the terminal would need to be aware of the entire logical line of the file, however, the application (e.g. mcview) simply doesn't give this data to the terminal. So, for complex apps it's solely your approach (1), that is, BiDi done in the applications that can work.

On the other hand, it's a reasonable expectation that a simple "cat" produces readable text. Hence a terminal emulator should, by default, probably perform a BiDi for each logical line (that is, between explicit newlines). Which brings in a further question about how further cursor positioning or character overriding should work in this case. I've discussed it in https://bugzilla.gnome.org/show_bug.cgi?id=321490#c29, I believe #2 there would be the right thing to do (and the one that does not terribly break the look of ncurses apps).

So, we would already need two terminal modes, probably controllable via an escape sequence. Implicit BiDi (the default) is when VTE applies the BiDi algorithm (this is I guess what konsole does), which fixes simple utilities, whereas probably breaks complex apps big time. And an explicit BiDi where VTE reverts to its current behavior, that is, does not do any BiDi, to be used by apps that perform the BiDi algorithm themselves.

I _guess_ this is what ECMA TR53 (also discussed in the aforementioned link) might be about, although I haven't had the chance to study it.

[Egmont Koblinger (egmont-gmail)]

A few other issues:

In implicit BiDi mode, how should BiDi control characters (LRE/RLE/LRO/RLO/PDF; LRM/RLM) be handled? We need to remember them. Which cell should they belong to? How to later override the displayed string to remove any of these, or insert new ones?

In explicit BiDi mode, copy-pasting would reverse the strings.

Some people prefer their text editor not to switch to the alternate screen, so that e.g. upon exiting the contents of the file is still visible. However, exiting would switch off the explicit mode. Would there be one global flag for the terminal to state the BiDi mode: then the screen (showing a part of the file) be repainted with incorrect BiDi? Or would this flag be remembered for each character cell?

Also note that Arabic font rendering on fixed width grid is another quite troublesome issue.

I'm pretty sure that many other bugs would arise upon implementing a proof of concept.

I guess I'd second Diego's "TL.DR." conclusion.

[Egmont Koblinger (egmont-gmail)]

GNOME Terminal 3.33.3 (VTE 0.57.3) implements BiDi support, according to the proposal at https://terminal-wg.pages.freedesktop.org/bidi/ .

In alignment with Diego's comments and my responses to them, it implements multiple modes. Shuffling the characters according to the BiDi algorithm is enabled by default, this is required for simple utilities such as "echo", "cat" and friends. An escape sequence can turn off BiDi in the terminal, to be used if BiDi is performed by the application running inside.

As mentioned, this won't magically make fullscreen apps (e.g. "mc") BiDi-aware, it's impossible without adding this feature to all those applications.