Live CD, first screen, Hebrew not correctly displayed

[Ofir Klinger]

When booting from the live cd of Ubuntu 8.04, a menu with all available languages appear. Choosing Hebrew switches the language to Hebrew, but displays it mirrored. It seems to be only an ltr=>rtl issue, but I am not an expert...

I will attach a screenshot, if needed.

[Yotam Medini (yotam-medini-gmail)]

Same here.
It appears below 'Gujarati' and above 'Hindi'.

עברית
vs.
תירבע

[Eli Daian (elidaian)]

Same happened with me.
I thinks that it is very important to solve this bug, because this is the first impression of Ubuntu that the Hebrew users see in Ubuntu, and we don't want them to think that this system has a lot of bugs because it has a simple problem in showing Hebrew text in the right direction. It is also very important to the Israeli loco team to fix this bug, because this bug only damages the name of Ubuntu, and we are trying to show the people that Ubuntu is a stable system.
Thanks,
Eli

[Ofir Klinger (klinger-ofir)]

Please pay attention to this bug!

This bug is very important.

If the Hebrew community can help someway, please say so.

[Mark Krapivner (mark125)]

seems mandriva has a similar problem.
https://qa.mandriva.com/show_bug.cgi?id=30862

[Ofir Klinger (klinger-ofir)]

No. The problem there is not displaying the translation at all, but instead, some garbled text.

Here, the translation is displayed, but mirrored, like:
What -> tahW

[Colin Watson (cjwatson)]

Anyone in the Hebrew community who would like to try to add RTL support to gfxboot is very welcome.

In the meantime, milestoning a gfxboot-theme-ubuntu task for 8.04.1, as we will probably just have to disable the Hebrew translation (as Mandriva did).

[Ofir Klinger (klinger-ofir)]

Colin: Can you give us some directions on how we can fix this bug?
It will be best if you will guide us what to do.

[Colin Watson (cjwatson)]

The "show" primitive in gfxboot needs to be taught to detect right-to-left text and display the characters in the right order. The implementation is in x86 assembly, starting at the prim_show label in the bincode.asm file in the gfxboot source package; I think the main bulk of the text display code is at the text_xy label. Somebody will need to implement the Unicode bidirectional algorithm (http://unicode.org/reports/tr9/) there, probably a reduced form for preservation of sanity and so that it fits in the available space.

[Colin Watson (cjwatson)]

Filtered out of gfxboot-theme-ubuntu, so that it'll display in English for now; we need another upload for a font rebuild anyway, so I'm moving the milestone for this to 8.04.

[Ofir Klinger (klinger-ofir)]

So what will I see when I choose Hebrew in the Live CD (boot menu)?

Is it going to be fixed later? When?

[Colin Watson (cjwatson)]

It'll just say "Hebrew" in the boot menu, and switch to proper Hebrew text once you get into the installer.

At present I unfortunately can't commit to any kind of timeframe, much though I'd like to, as this is actually rather hard to fix and requires an unusual combination of skills.

[Ofir Klinger (klinger-ofir)]

Thanks Colin!

I think that your solution is the best one (e.g "It'll just say "Hebrew" in the boot menu, and switch to proper Hebrew text once you get into the installer.").

When the changes will be applied to the iso images?

[Colin Watson (cjwatson)]

Some time between release candidate and final release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package gfxboot-theme-ubuntu - 0.5.19

---------------
gfxboot-theme-ubuntu (0.5.19) hardy; urgency=low

  * Rebuild font based on debian-installer 20070308ubuntu39.
  * Filter out Hebrew, since gfxboot has no right-to-left text support
    (LP: #212491).
  * Don't show language menu on startup if a default language is set in the
    "lang" file.

 -- Colin Watson <email address hidden> Thu, 17 Apr 2008 03:15:50 +0100

[Shlomil (shlomister)]

IMO, Bidi algorithm implementation is not required here because we have no need to read Hebrew/Arabic user input in the boot loader. We just have to display the text in a readable manner. For this, all we have to do is reverse the text according to the bidi algorithm and display it visually. This can be achieved using fribidi tool (found in libfribidi0) at package building time and it requires no code changes or further hacking.

A proposed patch is attached.

[Shlomil (shlomister)]

As you can see in the attached image, Hebrew text is displayed correctly. I guess some string are still in English because new translation havn't been fully exported yet.

[Colin Watson (cjwatson)]

This looks pretty nice, and I'm minded to apply something similar.
Thanks! A few comments, though:

 * Could you 'bzr branch
   lp:~ubuntu-core-dev/gfxboot-theme-ubuntu/mainline bidi' (or whatever
   branch name you choose), commit your changes there, push them to a
   branch on Launchpad, and create a merge proposal? That way you can be
   properly credited in revision control history, and we can do patch
   review more easily.

 * Please use the form "(LP: #nnnnnn)" to close Launchpad bugs, and use
   the correct bug number too. :-) Forms such as "closes: Malone
   #nnnnnn" have been deprecated for several years.

 * Are you sure that this works as well for Arabic as it does for
   Hebrew? I don't read or write either language, but I've worked with
   people who do, and I'm told that Arabic script needs a shaping
   algorithm to apply connections between letters. Without shaping, my
   understanding is that it looks amateurish at best and is likely to be
   difficult to read.

 * The string equality operator in Perl is 'eq', not '==' which is for
   numeric equality.

     $ perl -le 'print "hello" if "ar" == "he"'
     hello

 * It would be better to use safe opens rather than unsafe shell
   constructs; not that I actively distrust translators or anything, but
   I don't think it's generally a good idea to pass translations through
   the shell where a misplaced double quote could cause disaster.
   Instead of messing around escaping shell metacharacters, I think it
   would be better to do something like this (completely untested):

     use IPC::Open2; # at top of program

     my $pid = open2(\*BIDI_OUT, \*BIDI_IN, 'fribidi', '--nopad', '--nobreak');
     print BIDI_IN $txt;
     close BIDI_IN;
     { local $/ = undef; $txt = <BIDI_OUT>; } # might have to fiddle with trailing newlines
     waitpid $pid, 0;

Could you adjust your patch along these lines? Thanks again.

[Shlomil (shlomister)]

hi,

1. I Committed a branch called "hebrew-fix" using bzr in my own user branches. I could not commit it under ubuntu-core-dev. I guess I don't have the permissions to do so:

$ bzr push sftp://<email address hidden>/~ubuntu-core-dev/gfxboot-theme-ubuntu/hebrew-fix
bzr: ERROR: Permission denied: "/~ubuntu-core-dev/gfxboot-theme-ubuntu/hebrew-fix": [Errno 13] mkdir failed
$ bzr push sftp://<email address hidden>/~shlomister/gfxboot-theme-ubuntu/hebrew-fix
Created new branch.

Is that OK? Can you merge it like this?

2. Used LP: ##### form this time.

3. Not sure it works as well for Arabic (I know some basics of the language but not enough to be a tester) so for now I will only apply the patch for Hebrew.

4. You are right... I forgot I have to use 'eq' - my perl skills are a little rusty.

5. You are right .. again - passing these strings in shell in not secure nor stable. I changed it to use open2. There are no trailing newlines to fiddle with but, as you might have noticed in my first patch, the bidi algorithm converts C format string parameters from %s to s% and that is fixed by that little additional regex over there.

Thanks for the comments and guidance.

[Steve Langasek (vorlon)]

> * Are you sure that this works as well for Arabic as it does for
> Hebrew? I don't read or write either language, but I've worked with
> people who do, and I'm told that Arabic script needs a shaping
> algorithm to apply connections between letters. Without shaping, my
> understanding is that it looks amateurish at best and is likely to be
> difficult to read.

Yes, Arabic glyphs do need shaping handling; fortunately, the shaping algorithm is very simple (the shape is determined almost entirely by the letter's position within the word) and separate Unicode codepoints are available for each of the glyphs, so "pre-composed" strings are possible. IMHO it's probably worth making sure the text is right-way-round as a first step, even if the shaping still needs to be implemented.

[Shlomil (shlomister)]

> IMHO it's probably worth making sure the text is right-way-round as a first step,
> even if the shaping still needs to be implemented.

The proposed patch will fix Arabic text direction but with no char shaping (as can be seen in the attached screenshot in which I enabled Arabic again).

It seems that shaping can be achieved by using a newer version of fribidi (latest is 0.19.2).
There is a LP bug for including the newer fribidi version (LP bug #191241) in Ubuntu and Debian (is there a Debian bug for it?).
BTW, I think that shaping can be done using other libs (ICU? , Pango?). Any clue?

I built fribidi 0.19.2 on my system using Tzafrir's package ( http://tzafrir.org.il/~tzafrir/fribidi/ ) and rebuilt gfxboot Ubuntu theme again with shaping. This time there was no text displayed at all. (see second attached screenshot).

It seems that gfx-theme-ubuntu uses X keymap chars (I think) to build the bitmap fonts. But the joined chars do not exists in the keymaps because shaping occurs automatically under X. So I guess that a better way to get a list of all Arabic chars is required. I don't know how to do that and if you have an idea please share.

By the way, there is another LP Bug #119393 that is about Arabic char shaping ...
So I suggest accepting my branch and closing this bug and deal with the shaping issue described in LP bug #119393.

[Launchpad Janitor (janitor)]

This bug was fixed in the package gfxboot-theme-ubuntu - 0.9.1

---------------
gfxboot-theme-ubuntu (0.9.1) lucid; urgency=low

  [ Colin Watson ]
  * Remove "Safe graphics mode" message, which is no longer used.
  * Add nomodeset to "Other options" menu.
  * Update language names from localechooser 2.12ubuntu3, adding Estonian
    (LP: #496734) and using the localised name for Hebrew now that we can.

  [ Shlomi Loubaton ]
  * Added back Hebrew support by using visual text (LP: #212491)
 -- Colin Watson <email address hidden> Fri, 12 Feb 2010 13:27:08 +0000

[Shlomil (shlomister)]

I tested the daily lucid build and Hebrew is displayed fine in the boot menu but the language name (which appears in the list of languages to pick from) is reversed. in previous versions is was labeled as "Hebrew" but now its labeled as תירבע again which means "Hebrew" written backwards.

This was probably caused by Colin's change to use the localized name again.

Possible solution:
1. We should leave it as "Hebrew" in list
or
2. Make the packaging script reverse the Hebrew name as well.

[Shlomil (shlomister)]

In the file langnames.inc , line 25 is :
[ "he" "he_IL" "עברית" ]

http://bazaar.launchpad.net/~ubuntu-core-dev/gfxboot-theme-ubuntu/mainline/revision/398

We should change it to תירבע (reverse the word).
Should I create a bzr branch for such a tiny fix?

is this file auto-generated somewhere?

[Colin Watson (cjwatson)]

Incomplete means that the bug needs more information from the submitter, not that the fix is incomplete. :-)

Yes, this is autogenerated by scrape-localechooser.

bzr branches are always helpful, but in this case it's OK, I have it covered. Thanks for the check!

[Launchpad Janitor (janitor)]

This bug was fixed in the package gfxboot-theme-ubuntu - 0.9.2

---------------
gfxboot-theme-ubuntu (0.9.2) lucid; urgency=low

  * Use fribidi for Hebrew in the language list as well (LP: #212491).
 -- Colin Watson <email address hidden> Mon, 22 Feb 2010 12:12:36 +0000

[Shlomil (shlomister)]

After testing today the daily snapshot I can confirm that this issue is resolved.
Thank you Colin and everybody else who helped.