Default Locale-Based Chinese Mode Setting is Inconvenient to Hong Kong Users

[Anthony Wong]

BACKGROUND
en_HK is popular among Hong Kong GNU/Linux users. However, according to current practice of IBUS, Simplified Chinese mode would be used by default. This is very annoying for these users.

PINYIN CANGJIE DIFFERENCE
Pinyin for Simplified and Traditional Chinese characters are the same. If a Traditional Chinese user meets Simplified Chinese mode, she can still express her meaning in most cases. However, Simplified and Traditional Chinese characters generally do not have same Cangjie code. A Traditional Chinese user would meet a very hard time in Simplified Chinese mode because she cannot get expected character by typing her familiar Cangjie code.
BTW, Cangjie and related Quick is still very popular in Hong Kong.

UPSTREAM BUG
https://github.com/acevery/ibus-table/issues/9

UPSTREAM FIXES
https://github.com/acevery/ibus-table/commit/487d9b6825492498cfc1f49b66e243c2eb4c5bcf
https://github.com/acevery/ibus-table/commit/d4df7ed4c361a30f9c5d505e9ffc8b5fc7c163a7
https://github.com/acevery/ibus-table/commit/a44d50c6face110c7cd1252067eade3f8a9193e8

[Anthony Wong (anthonywong)]

Back to New according to Freeze Exception Process.

[Roy Chan (roy-chan-linux)]

I try to provide more information about the bugs and the affect to HK people. Although most Hong Kong people are Chinese, most of them usually use English for work as Hong Kong was a colony of UK for over 100 years. Therefore, a big number Hong Kong people are more familiar with English interfaces on their computer even that we have locale zh_HK (Chinese (HK)). However, while using the English interface as default, they would like to read Chinese document/web page and input Chinese in the English interface. Such a using habit is common in Windows, Mac and GNU/Linux environment. And en_HK is the locale that such a kind people usually use.

For locale zh_HK, ibus-table use Traditional Chinese as the default Chinese mode. But for non zh locale (e.g. en_HK) , ibus-table will use Simplified Chinese as the default Chinese mode if the active input method is Chinese input mode. The Simplified Chinese mode restricted the input method can only input Simplified Chinese character which make many HK people confused.

Unfortunately, such a case is worse in both Ubuntu and Debian compared with Fedora. In Fedora 16, user click on the ibus icon on the indicator bar to show the current Chinese mode and switch the Chinese mode as well as switching the Input Method. In Ubuntu and Debian, clicking the ibus icon on indicator bar can only switching the input method. User have no way to know what Chinese mode they are on in Ubuntu and Debian except configure the ibus to show the language panel to always use in ibus-setup. Such an issue mis-leading that many Ubuntu/Debian use think that ibus cannot input Traditional Chinese.

As Ubuntu 12.04 LTS is a very important release, we wish that such a bug can be fix in these release to allow HK Ubuntu lover to promote Ubuntu in HK easier.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[wanleung (wanleungwong)]

The bug affects me for a long time. Also, it is hard to promote Ubuntu for those newbie like to use English interface but have to input Traditional Chinese in Hong Kong. As this could not be happened on Windows and Mac OSX.

If we would like to fix the bug #1, the default environment should be easy for newbie to use.
However, this bug makes those newbie think Ubuntu is hard to use because the only way to fix this iBus bug in English environment is to changing the iBus code by their own.

[Ric (deryckchan)]

I've previously reported the exactly the same problem at
https://bugs.launchpad.net/ubuntu/+source/ibus-table-cangjie/+bug/701319
No reply was received there. We could merge that bug report into this one.

As I said there, a workaround is currently offered by Pigeonman's first reply at
https://bugs.launchpad.net/ubuntu/+source/ibus-table-cangjie/+bug/652841

However, that's a workaround, not a fix. The real problem is that it is *impossible* to type traditional Chinese in Ubuntu 10+ without tweaking with config files, not just that it isn't the default.

[Roy Chan (roy-chan-linux)]

I attach a screenshot of iBus in Fedora 16 to show that it allow user to know which Chinese mode they are in and allow them to switch it:

[Roy Chan (roy-chan-linux)]

However, even in the current beta of Ubuntu 12.04 LTS, there is no way for user to aware the Chinese mode exists nor switch it:

[Sebastien Bacher (seb128)]

Thank you for your bug report, I'm not writing chinese and unsure to understand the issue though:

- why is ibus behaving differently in Fedora than Debian?
- isn't the screenshot in the previous comment showing the keyboard switcher rather than ibus (the input method menu)?
- is the issue in ibus-table?
- did anyone try those 2 commits? are they enough to fix the issue?

[Isaac (v1)]

@Roy on Ubuntu the only way to show the Chinese mode is to set "Show language panel" to "Always".

[Isaac (v1)]

The fix in ibus-table should resolve the issue experienced by peoples type traditional Chinese under en_HK (including me).
Which makes Ubuntu works out-of-the-box without applying any workarounds manually.

[wanleung (wanleungwong)]

- why is ibus behaving differently in Fedora than Debian?
No idea.

- isn't the screenshot in the previous comment showing the keyboard switcher rather than ibus (the input method menu)?

the fedora screenshot is different from the ubuntu one. The second row of the fedora one shows which mode (Simplified Chinese or Traditional Chinese) of the output is. The mode is a filter to filter out the words and only shows the words which are in that mode. The mode choosing row is not in the ubuntu. This is noting about the input method, it is just about the output mode.

For example, 戰(Traditional) and 战(Simplified) are the same word in Chinese, the input method PinYin are using the same code to represent those words. So, the iBus engine itself has a word filter to filter out this 2 words, if you are in Simplified mode, it will filter out the Traditional one and give out simplified. Which the flow is make scene in PinYin in Mainland China as they use simplified Chinese, but not in HK and Taiwan.

As en environment on default is at simplified mode because the current code is only choose to use Traditional Chinese mode when seeing locale is zh_HK or zh_TW. And as the UI does not have a place to choose which mode I would like to use, we cannot type Traditional Chinese in en_HK or zh_TW locale as the ibus-table thinking we are using Simplified Chinese but actually it is not.

If we use other input method such as Cangjie, although 戰(Traditional) and 战(Simplified) are the same word in Chinese, they have different input code, 戰 is "rji" and 战 is "yri". If ibus is at Simplified Chinese mode, it doesn't give me a word 戰 even if I type a right cangjie code "rji" as the Simplified mode will filter out all the Traditional words.

- is the issue in ibus-table?
Yes. the problem is inside "engine/table.py"

- did anyone try those 2 commits? are they enough to fix the issue?
Yes, I have tried it. they are enough to fix the issue but not the perfect one.

The issues had been there for a few years when iBus come out.
Most of the people, when they were facing this problem, will change to use scim and gcin.
But as scim and gcin are going to remove from the upstream, this problem is now coming up again and becoming a big concern.

[Sebastien Bacher (seb128)]

https://github.com/acevery/ibus-table/commit/d4df7ed4c361a30f9c5d505e9ffc8b5fc7c163a7#diff-0 doesn't seem to apply to the current version ... how did you test it? could you add your diff here?

[wanleung (wanleungwong)]

Here is my quick fix patch.

But of course, it is not a perfect fix as I told before.

In Cangjie Input Method, although it can now give out Traditional Chinese result, ibus-tables will filter out Simplified Chinese Result when user types a Cangjie Code that mapping to Simplified Chinese Table.

But this fix can solve the problem that people in Taiwan are facing, and 99% users usage in Hong Kong.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "quickfix.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Sebastien Bacher (seb128)]

to what source do you apply that? ibus-table in precise has no 'zh_' anywhere in its source which is the first line the patch listed there and in git tries to delete...

[wanleung (wanleungwong)]

Source is from

http://packages.ubuntu.com/precise/ibus-table

and apply this diff

https://github.com/acevery/ibus-table/commit/d4df7ed4c361a30f9c5d505e9ffc8b5fc7c163a7

into the source

[Anthony Wong (anthonywong)]

@seb128

Sorry for the wrong order of the commits. There are should be three commits which I corrected in the bug description.
For your convenience you can refer to the attached patch, which composes of the three commits.

[Anthony Wong (anthonywong)]

Hey wanleung, to the problem you outlined in #13, which ibus will filter out Simplified Chinese, we need to talk to upstream developers to let him understand the problem as it is likely a design flaw in ibus-table, and upstream is where the problem should be fixed once and for all.

[wanleung (wanleungwong)]

Yes, it is about the design problem of the ibus-table.

As inside the table.py around line 126, there is a comment of the return value of the filter
        # self._chinese_mode: the candidate filter mode,
        # 0 is simplify Chinese
        # 1 is traditional Chinese
        # 2 is Big charset mode, but simplify Chinese first
        # 3 is Big charset mode, but traditional Chinese first
        # 4 is Big charset mode.
        # we use LC_CTYPE or LANG to determine which one to use

The filter make sense in any kind of PinYin input methods such as 拼音, 注音, 粵拼,
but it will give out a strange behavior in all kinds of graphological aspect input methods such as倉頡, 速成, 大易, 筆順.

It doesn't make sense to use locale to define which filter mode should be use in graphological aspect input methods.
The filter mode should be defined by not only locale but also the input method.
As we can see the filter mode actually has Big charset mode which can apply to all graphological aspect input method.

That's why I said the current patch is not a perfect fix of the issues.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ibus-table - 1.3.9.20110827-1ubuntu1

---------------
ibus-table (1.3.9.20110827-1ubuntu1) precise; urgency=low

  * debian/patches/en_hk_fix.patch:
    backport 3 upstream commits to fix the bug "Default Locale-Based
    Chinese Mode Setting is Inconvenient to Hong Kong Users", (lp: #947908)
    thanks Anthony Wong for reporting the issue and doing the backport
 -- Sebastien Bacher <email address hidden> Thu, 08 Mar 2012 22:36:11 +0100

[sonya (sonya-huang)]

I filed a bug upstream over a year ago about this issue at ibus' Google Code project page: http://code.google.com/p/ibus/issues/detail?id=1188

Personally I think a better solution is to separate traditional and simplified cangjie into two input methods called cangjie-traditional and cangjie-simplified. This is how Mac OS X treats them and it is very straightforward. If I have time I will fork the project on github, but not if someone here gets to it first.

[Adrian (adrian-alexander-may)]

That isn't it I'm afraid. I just hacked that get_chinese_mode function to return 1 all the time but still I ony get simplified. I already had the aforementioned fix on my box cos I updated it today. My box basically has to speak English cos I am English, but I live here and need to type chinese sometimes. I can see traditional characters in firefox and in a shell. Maybe the clue is in here somewhere:

ad@bluefin:~$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
ad@bluefin:~$ locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
POSIX
zh_CN.utf8
zh_SG.utf8
zh_TW
zh_TW.big5