scim-chewing: user database file can be corrupted, which render the whole package useless

[Paul Sladen]

The version of scim-chewing in dapper, 0.2.1, has a bug that can cause the user's database file be corrupted when scim exits abnormally. Once the database file is corrupted, the user can not use scim-chewing anymore.

This bug is fixed in scim-chewing 0.3.1, but the bugfix can't be easily backported to 0.2.1, as it's a rewrite of the database-handling code.

Therefore this bug should have been fixed in edgy (edgy has scim-chewing 0.3.1-1ubuntu3), and affects dapper only.

The Debian maintainer of scim-chewing suggests backporting version 0.3.1 to dapper.

[Paul Sladen (sladen)]

<AndrewLee> sladen: It was mentioned as a known problem in README.Debian

[Andrew Lee (andrew-linux)]

Seems the im-switch's conffile use qtimm instead of xim, so it cause all 3rd-party qt application(included skype) doesn't work with this package.

Please change it back to XIM in everywhere, cause it always
works (in Linux).

There is a related bug report in debian:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=349397

[Andrew Lee (andrew-linux)]

Hey guys,

It has been 10 days past, our users are still waitting for the fix.
Any progress?

What changes I expected should made for the fix:
1) Upgrade libchewing to 0.3.0, and scim-chewing to 0.3.1
This will solve the "connot enter any Chinese character issue"

2) Chnages in im-switch's config file:
-QT_IM_MODULES=scim
+QT_IM_MODULES=xim

Cause the scim varint does not work very well... lots of apps and all kind of 3rd-party qt apps can not deal with that, but xim is always fine.

Cheers,

-Andrew

[ZhengPeng Hou (zhengpeng-hou)]

To solve this bug:
 1 backport scim-chewing from edgy back to dapper
 2 revert scim-qtimm to preview release

2006/8/30, Andrew Lee <email address hidden>:
> Hey guys,
>
> It has been 10 days past, our users are still waitting for the fix.
> Any progress?
>
> What changes I expected should made for the fix:
> 1) Upgrade libchewing to 0.3.0, and scim-chewing to 0.3.1
> This will solve the "connot enter any Chinese character issue"
>
> 2) Chnages in im-switch's config file:
> -QT_IM_MODULES=scim
> +QT_IM_MODULES=xim
>
> Cause the scim varint does not work very well... lots of apps and all
> kind of 3rd-party qt apps can not deal with that, but xim is always
> fine.
>
> Cheers,
>
> -Andrew
>
> --
> scim-chewing cannot enter any Chinese character
> https://launchpad.net/bugs/57081
>

[Andrew Lee (andrew-linux)]

Should it put in dapper-backport or dapper-update?

1. Could you please show me the im-switch config file of scim-chewing in edgy before doing the update or backport? I want to make sure it use XIM.

2. I still think the scim-qtimm would brings problem, not solving any problem, and xim is always work.

-Andrew

[ZhengPeng Hou (zhengpeng-hou)]

conffile of scim-chewing for im-switch in edgy:

XIM=SCIM
if [ -e /usr/bin/skim ]; then
    XIM_PROGRAM=/usr/bin/skim
else
    XIM_PROGRAM=/usr/bin/scim
fi
XIM_ARGS="-d"
if [ -e /usr/lib/gtk-2.0/2.4.0/immodules/im-scim-bridge.so ]; then
    GTK_IM_MODULE=scim-bridge
else
    GTK_IM_MODULE=xim
fi
if [ -e /usr/lib/qt3/plugins/inputmethods/libqscim.so ]; then
    QT_IM_MODULE=scim
else
    QT_IM_MODULE=xim
fi
DEPENDS="scim | skim"

[ZhengPeng Hou (zhengpeng-hou)]

I hope scim-pinyin can be backported back to dapper, and will solve some issue under zh_CN locales.

[Andrew Lee (andrew-linux)]

I think QT_IM_MODULE=scim should go away in dapper, and use xim instead.

Let me mention this again, the qt patch and qtimm in dapper is incompatible with 3rd-party qt apps, if you want to use qtimm, you shuold wait untail qt support it directly(impossible in dapper).

-Andrew

[Andrew Lee (andrew-linux)]

Dear freeflying,

For scim-pinyin issue, you shuold create another bug to scim-pinyin, here is for scim-chewing.

-Andrew

[Andrew Lee (andrew-linux)]

22:46 <@freeflying> AndrewLee: I've mailed Matt mermman , poke them approving
                    backports or updates in dapper
22:46 <@freeflying> s/Matt mermman/ mdz

[Andrew Lee (andrew-linux)]

Hi guys,

It's another week past since freeflying said he sent mail to dmz, any progress?

-Andrew

[Matt Zimmerman (mdz)]

We have exchanged some emails (slowly), but his explanation of the problem is not very clear to me.

If there is a patch in Edgy which you would like to propose for inclusion in Dapper, the process is documented here:

https://wiki.ubuntu.com/StableReleaseUpdates

One of the requirements is attaching the proposed patch to the bug so that it can be reviewed. I'm subscribed to this bug now and will monitor it.

[Andrew Lee (andrew-linux)]

Sorry, I felt so tired to following this bug, I don't think I would have time to following the document. I just wish to help but seems it's so hard for me.

I know it's so complicated to explain such mess, I think that has been wrong since the im-switch integration in dapper.

And scim-chewing 0.2.1 has a known bug in upstream, they got the problem solved since 0.3.x
And dapper took the buggy 0.2.1 with a messed im-switch configuration file.

So that become two problems and both results are made users cannot type Chinese.

-Andrew

[ZhengPeng Hou (zhengpeng-hou)]

To:AndrewLee.
  the problem you mentioned that scim-chewing-0.2.1 can not input any characters in dapper, was not caused by the connfile for im-switch, it was the bugs of scim-chewing itself. as to guys saying can not input anything in kde/qt applications under ubuntu, it was the lack of scim-qtimm. As to can not input anything in skype/opera, we shall know that these stuffs were not supportted in ubuntu, some bugs filed relate with these stuffs were rejected already. As I pointed out early, to solve these problems, we need backport relate packages such as scim-chewing, scim-pinyin from edgy, or we use xim defaultly for scim in dapper.

[Matt Zimmerman (mdz)]

On Wed, Sep 20, 2006 at 03:52:59AM -0000, ZhengPeng Hou wrote:
> To:AndrewLee.
> the problem you mentioned that scim-chewing-0.2.1 can not input any characters in dapper, was not caused by the connfile for im-switch, it was the bugs of scim-chewing itself. as to guys saying can not input anything in kde/qt applications under ubuntu, it was the lack of scim-qtimm. As to can not input anything in skype/opera, we shall know that these stuffs were not supportted in ubuntu, some bugs filed relate with these stuffs were rejected already. As I pointed out early, to solve these problems, we need backport relate packages such as scim-chewing, scim-pinyin from edgy, or we use xim defaultly for scim in dapper.

Since scim-chewing has not changed since 2006-04, this problem must be at
least 5 months old. How is it that it wasn't reported and fixed prior to
the 6.06 LTS release?

--
 - mdz

[Andrew Lee (andrew-linux)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt Zimmerman wrote:
> Since scim-chewing has not changed since 2006-04, this problem must be at
> least 5 months old. How is it that it wasn't reported and fixed prior to
> the 6.06 LTS release?

I wonder that too!!! How is it that is wasn't changed after I noticed
ubuntu member when I preparing the upload for Debian. I have
got/heard/seen our users reported/complained/asked by mail/irc/forum
many times, but ubuntu members don't? I tried notice this again so this
bug number has been created on 2006-08(Thanks Paul). But how come it
still wasn't fixed until now?

I know you guys is not using scim-chewing for everyday life, but our
user does. They depend on this for input their mother language. Please
think about this in users side.

Don't make me cannot believe this is a Linux for human beings....please.

- -Andrew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFEOwynQYz4bYlCYURAnmkAKCnC6pSjhFRNdRhSq/CPXto2mUQdgCeLKQF
TfDDKuxHrde75UE9xMij37Q=
=II5H
-----END PGP SIGNATURE-----

[Enrico Zini (enrico)]

After a chat with Andrew Lee (upstream maintainer of scim-chewing), I'll try to summarise the situation as concisely as I can:

Finding the bug:

  The bug has been found right after the release of Dapper, when skype stopped
  accepting chinese input right after the upgrade from Hoary.

  It has not been reported right away because Andrew Lee was trying to
  investigate what was wrong.

  As to why more people didn't report this bug, I have a few ideas which I
  think are not much relevant to the discussion.

Cause of the bug:

  The cause of the bug is that Dapper uses scim-qtimm, which requires a patch
  to libqt in order to work. The patch changes the API/ABI of libqt.

  Applications built with the patched libqt work fine with scimm-qtimm, but all
  3rd-party applications that weren't built with the patched libqt do not work
  with scimm-qtimm.

Proposed work-around:

  The work around is not use scim-qtimm in the im-switch configuration file,
  but use xim instead.

Further issues:

  scim-0.2.1 in dapper has a nasty bug: the user's phrase database file gets
  corrupted when scim does not exit gracefully. After this happens, it's
  impossible to input Chinese.

  The bug is fixed on 0.3.1. It's unfortunately hard to backport the patch,
  because the fix consisted in rewriting the code that handles the user
  database.

[Matt Zimmerman (mdz)]

On Wed, Sep 20, 2006 at 08:55:24AM -0000, Enrico Zini wrote:
> After a chat with Andrew Lee (upstream maintainer of scim-chewing), I'll
> try to summarise the situation as concisely as I can:

Thank you so much. Your explanation has helped me to reach a breakthrough
understanding of the situation.

> Finding the bug:
>
> The bug has been found right after the release of Dapper, when skype stopped
> accepting chinese input right after the upgrade from Hoary.

I assume you mean 'from Breezy' here.

> It has not been reported right away because Andrew Lee was trying to
> investigate what was wrong.
>
> As to why more people didn't report this bug, I have a few ideas which I
> think are not much relevant to the discussion.
>
> Cause of the bug:
>
> The cause of the bug is that Dapper uses scim-qtimm, which requires a patch
> to libqt in order to work. The patch changes the API/ABI of libqt.
>
> Applications built with the patched libqt work fine with scimm-qtimm, but all
> 3rd-party applications that weren't built with the patched libqt do not work
> with scimm-qtimm.

OK, so our libqt includes said patch, and this problem is limited to
third-party Qt applications not distributed with Ubuntu?

> Proposed work-around:
>
> The work around is not use scim-qtimm in the im-switch configuration file,
> but use xim instead.
>
> Further issues:
>
> scim-0.2.1 in dapper has a nasty bug: the user's phrase database file gets
> corrupted when scim does not exit gracefully. After this happens, it's
> impossible to input Chinese.
>
> The bug is fixed on 0.3.1. It's unfortunately hard to backport the patch,
> because the fix consisted in rewriting the code that handles the user
> database.

My input method vocabulary is a bit limited; here is what I think I know:

scim - input method platform used in Ubuntu
scim-qtimm - input method plugin for Qt which talks to SCIM
scim-chewing - the default Chinese input method used in Ubuntu
xim - an X protocol for input methods, used by SCIM
im-switch - a framework which selects input methods at login, based on locale

If I apply these definitions to what you've said, here's my current
understanding:

- Most programs talk to SCIM via the X server's XIM protocol

- GTK and Qt programs may instead use a plugin which talks to SCIM by some
  other means

- In Dapper, scim-chewing instructs im-switch to select the following input
  method settings for Chinese:

  - Run SCIM in the user's session
  - Tell GTK to use the native SCIM input method module (scim-gtk2-immodule)
  - Tell Qt to use the native SCIM input method module (scim-qtimm)
  - X programs not using GTK or Qt will use XIM, which should still reach
 SCIM (right?)

- scim can't be used with Qt applications unless a patched Qt and scim-qtimm
  are used, so this configuration doesn't work with those. This applies to
  all Qt applications when SCIM is in use, and this is the bug described in
  the original report here.

- Ubuntu does not ship with scim-qtimm, so even if the previous problem were
  fixed, Chinese input in Qt programs in Ubuntu doesn't work.

- Kubuntu does not ship with scim-gtk2-immodule, so Chinese inp...

[Wen-Yen Chuang (caleb-calno)]

Just for more background information. :-)

1. About XIM and GTK/QT immodules
GTK/QT immodules are better then XIM in many aspects. However, scim's GTK/QT immodules have ABI discompatibility issue, which is due to the C++. Other IM servers written in C have no this problem.

Scim have its C-rewritten GTK/QT immodules, scim-bridge. Scim-bridge have no C++ related problem, and is almost as stable as XIM.

For most people, GTK/QT immodules are good, but scim users should use scim-bridge instead of scim-gtk2-immodule. Dapper's scim-qtimm have its own patch, and is not suitable for end users (It can not work with skype).

Use XIM is the safest bet. For UTF-8 (Ubuntu's default encoding) users, XIM can satisfy most people's needs.

2. GTK/QT immodules' benefits
a. user needs not export or execute anything before using IM server.
b. user may select his/her own IM server runtime. Just right click and choose, then applications will invoke IM server automatically.
c. user may input anything under any locale setting. GTK/QT immodules work in all locales, even in POSIX. XIM only works in specific locales, and is highly locale-dependant.

Scim's XIM mode supports all UTF-8 locales. It is enough for a default IM server.

3. About scim-chewing and libchewing2/libchewing3
Dapper's scim-chewing is libchewing2-based. It works, but crashs very often. That is why no people report bugs for "scim-chewing does not work". It do works, but is very unstable.

However, upstream did not release patches for this. Upstream just released API-discompatible new versions, libchewing3 and new scim-chewing.

That is why it can not be fixed by simple backporting. New scim-chewing is libchewing3-based, while dapper has only libchewing2. Dapper must have libchewing3 before using new scim-chewing.

No other applications uses libchewing2/3, so it is safe to introduce libchewing3 and new scim-chewing into dapper. But it is a big change for a stable distribution, I think.

4. Dapper's default im-switch setting
Dapper's default setting uses scim's GTK/QT immodules. However, default installation have no scim-qtimm, so nobody can use scim in QT-based applications.

Scim-gtk2-immodules have many problems, and is not suggested by upstream (upstream suggests XIM or scim-bridge). However, dapper's default is scim-gtk2-immodule.

The simplest way to fix this, is setting XIM as default. It should be done in Ubuntu 6.06.2.

[Ming Hua (minghua)]

On Thu, Sep 21, 2006 at 01:29:43AM -0000, Matt Zimmerman wrote:
> On Wed, Sep 20, 2006 at 08:55:24AM -0000, Enrico Zini wrote:
> > After a chat with Andrew Lee (upstream maintainer of scim-chewing), I'll
> > try to summarise the situation as concisely as I can:
>
> Thank you so much. Your explanation has helped me to reach a breakthrough
> understanding of the situation.

Enrico and Matt, thank you both for taking time to help on the input
method issues.

I am not currently following the development of Ubuntu edgy. But I was
involved in the dapper development, and I'll pitch in what I know about
these things.

[snipped]
> > Cause of the bug:
> >
> > The cause of the bug is that Dapper uses scim-qtimm, which
> > requires a patch to libqt in order to work. The patch changes the
> > API/ABI of libqt.
> >
> > Applications built with the patched libqt work fine with
> > scimm-qtimm, but all 3rd-party applications that weren't built
> > with the patched libqt do not work with scimm-qtimm.
>
> OK, so our libqt includes said patch, and this problem is limited to
> third-party Qt applications not distributed with Ubuntu?

That's true as far as I understand.

> > Further issues:
> >
> > scim-0.2.1 in dapper has a nasty bug: the user's phrase database
> > file gets corrupted when scim does not exit gracefully. After
> > this happens, it's impossible to input Chinese.
> >
> > The bug is fixed on 0.3.1. It's unfortunately hard to backport
> > the patch, because the fix consisted in rewriting the code that
> > handles the user database.
>
> My input method vocabulary is a bit limited; here is what I think I
> know:
>
> scim - input method platform used in Ubuntu
> scim-qtimm - input method plugin for Qt which talks to SCIM

I believe the correct description is "a plugin that Qt apps talks to,
which handles user's input in the app, and then talks to lower-level Qt
libraries to deal with these (maybe already modified) input". But I am
not positive that I understand the technical part very well.

You probably also want to know about scim-gtk2-immodule, which is the
counterpart plugin on GTK+ side.

> scim-chewing - the default Chinese input method used in Ubuntu

Traditional Chinese (Taiwan) input method. Simplified Chinese (Mainland
China) use different input methods.

> xim - an X protocol for input methods, used by SCIM
> im-switch - a framework which selects input methods at login, based on locale
>
> If I apply these definitions to what you've said, here's my current
> understanding:
>
> - Most programs talk to SCIM via the X server's XIM protocol

My (limited) understanding says that programs using XIM protocol doesn't
talk to SCIM directly, they talk to some kind of general "XIM server",
of which SCIM is one. The X server take the user input and give them to
the XIM server, then take the output of the XIM server and give them to
lower-level X libraries to process.

> - GTK and Qt programs may instead use a plugin which talks to SCIM by some
> other means

Yes, essentially GTK and Qt has their own input method framework which
replaces XIM protocol (although they can use XIM in their framework).

> - In Dapper...

[Wen-Yen Chuang (caleb-calno)]

Just clear one point which Ming Hua mentioned. :-)

> > - X programs not using GTK or Qt will use XIM, which should still reach
> > SCIM (right?)
> By what I have seen, no. scim-chewing package doesn't seem to set the
> proper environment (the XMODIFIERS env variable) for XIM, so I don't
> think the non-GTK/Qt X programs will work if the im-switch setting is
> done by scim-chewing.

The first line in
/etc/X11/xinit/xinput.d/scim-chewing, "XIM=SCIM", will do the magic. It is very common for im-switch's conffile to use "XIM" instead of "XMODIFIERS". It do work in X programs not using GTK or Qt.

[Ming Hua (minghua)]

On Fri, Sep 22, 2006 at 02:19:02AM -0000, Wen-Yen Chuang wrote:
> Just clear one point which Ming Hua mentioned. :-)
>
> > > - X programs not using GTK or Qt will use XIM, which should still reach
> > > SCIM (right?)
> > By what I have seen, no. scim-chewing package doesn't seem to set the
> > proper environment (the XMODIFIERS env variable) for XIM, so I don't
> > think the non-GTK/Qt X programs will work if the im-switch setting is
> > done by scim-chewing.
>
> The first line in
> /etc/X11/xinit/xinput.d/scim-chewing, "XIM=SCIM", will do the magic.
> It is very common for im-switch's conffile to use "XIM" instead of
> "XMODIFIERS". It do work in X programs not using GTK or Qt.

Yes, you are absolutely right.

My comments quoted above are wrong. Sorry for the confusion.

Ming
2006.09.22

[Ming Hua (minghua)]

Okay, I have changed the bug title and description to reflect the database file corruption bug only (which should be already fixed in edgy), according to Matt's suggestion.

The problems about im-switch settings should go to a different bug. Please file a new bug on that issue. I won't file it myself because I am neither familiar with im-switch, nor using Ubuntu right now.

[Matt Zimmerman (mdz)]

Michael, now that the issues described here are better understood, please have a look at the problem and ensure that everything is as it should be in Edgy.

[Michael Vogt (mvo)]

Thanks for all this information and sorry for the late reply.

For edgy we now have scim-chewing 0.3.1 and libchewing 0.3.0.

If scim-bridge is installed, we use it for the GTK_IM_MODULE, otherwise we use xim.

For Qt we use scim as QT_IM_MODUL if scim-qtimm is installed and xim otherwise.

From my reading of this bugreport this sounds like we are fine in edgy (both database corruption wise and im-switch default settings).

I would like to close this report (if my analysis is correct) and add a new task to backport the current scim-chewing to dapper-backports.

Cheers,
 Michael

[wolfger (wolfger)]

This bug has been untouched for over 2 years. Based on my understanding of it (and the fact that Edgy is out of support now anyway), I'm marking "fix released".