GVIM does not display Chinese characters not in UTF-8 encoding

[transmixer]

Binary package hint: vim

I'm using Ubuntu 10.04 x64, GVIM2 menu is strange and no Chinese character is displayed.
I've installed Chinese language support in advance.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: vim-gnome 2:7.2.330-1ubuntu3
ProcVersionSignature: Ubuntu 2.6.32-24.39-generic 2.6.32.15+drm33.5
Uname: Linux 2.6.32-24-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Sat Aug 14 16:56:00 2010
ExecutablePath: /usr/bin/vim.gnome
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release amd64 (20100429)
ProcEnviron:
 LANGUAGE=zh_CN:zh
 PATH=(custom, user)
 LANG=zh_CN.utf8
 SHELL=/bin/bash
SourcePackage: vim

[An Yang (euroford)]

Hi there,

One year passed, still there!
No body care? or No body know how to fix it?

It's easy to fix, just add "set fileencodings=ucs-bom,utf-8,big5,gb18030" in /etc/vim/vimrc or in ~/.vimrc

[An Yang (euroford)]

gb18030 encoded file.

[An Yang (euroford)]

the content of gb18030.txt should look like this.

[Eric Miao (eric.y.miao)]

I don't see this problem happening on Precise. Am testing lucid.

[Eric Miao (eric.y.miao)]

This issue doesn't happen on Lucid. Attached is the snapshot of gvim running with chinese menu, with LANG="zh_CN.UTF-8". I'll first close this one unless there are more reports on this.

[An Yang (euroford)]

Hi Eric Miao,

You miss understand this bug, gvim could display any UTF8 encoded files, but it could NOT display GB18030 encoded files.
Please open a GB18030 encoded file with gvim, and paste what you see, thanks.

[Eric Miao (eric.y.miao)]

@An Yang, then apparently we need to change the title to something different. gvim _can_ display GB18030, the problem is really it has moved to use utf-8 internally, and guessing the encoding before read it in according to the list of fileencodings.

:set fileencodings

And you will see the list of default fileencodings being "ucs-bom,utf-8,default,latin1". Now I believe default is the charset of the current locale (most likely being 'UTF-8'), so vim will fall back to latin1 as the last resort, and trying to convert the gb18030 encoded characters internally to utf-8, treating them as latin1 encoded, which is obviously wrong. And once converted to utf-8, setting different encoding by 'set encoding=' won't work as the content of the buffer is already wrong.

One way to work around this,

:set fileencodings=gb18030,ucs-bom,utf-8,default,latin1

This will give vim a chance to convert from gb18030 firstly, and it works for me.

[An Yang (euroford)]

sure, there are many work around methods, but I do NOT think that the general user in China could find them.
So set these fileencodings in the package is the final solution.

[Eric Miao (eric.y.miao)]

@An Yang, yes, I'll bring forward the topic to see what's the best way out. In the meanwhile, I've changed the subject to make it more precise.