File with non-C/special characters in host computer disappear
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Wubi |
Medium
|
Tormod Volden | ||
| lupin (Ubuntu) |
Medium
|
Evan | ||
| Hardy |
Undecided
|
Unassigned |
Bug Description
TEST CASE: When I start Ubuntu (installed with wubi), it is almost prefect that the host drive (the drive "D" in my case with Wubi virtual hard disk installed) is mounted automatically. Subsequently, I found that all the files and directory with "Traditional Chinese" name is disappeared in the host drive. I have checked this out by reboot my computer, starting Windows XP and found that those files and directory are still there.
Locale support is needed in the initramfs when the ntfs host filesystem is mounted in order for proper i18n file support. As this is a patch to lupin, it only affects Wubi users.
A debdiff of the proposed change, written by Tormod Volden, can be found here:
http://
A debdiff of the entire 0.19 upload, including the fix for bug 226622, can be found here:
http://
Related branches
Agostino Russo (ago) wrote : | #1 |
Changed in wubi: | |
importance: | Undecided → Medium |
status: | New → Confirmed |
Tormod Volden (tormodvolden) wrote : | #2 |
This is not only for Chinese characters but all non-"C" characters. See also http://
Is there any workaround possible, like something to change in the initrd?
Tormod Volden (tormodvolden) wrote : | #3 |
There's a workaround sketched out at http://
Is this depending on or a duplicate of bug #118972?
The post http://
Agostino Russo (ago) wrote : | #4 |
Try to boot with the kernel argument: rootflags=
Tormod Volden (tormodvolden) wrote : | #5 |
Thanks, I will try it this weekend. I suppose rootflags=
Mads Peter Rommedahl (lhademmor) wrote : | #6 |
I can confirm that this is also true for Danish (æ, ø, å) and Icelandic (ð, á) characters (and is annoying as hell...)
Agostino Russo (ago) wrote : | #7 |
Does booting with rootflags=
Changed in wubi: | |
assignee: | nobody → ago |
Mads Peter Rommedahl (lhademmor) wrote : | #8 |
I tried adding rootflags=
Mads Peter Rommedahl (lhademmor) wrote : | #9 |
Disregard that last comment, it should have been rootflags=
BUT it still doesn't help. Is there any way I can check that I'm actually booting in da_DK...?
Tormod Volden (tormodvolden) wrote : | #10 |
Is it true that the Danish locale is "utf8" while the others are "UTF-8" ?
Tormod Volden (tormodvolden) wrote : | #11 |
(Reading the link I posted myself...) I guess the locale= value should correspond to one of the lines in the output from the command "locale -a"? So for instance en_US.utf8, or what Mads said.
Agostino Russo (ago) wrote : | #12 |
Can you pls try mounting the same partition with the appropriate locale, but off the Live CD?
Mads Peter Rommedahl (lhademmor) wrote : | #13 |
I'm sorry, but my patience ran out, so I ended up formatting the whole drive, and am now in the process of letting Ubuntu take over everything.
Agostino Russo (ago) wrote : | #14 |
This won't make the CD, but if I get more feedback on the issue I will add it to the website version and it will hopefully make the point release in July. I will do some more tests myslef. But it would be handy to have confirmation whether:
1) You can succefully mount other ntfs partitions (i.e. not the /host one) manually with locale mount option.
2) Whether the kernel option works at boot
Agostino Russo (ago) wrote : | #15 |
Also is the default locale correct in your installation?
Changed in wubi: | |
milestone: | none → 8.04.1 |
Tormod Volden (tormodvolden) wrote : | #16 |
I am afraid I can't be of much help either, since I don't have direct access to any computers with ntfs.
Agostino Russo (ago) wrote : | #17 |
Álvaro M. Recio (naproxeno) wrote : | #18 |
I can confirm that the rootflags boot option doesn't work, at least with locale=es_ES.utf8. I get the same warning that Mads Peter Rommedahl mentioned, something about the locale that couldn't be set and pointing me to http://
I don't know if it's worth mentioning this, but when I installed Wubi I changed some boot option (before the first boot into Ubuntu) from es_AR to es_ES (I think Wubi should select that locale for the Spanish language instalation). I don't know if it could have an effect on my tests.
The version of Wubi I used is Wubi-8.
Agostino Russo (ago) wrote : | #19 |
Can you try to mount manually from Live CD?
Agostino Russo (ago) wrote : | #20 |
Please note that most locales end with "UTF-8" capitalized. E.G.
locale=pt_BR.UTF-8
For a full list see the link above
Álvaro M. Recio (naproxeno) wrote : | #21 |
I won't be able to try the live CD on the original computer until next Monday but I tried it on another computer and it isn't even necessary to mount the partition manually: it appears in the "Places" menu and it's mounted correctly automatically.
When I added my previous comment I tried to mount the partition both with "utf8" and "UTF-8", and got the same result (the warning).
I will comment again after I test the live CD on the original computer.
Tormod Volden (tormodvolden) wrote : | #22 |
I had an NTFS partition on a USB drive, so I could do some tests, although not with Wubi:
The drive mounts automatically with utf support in the live CD. (locale is en_US.utf-8, and if I use "LC_ALL=C ls" I see "??" instead of the utf characters.)
I booted my normal Hardy kernel with break=mount and mounted the ntfs partition manually from the initramfs shell. Whatever locale option I used, I got the same warning as Mads and Álvaro and the utf8-named directories were invisible. I also tried to set the LANG before mounting (as done in /etc/init.
I checked that the initramfs has nls_utf8.ko in /lib/modules/
Tormod Volden (tormodvolden) wrote : | #23 |
Looking at the ntfs-3g source code: It is setlocale(LC_ALL, val) in src/ntfs-3g.c (where val is the locale= argument) which fails and yields the "WARNING: Couldn't set locale to '%s' thus some file names may not..."
Tormod Volden (tormodvolden) wrote : | #24 |
WRT to my initramfs shell testing, it's rather /usr/lib/locale that I should symlink...
I guess everything will work if the initramfs contains the /usr/lib/locale subdirectory corresponding to the locale= value. I'll have to verify that later though.
Tormod Volden (tormodvolden) wrote : | #25 |
Yes, just tested it and it worked: I copied /usr/lib/
Tormod Volden (tormodvolden) wrote : | #26 |
I made this initramfs hook which includes the current locale to the initramfs. Together with the rootflags=
Agostino Russo (ago) wrote : | #27 |
Tormod, fantastic work! Thanks a log.
I am not sure the required changes can be added into initramfs-tools at this stage, but it helps a lot.
The proper fix is of course to be able to remount ntfs-3g AND loopfiles.
One thing you might want to clarify. In kernel options do you need:
rootflags=
Tormod Volden (tormodvolden) wrote : | #28 |
The instructions inside my file were wrong. It should be rootflags=
Yes, I tried remounting:
sudo mount -o rw,locale=C /dev/sdb1 /mnt
sudo mount -o ro,remount,
but it seems like the locale option is not honoured when remounting. (The "ro" is.) The clue to this might be in the fuse process which is hanging around:
7013 ? Ss 0:00 /sbin/mount.ntfs /dev/sdb1 /mnt -o rw,locale=C
Again I had a look at the source code, but there it seems like remount is not supported at all. So I don't know how the "ro" flag got changed. Maybe ro is handled on another level, for instance by the kernel without the involvement of the fuse process.
As for adding this to initramfs-tools: It can be added to any package, since it is just a hook. IMHO, the best would be if ntfs-3g could have native utf support, like the old ntfs driver, without having to go through locales. I don't think the NTFS partition has any "locale", it just uses utf.
Tormod Volden (tormodvolden) wrote : | #29 |
Agostino, just saw your tests and comments in bug #186117 :)
Agostino Russo (ago) wrote : | #30 |
+ bug #201750 ... :)
Both are scheduled for 8.10
We might be able to add your script via lupin-support (see launchpad.
Tormod Volden (tormodvolden) wrote : | #31 |
Minor adjustments to the locale name mangling to avoid potential issues with non-utf8 locales.
Michael Greifeneder (mikegr) wrote : my test | #32 |
My experience: special characters on /host partition don't work with wubi. "rootflags=
I don't know much about initramfs. So I will wait, until something to test is available.
Tormod Volden (tormodvolden) wrote : | #33 |
Michael, could you please test the "ntfs_locale" workaround? Just download the file and save/copy it to /usr/share/
Michael Greifeneder (mikegr) wrote : | #34 |
Hi,
First of all it works. I had to set the execute permission on the script. "sudo chmod ugo+x /usr/share/
You won't see any "Added locale:" messages without the verbose option to update-initramfs: "sudo update-initramfs -u -v"
My locale is de_AT.UTF-8.
I added "rootflags=
szkrep (remhorcsog) wrote : | #35 |
hi,
If I could understand you, since now it's possible to see files and directories with special characters in a wubi-installed ubuntu.
Could anyone write a "how to" for a beginner?:) I have never used linux, so nothing is obvious for me, but a step by step instruction would be useful for lots of people!
(for example if I'm hungarian, i have another locale, etc...)
thanks
Tormod Volden (tormodvolden) wrote : | #36 |
MIchel, can you please attach your menu.lst so we can figure out why the "# kopt=" line doesn't work?
szkrep, if we can verify that this workaround works well, and it will take some time to get an official fix released, I will think about making a .deb package you can just double-click.
Tormod Volden (tormodvolden) wrote : | #37 |
Michael (sorry for the mistyping), funny enough I don't need the verbose option to update-initramfs. Are you on Ubuntu 8.04?
Tormod Volden (tormodvolden) wrote : | #38 |
I made a first shot on a .deb package. Just double-click to install and then reboot.
Upon installation it will guess your current locale, add it to the initramfs and also add the rootflags option to menu.lst if there's not one already. The guessed locale will be stored in /etc/initramfs-
Note that if you if uninstall this package, the /etc/initramfs-
Agostino Russo (ago) wrote : | #39 |
Tormod,
If Evan agrees we can add the package to lupin > lupin-support. Which is installed by default in loopinstallations.
Tormod Volden (tormodvolden) wrote : | #40 |
We should also think about, if we add rootflags automatically to menu.lst, that it has to be taken away if the user moves the install to a real ext3 partition.
The best would be if we hacked mountroot() to add locale= if ROOT_LOCALE is set (through conf.d) and FSTYPE=ntfs (or any fs supporting the locale option). Actually we don't need to modify mountroot, we can just add a premount script. Then we wouldn't need to trick with menu.lst at all.
Finally we just set up ROOT_LOCALE in lupin-support.
Agostino Russo (ago) wrote : | #41 |
Sounds very good to me. You need to be in the ubuntu-installer team to commit to lupin though.
Agostino Russo (ago) wrote : | #42 |
I have created a separate branch called lupin/hardy.
Agostino Russo (ago) wrote : | #43 |
In fact feel free to upload your branch I will merge in hardy.proposed. Then if/when we get approval I will merge into hardy branch.
Tormod Volden (tormodvolden) wrote : | #44 |
Here is a new workaround package for testing. For new testers: Double-click and reboot. If you already did some testing (like the previous package) please also remove any "rootflags=" option from the "# kopt" line in your /boot/grub/menu.lst and run "sudo update-grub".
Tormod Volden (tormodvolden) wrote : | #45 |
Agostino, I made a branch on https:/
If anyone installed the above root-locale-patch packages, you can uninstall them and rather try this updated version of the lupin-support package.
Tormod Volden (tormodvolden) wrote : | #46 |
3377 (wanghaid) wrote : | #47 |
Tormod, I am using Ubuntu 8.04 installed by Wubi in a chinese Windos XP. locale=cn_ZH.utf8
All filename include chinese characters in Diver D: (Ubuntu installed) can not be display from Ubuntu. and after installed your last patch, lupin-support_
thanks a lot.
Changed in wubi: | |
assignee: | ago → tormod |
status: | Confirmed → Fix Committed |
Changed in wubi: | |
assignee: | tormod → tormodvolden |
szkrep (remhorcsog) wrote : | #48 |
It doesnt work with hungarian:(
I downloaded, double clicked, rebooted, but still nothing happens.
"ö,ü,ó,ő,á,ű, etc" can't be seen.
szkrep (remhorcsog) wrote : | #49 |
Does it help anything?...
szkrep@
LANG=hu_HU.UTF-8
LC_CTYPE=
LC_NUMERIC=
LC_TIME=
LC_COLLATE=
LC_MONETARY=
LC_MESSAGES=
LC_PAPER=
LC_NAME=
LC_ADDRESS=
LC_TELEPHONE=
LC_MEASUREMENT=
LC_IDENTIFICATI
LC_ALL=
Tormod Volden (tormodvolden) wrote : | #50 |
szkrep, can you please give the output of these commands:
ls -d /usr/lib/
cat /etc/initramfs-
eduardotv (eduardotv) wrote : | #51 |
same problem,
im my case :
eduardo@
LANG=pt_BR.UTF-8
LC_CTYPE=
LC_NUMERIC=
LC_TIME=
LC_COLLATE=
.......
LC_ALL=
the output is?
ls -d /usr/lib/
/usr/lib/
cat /etc/initramfs-
ROOT_LOCALE=
export ROOT_LOCALE
export PSTYPE
eduardotv (eduardotv) wrote : | #52 |
OPS.. ntfs_locale doesnt exist.. using root_locale
cat /etc/initramfs-
ROOT_LOCALE=
export ROOT_LOCALE
export PSTYPE
Álvaro M. Recio (naproxeno) wrote : | #53 |
I've just tested lupin-support_
Good job!
Tormod Volden (tormodvolden) wrote : | #54 |
eduardotv, sorry for the confusion, I had used "ntfs_locale" in an earlier workaround but it should now be "root_locale" everywhere. You have FSTYPE and not "PSTYPE" in the output, right? And you used the 0.17 package? Can you please run:
gzip -dc /initrd.img|cpio -t > cpio.listing
and attach cpio.listing? (If you don't have /initrd.img, use /boot/initrd.
b@z (antoinebison) wrote : | #55 |
Hello, I tested the lupin-support_
ls -d /usr/lib/
/usr/lib/
cat /etc/initramfs-
ROOT_LOCALE=
export ROOT_LOCALE
export FSTYPE
and I joined the cpio listing. The "sudo update-initramfs -k all -u" doesn't work.
Thanks for your great work !
Michael Greifeneder (mikegr) wrote : | #56 |
Hi, lupin-support_
Anyway, my girlfriend is satiesfied that she can access all her files. Thanks.
Tormod Volden (tormodvolden) wrote : | #57 |
Michael, you'll have to restore menu.lst manually. Take away the "rootflags=.." stuff on the "# kopt" line, and run "sudo update-grub".
b@z and eduardotv, does it help if you add "rootflags=
Tormod Volden (tormodvolden) wrote : | #58 |
Agostino, I am thinking about some changes with regards to the configuration/
Current flow:
1. Upon package installation, the locale is "guessed" from $LANG, and stored in /etc/initramfs/
2. update-initramfs will read from the above file, and copy in the locale files together with the configuration file itself.
3. Every boot, the mount options are generated and applied using the conf file copy in the initramfs.
This is nice and flexible, I thought. And good for a tweaking workarounds. However, at this point I don't see the need for this flexibility. So far, we have no reason to think the guessing in (1) will be wrong. OTOH, the user might change the default locale on his computer ($LANG) but initramfs will always pick up the old, initial setting.
I will therefore move the guessing to the initramfs hook, and to be done fresh each time. The conf file will just witness to what locale was chosen at last initramfs, but will not be used for anything.
We can then get rid of the postinst script. No configuration left around to become stale. A postrm can safely remove the conf file which is not used anyway (only internally in update-initramfs).
If needed we can add more flexibility later. It's easier that way, rather than drag along complexity from the start that we have to deal with when we change things in the future.
New flow:
101. update-initrams reads LANG from /etc/default/locale (to be sure to use the system default and not the one of the current session where it is run). Guesses locale from LANG and if it finds the matching locale files, copy them in and write out the conf file that is copied into the initramfs.
102. Every boot, the mount options are generated using the conf files in the initramfs.
Thinking about it, we even keep all flexibility, since the user can always create an root-locale-
Agostino Russo (ago) wrote : | #59 |
I agree, I like it better. FYI, we call apt-install lupin-support within autopartition-loop (partman-
Tormod Volden (tormodvolden) wrote : | #60 |
Agostino,
- about that root-locale-
- about postinst: still needed to run update-initramfs (as is postrm)
- re about autopartition-loop: is /etc/default/locale written at this point?
Fixed a local(e) typo that probably caused problems for some people. I wonder if it could work at all.
Here's a new version. Please test!
Note the version number is lower than the previous 0.17. You will have to "downgrade". This is to make sure the final released version (probably 16.1) will supersede these test versions.
Agostino Russo (ago) wrote : | #61 |
Tormod, I am not at my Linux machine, but Evan confirmed that lupin-support should be installed late enough and hence postinst should be fine.
Tormod Volden (tormodvolden) wrote : | #62 |
Thanks Agostino, since all configuration now is done in update-initramfs, the postinst doesn't matter much. update-initramfs will be called late in the installation anyway, also by other packages, and most likely well after all locales are installed and configured.
BTW, the local(e) typo in my 0.16 and 0.17 was probably responsible for all the cases where it didn't work. Those packages only worked for people who still had rootflags= on the kernel line.
b@z (antoinebison) wrote : | #63 |
I tried this new version 0.16 of lupin-support in my Wubi installation and it seems to work perfectly this time. The "older" version 0.17 didn't work for me before. Thanks a lot.
szkrep (remhorcsog) wrote : | #64 |
I've also tried it, now everything works fine! hungarian characters can be seen. Thank you!
Agostino Russo (ago) wrote : | #65 |
Tormod,
Very well then, I will merge it today or in the coming days then into a separate lupin.proposed branch. I have some small changes to add as well (more colorful panic message re #226622) then we will submit that to hardy.proposed.
Agostino Russo (ago) wrote : | #66 |
In fact we might keep the changes in separate branches, do you think it is in a good stage for approval? I will do some testing tonight.
Agostino Russo (ago) wrote : | #67 |
Tormod, the code looks good to me :)
Haven't tested it though. I will also as #ubuntu-testing to give it a go
Tormod Volden (tormodvolden) wrote : | #68 |
Agostino, I have a few improvements left: adjust verbosity, get rid of that dangling conf file. It's easier to polish it now than after it's been released.
Agostino Russo (ago) wrote : | #69 |
Absolutely, do all the changes you deem necessary. Then once you think you are ready you may want to notify myself and evand.
PS Not sure if allowing for an environment variable might make things a bit more flexible:
[ -z "$ROOT_LOCALE" ] && ROOT_LOCALE=`echo $LANG .....
Tormod Volden (tormodvolden) wrote : | #70 |
Everybody, please test this new package. Even if you have it working already. And important, before you install the new package:
1. Remove all previous test packages: sudo apt-get remove root-locale-patch lupin-support
2. Remove any "rootflags" from /boot/grub/menu.lst by editing the "# kopt" line and running: sudo update-grub
3. Remove any old configuration file: sudo rm -f /etc/initramfs-
Then install this package and reboot.
If we get good feedback from you on this "0.16+locale112" version, we will push it to an official release soon.
Tormod Volden (tormodvolden) wrote : | #71 |
Test this one instead :)
b@z (antoinebison) wrote : | #72 |
For me, the last version (lupin0.16 & locale113) works fine.
Sun Bok Kim (cylon21) wrote : | #73 |
Tormod, I tried it (lupin0.
Thanks for your great work.
Agostino Russo (ago) wrote : | #74 |
Tested and works for me
Álvaro M. Recio (naproxeno) wrote : | #75 |
Tested lupin-support_
description: | updated |
description: | updated |
Launchpad Janitor (janitor) wrote : | #77 |
This bug was fixed in the package lupin - 0.17
---------------
lupin (0.17) intrepid; urgency=low
[ Agostino Russo]
* Provide a clearer error when required installation files cannot be
found (LP: #226622)
[ Tormod Volden ]
* Add locale support to ntfs host filesystems, to keep files and
directories with non-C characters (utf) from disappearing.
(LP: #136682)
-- Evan Dandrea <email address hidden> Fri, 09 May 2008 13:53:14 -0400
Changed in lupin: | |
status: | Fix Committed → Fix Released |
Tormod Volden (tormodvolden) wrote : | #78 |
AFAICS, this has not made it into the hardy-proposed repo nor even the wubi/hardy.proposed branch yet? 8.04.1 deadline approaching :)
Martin Pitt (pitti) wrote : | #80 |
Accepted into -proposed, please test and give feedback here
Changed in lupin: | |
status: | New → Fix Committed |
Changed in wubi: | |
status: | Fix Committed → Fix Released |
Paulos (valach-pavel) wrote : | #81 |
No problem with it.
Martin Pitt (pitti) wrote : | #82 |
Paulos, thanks for your response. Does that mean you got the original problem, tested the updated package and it works for you now? How did you test?
Emmet Hikory (persia) wrote : | #83 |
Note that while this patch seems a good improvement for many users, it may not be a complete solution for other users, especially those using NTFS in a "MUI" configuration. Bug #49217 covers some of the charset translation issues that can happen for FAT, and to a significantly lesser degree to NTFS, as a result of assumptions made about the relationship of a filesystem and the internal system locale on legacy systems. While no regressions are anticipated, it may be that some users in certain locales or with certain legacy configurations may continue to experience a subset of this issue (especially if the system under consideration is not a native NTFS install). This comment should not affect any decision on promotion of the package as a Stable Release Update.
Agostino Russo (ago) wrote : | #84 |
I have tested for regression and couldn't see any problem with it.
I tested this on a French copy of Windows XP Professional and it had properly accented characters.
Steve Langasek (vorlon) wrote : | #86 |
copied to hardy-updates.
Changed in lupin: | |
status: | Fix Committed → Fix Released |
Confirmed, but it's not easy to remount /host at the moment. We'll see if that can fixed for 7.10