wubi install unusable - Buffer I/O error on device loop0

Bug #204133 reported by Brian Murray
34
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Wubi
Fix Released
High
Agostino Russo
8.04
In Progress
High
Agostino Russo
linux (Ubuntu)
Invalid
Undecided
Unassigned
Hardy
Invalid
Undecided
Unassigned
Intrepid
Invalid
Undecided
Unassigned
lupin (Ubuntu)
Fix Released
Undecided
Unassigned
Hardy
Won't Fix
Undecided
Unassigned
Intrepid
Fix Released
Undecided
Unassigned
ntfs-3g (Ubuntu)
Fix Released
Undecided
Unassigned
Hardy
Fix Released
Undecided
Unassigned
Intrepid
Fix Released
Undecided
Unassigned

Bug Description

LATEST ISO TESTED:
Ubuntu amd64 20080411 w/
wubi rev 482

ISO TESTED:
Ubuntu amd64 20080319

After what seemed to be a successful install via wubi I am unable to login to the system.
I booted into rescue mode and found in dmesg the following:

Buffer I/O error on device loop0, logical block 872512
lost page write due to I/O error, on loop0
...
EXT3-fs error (device loop0): ext3_get_inode_loc: unable to read inode block - inode=114030, block=229669
Aborting journal on device loop0.
__journal_remove_journal_head: freeing b_committed_data
(repeat 6x)
Remounting filesystem read-only

Related branches

Revision history for this message
Evan (ev) wrote :

Brian,

Were you by any chance using an amd64 image on VMWare?

Revision history for this message
Brian Murray (brian-murray) wrote :

No, this was / is on real hardware.

Revision history for this message
Brian Murray (brian-murray) wrote :

This is from a crash when booting into single user mode.

Revision history for this message
Brian Murray (brian-murray) wrote :

2nd call trace.

Revision history for this message
Agostino Russo (ago) wrote :
Changed in wubi:
assignee: nobody → ago
importance: Undecided → Low
status: New → Confirmed
assignee: ago → nobody
importance: Low → Undecided
status: Confirmed → New
Revision history for this message
David Burgess (apt-get) wrote :

Same issue here. I installed wubi x86_64 many times on a XP Home NTFS system and it always booted with / mounted read-only. Also tried the x86 version with same result. Sometimes I got a shell in single user mode, but not usually.

db

Revision history for this message
Agostino Russo (ago) wrote :

Does mounting the same windows partition in r/w mode from within a standard Ubuntu installation work?

Can you also run ntfs-3g.probe?

Thanks in advance

Agostino Russo (ago)
Changed in wubi:
assignee: nobody → ago
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Daniel Ryden (danryd) wrote :

Got the same problem. Got everything working, then I did the update as requested by the Update manager. DL and installed all updates and after that it wouldn't boot properly.

Revision history for this message
Agostino Russo (ago) wrote :

Receiving more reports on this one. Still need more info to track it down.

Changed in wubi:
importance: Medium → High
Revision history for this message
David Burgess (apt-get) wrote :

I no longer have access to the problem machine (until August). I'll try wubi on a couple other windows machines I have permanent access to.

db

Revision history for this message
Agostino Russo (ago) wrote :

We did some tests with Brian Murray yesterday (thanks a lot for the help), to summarize:

* Up to run-init everything seems good /host (ntfs device) is rw and /root (ext3 loopfile within ntfs filesystem) is not, which is correct since /root will have to be remounted rw by checkroot.sh later on. Remounting /root rw manually works.

* We booted with init=/bin/sh and Brian did run manually all the scripts in /etc/rcS.d. The scripts were successful and / became rw as expected.

* Yet when booting in single user mode the above mentioned errors did appear and / was ro.

Brian, feel free to correct me if the above is inaccurate.

Revision history for this message
Agostino Russo (ago) wrote :

I have experienced somewhat similar. In my case errors are not always reproducible and generally manifest in late stages of rc2.d (gdm). I managed to get some logs off one of the crashes that was less debilitating.

PS: Not sure if it is relevant but the following sysctl settings are in place:

sysctl $quiet -w vm.dirty_background_ratio=0
sysctl $quiet -w vm.dirty_ratio=0
sysctl $quiet -w vm.dirty_expire_centisecs=1
sysctl $quiet -w vm.dirty_writeback_centisecs=1

http://bazaar.launchpad.net/~ubuntu-installer/lupin/hardy/annotate/evan.dandrea%40canonical.com-20080407154533-rztovcdf8foxctkt?file_id=lupinsysctl-20070830094942-u4sn2uojcan8bqi7-1

Revision history for this message
Agostino Russo (ago) wrote :
Revision history for this message
Agostino Russo (ago) wrote :
Revision history for this message
Agostino Russo (ago) wrote :

The issue seems related to the sysctl settings modified by /etc/init.d/lupin-sysctl. Uninstalling lupin-support should make the system work again without hick ups. The sysctl settings were suggested by Szaka in order to minimize the effects of hard reboots with nested filesystems (where the nested fs journal is at risk of becoming corrupted because of data loss in the host fs) and used to work fine with older kernels. We should reassess whether it is appropriate to keep such settings and investigate whether some of them are incompatible with the current kernel (as it seems).

Changed in wubi:
status: Confirmed → In Progress
Revision history for this message
Colin Ian King (colin-king) wrote :

The kernel division by error is perplexing. I need to add some diagnostics into the kernel and see why the denominator is zero - looking at the source it is not straightforward - there are some time decay process values that produce the denominator and they should in theory be > 0. Will try to reproduce and figure out a fix or appropriate work around.

Revision history for this message
Colin Ian King (colin-king) wrote :

Hi,

I've dug into this and come across several unexpected issues with the tweaks to the vm settings which could be causing the corruption.

1. Setting the dirty_expire_centisecs and dirty_writeback_centisecs to 1 may not actually be setting the values you expect due to conversions between USER_HZ and HZ in the kernel. For example

setting: sysctl -w vm.dirty_expire_centisecs=1

and reading it back: sysctl vm.dirty_expire_centisecs returns 0 and not 1. This is due to conversion between USER_HZ and HZ to store the value in terms of the jiffy clock and then back from HZ to USER_HZ to read the value back out. I therefore suggest anything below 2 may be problematic in not being set correctly. See the next item.

2. Setting dirty_writeback_centisecs to zero actually turns off the periodic write back (pdflush is stopped) - definitely not what we want to do.

3. There is a currently outstanding Linux kernel bug that is rare and difficult to trigger even intentionally on most kernel versions. However, it is easier to encounter when reducing dirty_ratio setting below its default. An introduction to the issue starts at http://lkml.org/lkml/2006/12/28/171 and comments about it not being specific to the current kernel release are at http://lkml.org/lkml/2006/12/28/131 So don't set dirty_ratio too low. I suggest keeping it at the default value and see if this helps

4. It may be worth modifying the filebase looped back filesystem mount options to force data and directory sync'ing and force , for example,

    mount -o loop,sync,dirsync,commit=1

    (see man 8 mount for how these work)

5. I suggest trying:

sysctl $quiet -w vm.dirty_background_ratio=0
sysctl $quiet -w vm.dirty_ratio=40
sysctl $quiet -w vm.dirty_expire_centisecs=2 (or 4)
sysctl $quiet -w vm.dirty_writeback_centisecs=2 (or 4)

(for a system where USER_HZ=100, HZ=250). After setting the centisecs times, it is worth checking the dirty_writeback_centisecs is not zero otherwise the pdflush daemon may be turned off!)

6. When the PC shuts down or reboots, is it possible to force a sync before umounting the loopback device?

References:

http://www.westnet.com/~gsmith/content/linux-pdflush.htm
http://lkml.org/lkml/2006/12/28/171
http://lkml.org/lkml/2006/12/28/131

Let me know if this helps.

Colin

Revision history for this message
Colin Ian King (colin-king) wrote :

Forgot to say that I am still trying to figure out why the kernel division by zero error occurs in code where the denominator is never actually zero. As yet, I cannot reproduce this error. Has anyone a Win XP CD + valid license so I can do a Wubi install and reproduce this error?

Revision history for this message
Colin Ian King (colin-king) wrote :

OK, I seem to have a handle on the real problem now.

Basically the final umounts of the filesystems is probably not working correctly on shutdown/reboot causing the ext3 loopback filesystem to get corrupted.

I think what is happening is that the NTFS filesystem is being unmounted before the looped back ext3 filesystem on that NTFS filesystem and the of course the ext3 filesystem cannot be umounted correctly and then the corruption found on the next reboot.

The kernel panics we see are because of the severe corruption in the loop mounted ext3 filesystem. I've managed to reproduce this on my system in a way that happens every time, no matter what kind of vm dirty flags are being tweaked. The vm dirty flags cannot work around such an issue - I believe somebody with better knowledge of Wubi and the Ubuntu shutdown process than I needs to investigate this.

In summary: Make sure the ext loopback file systems are umounted before the NTFS filesystems.

Colin

Revision history for this message
Agostino Russo (ago) wrote :

Colin,

Thanks a lot, the above is invaluable. Changes to mount options in initramfs and sysctl settings should be doable. I am a bit more concerned about the last comment...

The loopback filesystem (file containing ext3) contains /, so that is never really unmounted, only remounted r/o in /etc/init.d/umountroot. As for /host (ntfs filesystem containing the root loop file) that is not even set r/o. A patch to remount /host r/o is in #186114, but was not applied since Szaka mentioned that remounting an ntfs r/o can lead to data loss (#186117). Do you think that we should proceed and have /host r/o? Would that be sufficient?

Hints welcome!

Revision history for this message
Agostino Russo (ago) wrote :

If anyone without Windows wants to emulate, the steps are the following

1) Create an ntfs partition
2) Have a file containing a hardy root filesystem (ext3)
3) Keep the hardy kernel and initrd in another filesystem
4) Use grub to boot off the above kernel/initrd with the parameters: root=/dev/sdaX loop=/path/to/loop/file
5) The fstab inside of the loop filesystem should have an entry for loopmounting / from /host/path/to/loop/file (/dev/sdaX will be mounted as /host)

where sdaX is the ntfs partition.

The vm dirty hacks are in /etc/init.d/lupin-support
Also of interest is what happens immediately after /etc/init.d/umountroot

I.E. if we are in a state such that /sbin/reboot is safe. To test that you might want to skip /etc/init.d/reboot

Unmounting /host manually is always unsafe (hosted fs data loss may result, not sure if hosted fs journal is at risk), but unmounting it via rc6.d should be safe

Note that at the moment, as mentioned above, /host is NOT remounted r/o in umountroot. You may want to apply the patch in #186114 and test that. Also I have to double check the ordering in which filesystems are remounted ro in umountroot. The correct order is inverse order of /proc/mounts.

Revision history for this message
Agostino Russo (ago) wrote :

Colin, about point 4 above, are you sure I should add the sync options to the loopfile mount?

I would have thought that I should have had sync & co for the host filesystem (ntfs) but not for the nested one (ext3 in loopfile). The idea being that the data should hit the metal when the nested fs syncs, so it is the host fs that has to avoid caching. Caching in the nested fs is not necessarily a bad thing.

Revision history for this message
Agostino Russo (ago) wrote :
Revision history for this message
Agostino Russo (ago) wrote :
Revision history for this message
Agostino Russo (ago) wrote :

Have attached above a couple of files to test.

The first one is a new initramfs with sync mount options (replace /usr/share/initramfs-tools/scripts/local and rebuild the initrd). The sync options are for the host filesystem as opposed to the nested one (see above comment).

The other file is /etc/init.d/umountroot. I tested it, but it does not seem to manage to remount /host (ntfs-3g) ro. I get the error "Remounting is not supported at present" (see bug #186117). One way to test is to disable /etc/init.d/reboot and then run sudo reboot.

You may want to change sysctl settings manually to emulate a wubi environment:

sysctl -w vm.dirty_background_ratio=0
sysctl -w vm.dirty_ratio=40
sysctl -w vm.dirty_expire_centisecs=2
sysctl -w vm.dirty_writeback_centisecs=2

Revision history for this message
Agostino Russo (ago) wrote :

If /host cannot be remounted (as it seems) a solution might be to modify /etc/init.d/reboot so that if a root or a host filesystem in rw mode is detected, it will create a ram disk with /sbin/reboot and any other file that might be required at that stage, unmount / and /host, and reboot. Colin, was that what you hinted at on IRC? Would something like that be reasonable?

Revision history for this message
Colin Ian King (colin-king) wrote :

I think was thinking this may be a via solution. Not sure how one does this in practice though as I am not really familiar with this part of the shutdown process and if one can umount /. Can one use pivot_root to twiddle things around to get access to /host? Needs some exploration I believe.

Revision history for this message
Colin Ian King (colin-king) wrote :

Just to add, if one needs to check how effective the sysctl's are, I suggest the following:

watch -n 1 grep -A 1 dirty /proc/vmstat

also if one wants to see which blocks are being flushed and by which process one can do:

sudo echo 1 > /proc/sys/vm/block_dump

and looking at the output from dmesg will show that the data is being flushed out.

There still seems to be a bit of delay in fully flushing blocks out with the suggested sysctl's. I've fiddled with various settings but I could not seem to improve things.

Revision history for this message
Agostino Russo (ago) wrote :

Colin,

A couple of things I wanted to mention.

1) In the CD lupin-sysctl is not installed, so in the first reboot the sysctl hacks are simply not used. The sysvinit scripts are identical.

2) In my case I had to reboot several times in order to get a situation where I could access the logs, and in a few cases I had more severe locks and had to use alt+sysrq+r+s+u+b (possibly also hard reboot) and all that was in vm which has been mistreated in the past. All this to say that fs corruption might have already been there to begin with and may or may not be a casue of the locks.

So to summarize it seems that sysctl setting do have an important role in addressing the original poster error. It is uncertain whether such sysctl settings do actually help in preventing fs corruption (in hard reboot scenarios and in normal reboot scenarios). It is uncertain whether fs corruption is a possible outcome of a normal reboot, but from your tests it seems not (was that with or without sysctl hacks?).

description: updated
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

I tested installing from an Ubuntu amd64 daily, 20080411, and was able to boot an install without any hacks.

Revision history for this message
Colin Ian King (colin-king) wrote :

Agostino,

There is one more thing to consider and that is the sysctl's do not help with any pending blocks in the loopback device. There can be a substantial number of pending writes in the loopback block device which have yet to be pushed out to the filesystem. Thus there there are several points where unwritten data is vulnerable:

1. Blocks in the loopback device that have yet to be written to the filestore (pending block I/Os).
2. Blocks that have been written, but not yet flushed by pdflush to the underlying disk.
3. Blocks pending in the disk cache (in the drive itself) that have not been physcially written back.

Point 1 cannot be solved by sysctl style tweaks, and must be addressed in the loopback device. However, there is no provision for this in loop.c. It appears that the loop back threads are running at maximum priority, but there is some time lag between blocks being written to the loopback device and the blocks being pushed out by these worker threads.

Point 2 is address a little by sysctl tweaks, but there is still a few centisceconds worth of latency between blocks being written and the blocks actually being flushed to the device.

So, in the case of catastrophic power failure, the loop backed ext3 filesystem is evidently more vulnerable to file system corruption than the normal non-looped back filesystem.

Colin

Revision history for this message
Agostino Russo (ago) wrote :

Colin,

Thanks again for your very helpful insights! I was aware about the nested filesystem vulnerabilities in power failure scenarios, not from code analysis, but from "empirical evidence". And that is in fact stated in http://wubi-installer.org/faq.php (requirements > any gotcha). I am mostly interested in a reasonable set of sysctl/mount args to minimize said risks (given that they cannot be eliminated completely). And also to ensure that there is no data loss of any type in a normal reboot (via rc6.d). In that respect, do you think there is anything we should change to improve the situation?

The new initramfs/umountroot will not be used since at this stage it is my understanding that they do not change much (extra mount options are ignored by ntfs-3g and /host cannot be remounted r/o). Evan already added your sysctl suggestions to lupin and they are in the new ISOs. In fact they seem to be enough to "close" the original bug! Anything else we should consider?

On a side note, you might also be interested in this discussion: http://lkml.org/lkml/2008/1/9/50

Ago

Revision history for this message
Colin Ian King (colin-king) wrote :

Hi Agostino,

Thanks for the link to the discussion, I missed that one when I looked at the loop device this morning.

I think these a rather "radical" changes so late in Hardy, for example, one of the listed "disadvantages" impact loop encryption which some users will definitely miss. However, I think it's well worth considering this kind of approach for Hardy + 1, especially as these changes may be incorporated officially upstream with the necessary modifications to handle loop encryption.

Colin

Revision history for this message
Agostino Russo (ago) wrote :

Changed to "fix released" for Wubi (8.04) since Brian's problem was addressed by Colin's suggestion and it seems there isn't much else to be done in this release . Work on loops and ntfs-3g remounts will be resumed in intrepid (possibly via separate bugs).

Changed in wubi:
status: In Progress → Fix Released
Revision history for this message
Colin Ian King (colin-king) wrote :

I've looked at the sysctl's again and think there is a cleaner solution than using:

sysctl -w vm.dirty_background_ratio=0
sysctl -w vm.dirty_ratio=40
sysctl -w vm.dirty_expire_centisecs=2
sysctl -w vm.dirty_writeback_centisecs=2

1. mount the Linux filesystems on the loopback using mount options -o loop,sync
2. modify ntfs3g with the attached debdiff which adds a -o syncio mount option to open the NTFS device using O_SYNC, e.g.

ntfs-3g /path/to/ntfs-test.img /path/to/mountpoint -o syncio

  this makes all ntfs dirty writes to get written back synchronously.

3. don't use the sysctl's

I've tested an ext3 file system mounted via the loopback with -o sync (point 1 above) and used my patched ntfs3g with the -o syncio mount option and using vmstat 1 shows that I/O is flushed immediately, reducing the possibility of file system corruption on power outages.

Revision history for this message
Agostino Russo (ago) wrote :

Colin, that is fantastic, I think we should try to have it in hardy.proposed as this is a clear improvement over the sysctl hack and should better address data loss situations in hard reboot scenarios.

One question: I can certainly appreciate the value of quick syncs for the host filesystem (2), but why do we exactly need -o loop,sync in the nested filesystem? Is there any benefit for that? My understanding is that if the host flushes quickly, the nested journal should be safe, whether the nested fs syncs quickly or not.

Revision history for this message
Agostino Russo (ago) wrote :

Szaka, would it be possible to merge Colin's patch above within ntfs-3g?

Revision history for this message
Agostino Russo (ago) wrote :

<cking> xivulon: ext3 journal writes will be flushed straight through to the device if ntfs-3g has the -syncio mount option. However, if you want extra resiliance to make sure writes get flushed immediately then I recommend -o sync on the loop too

<cking> It's a kind of performance vs reliability tradeoff

<xivulon> any idea of the performance hit for that?

<cking> My tests showed -o sync on the loop was 1/3 to 1/2 slower. But I did just raw block write tests using dd. Normal usage may differ.

Changed in wubi:
status: Fix Released → In Progress
Revision history for this message
Agostino Russo (ago) wrote :

Tested the Colin's debdiff and it works well for me, for instance:

ago@ago-laptop/%sudo mount -t ntfs /dev/sda2 /mnt
ago@ago-laptop/%rm /mnt/ubuntu-8.04-desktop-i386.iso && time cp -f /home/vm/cd/ubuntu-8.04-desktop-i386.iso /mnt
cp -f /home/vm/cd/ubuntu-8.04-desktop-i386.iso /mnt 0.05s user 2.16s system 5% cpu 41.809 total
ago@ago-laptop/%sudo umount /mnt
ago@ago-laptop/%sudo mount -t ntfs /dev/sda2 /mnt -o syncio
Mounting with synchronous I/O
ago@ago-laptop/%rm /mnt/ubuntu-8.04-desktop-i386.iso && time cp -f /home/vm/cd/ubuntu-8.04-desktop-i386.iso /mnt
cp -f /home/vm/cd/ubuntu-8.04-desktop-i386.iso /mnt 0.07s user 1.99s system 5% cpu 36.736 total

Revision history for this message
Agostino Russo (ago) wrote :

New debdiff which fixes the changelog as suggested by Martin Pitt

Revision history for this message
Martin Pitt (pitti) wrote :

Thanks, Agostino and Colin. Some questions:

 * Was this ntfs-3g patch sent/ack'ed upstream? Is it fixed in Intrepid already?

 * What is the rationale for adding the change to ntfs-3g and dropping the equivalent sysctl call in lupin? Is the hack in lupin insufficient to fix the problem? I need to understand why we need to make this "shift" in the hardy stable release (it's certainly good to clean up hacks in Intrepid, of course)

Revision history for this message
Agostino Russo (ago) wrote :

Martin, I sent an email to Szaka (ntfs-3g author) and took the liberty to subscribe him to this bug report. Szaka ping ;)

My understanding is that Szaka is aware of the patch and that it will be incorporated upstream possibly with some additional modification. It is unclear though whether that will be done in time for Intrepid. Colin will probably prepare a patch for intrepid to be used if the upstream version is not ready in time.

The lupin hack were less then perfect, for at least two reasons:

1) they did not really achieve full sync, hence exposing users to potential data losses as a result of hard reboots (comment #31),
2) they were not always "well tolerated" as the original bug report shows (comment #17), the values used at present are less strict because of that and while I haven't seen any complaint I cannot exclude that the issue is still lurking.

I am sure that Colin can clarify it further.

Revision history for this message
Martin Pitt (pitti) wrote :

Thanks for the clarifications so far. What about the "linux" portion of that bug? I take it there is still something to be fixed in the kernel? Do we need a fix in hardy as well, or is the workaround in ntfs-3g sufficient?

I'll wait for upstream's ack, then I'll sponsor that.

Changed in ntfs-3g:
status: New → In Progress
Revision history for this message
Agostino Russo (ago) wrote :

Martin, I am not sure what you are referring to as the "linux" bug, there a few related kernel issues I am aware of:

1) The "linux" bug mentioned in comment #17 is relevant for low levels of dirty_ratio. The ntfs-3g patch will avoid lowering the dirty_ratio and will indirectly help (which is a subcase of #2 above).

2) There was some discussion on the performance / reliability of the loop device in write intensive set-ups. See for instance http://lkml.org/lkml/2008/1/9/50. Colin seems to like the dm-loop approach. I am not sure if there are data loss scenarios that could be improved upon by using dm-loop or whether it is mostly a performance issue. My guess is that dm-loop might help with point #1 in comment #31

3) It would be nice to be able to use the standard mount option name "-o sync" as opposed to have to create a new one for each filesystem "-o syncio". Not least because it will create a bit of special casing in other code (lupin/initramfs-tools). From a past discussion with Colin it emerged that the naming was to work around fuse -o sync handling.

4) There is also a remount issue with loopdevices (see #201750). On top of that, there are separate remount issues for ntfs-3g (see #186117), even worse, it seems that remounting ntfs-3g ro/rw can produce data loss (https://bugs.edge.launchpad.net/wubi/+bug/186117/comments/10). Not sure if newer versions of fuse/ntfs-3g address this.

5) Related to the above there are the shutdown issues mentioned by Colin in comment #19. I am not sure what is that state of that given A) the ntfs-3g patch and possibly B) dm-loop usage.

Revision history for this message
Martin Pitt (pitti) wrote : Re: [Bug 204133] Re: wubi install unusable - Buffer I/O error on device loop0

Agostino Russo [2008-07-08 9:56 -0000]:
> Martin, I am not sure what you are referring to as the "linux" bug,

This bug has a linux task for intrepid and hardy. I wonder if these
are relevant at all, and for hardy in particular. They should be
triaged to Confirmed or Invalid/Wontfix.

Revision history for this message
Colin Ian King (colin-king) wrote :

Hi, Just to confirm Ago's statements:

> 1) The "linux" bug mentioned in comment #17 is relevant for low levels of dirty_ratio. The ntfs-3g patch will avoid lowering the dirty_ratio and will indirectly help (which is a subcase of #2 above).

Correct. The dirty ratios are at best a dirty hack, and when pushed to their limits, do show up some problems - which I don't think are appropriate to workaround in the kernel. The correct approach is to avoid the sysctl's on the dirty ratios and hit the crux of the problem and that is to make sure that ntfs-3g is flushing data using the O_SYNC open() flag rather than forcing it using some very aggressive vm tweaks.

> 2) There was some discussion on the performance / reliability of the loop device in write intensive set-ups. See for instance http://lkml.org/lkml/2008/1/9/50. Colin seems to like the dm-loop approach. I am not sure if there are data loss scenarios that could be improved upon by using dm-loop or whether it is mostly a performance issue. My guess is that dm-loop might help with point #1 in comment #31

I think that patching the loopback is more of a technical risk than using dm-loop. I think it's a case of getting Wubi working with dm-loop and comparing performance with the normal loopback and making a decision based in reliability and performance. As yet, I have no hard facts to make a decision either way on this. I suspect dm-loop will help with #1 in https://bugs.edge.launchpad.net/wubi/+bug/204133/comments/31 from my understanding of the code, but I have not verified this in practice.

> 3) It would be nice to be able to use the standard mount option name "-o sync" as opposed to have to create a new one for each filesystem "-o syncio". Not least because it will create a bit of special casing in other code (lupin/initramfs-tools). From a past discussion with Colin it emerged that the naming was to work around fuse -o sync handling.

Correct. For the -o sync fuse option to work, there needs to be considerable amount of effort required in honoring the -o sync flag in the kernel side of fuse and being passed down and handled in the user space side too. The simplest option was for ntfs-3g to pick up the -syncio option and handle it there rather than add generic -o sync support in fuse per se.

> 4) There is also a remount issue with loopdevices (see #201750). On top of that, there are separate remount issues for ntfs-3g (see #186117), even worse, it seems that remounting ntfs-3g ro/rw can produce data loss (https://bugs.edge.launchpad.net/wubi/+bug/186117/comments/10). Not sure if newer versions of fuse/ntfs-3g address this.

Not sure either.

> 5) Related to the above there are the shutdown issues mentioned by Colin in comment #19. I am not sure what is that state of that given A) the ntfs-3g patch and possibly B) dm-loop usage.

I suspect that the sync'ing of data with ntfs-3g will help considerably in ensuring that the data is being written back. My hope is that dm-loop will have less latency between writes to the loopback and data being written back than the normal loopback. It appears it should be better, but we need to test this hypothesis.

Colin

Revision history for this message
Agostino Russo (ago) wrote :

Martin from the points above I do not think that there is any strict kernel issue here and the bug can be set to invalid. In particular

1) nothing to do kernel-wise only involves switching to ntfs-3g syncio
2) is a separate issue: dm-loop in Intrepid (to be tested and used by wubi). We might want to open a separate bug report
3
) I guess we will keep "-o syncio" for the time being
4) already in separate bugs #201750 and #186117
5) to be tested after ntfs-3g and dm-loop and in any case will probably be a sysvinit umountroot issue

Changed in linux:
status: New → Invalid
status: New → Invalid
Revision history for this message
Colin Ian King (colin-king) wrote :

Martin,

Attached is the same modification as a debdiff for the Intrepid version of ntfs-3g. Can this be applied too?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ntfs-3g - 1:1.2506-1ubuntu2

---------------
ntfs-3g (1:1.2506-1ubuntu2) intrepid; urgency=low

   * Ensure ntfs-3g flushed dirty write data back to device
     added ntfs-3f mount option "syncio" which opens the underlying
     filesystem used by fuse with O_SYNC to ensure dirty write data is
     written back to ntfs immediately. This allows us to remove the
     vm.dirty sysctl hacks from wubi. (LP: #204133)

 -- Colin King <email address hidden> Wed, 09 Jul 2008 14:34:56 +0100

Changed in ntfs-3g:
status: New → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

Accepted into -proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in ntfs-3g:
status: In Progress → Fix Committed
Revision history for this message
Szabolcs Szakacsits (szaka) wrote :

Nice work.

Colin, FUSE doesn't support sync(2) but the 'sync' mount option should be ok because FUSE writes are always synchronous at the moment. Your patch fundamentally solved ntfs-3g 'sync' support minus some issues (cached metadata, disk caching, propagating the sync option to FUSE, portability, maybe others).

The patch is planned to be applied with some modifications within about a month as 'sync' mount option support.

Could you please point out where sync(2) is needed in FUSE if this is the case, or what do you mean it doesn't support what would be needed for more reliable WUBI operation? I think this would be a strong reason to raise the implementation priority.

The 'sync' performance impact is 20-3,000% depending on the workload but it was never more than 200% during normal usages.

Revision history for this message
Colin Ian King (colin-king) wrote :

Hi Szabolcs,

The reason for the O_SYNC option in the open on the NTFS device was mainly to ensure Wubi was writing data back to NTFS. Wubi relies on a loopback'd file system in a file in the ntfs fileystem, and hence there was a lot of latency between a write to the loopback, down through the ntfs filesystem via fuse. We were seeing problems with power-outages causing systems to become unbootable due to the write-back cached data latencies. One hacky approach was to tweak the vm settings to force writebacks as soon as possible to avoid blocks hanging around in the cache - this was a hack and not an elegant solution as it applies to all file systems across the board. Hence I added the O_SYNC option to ntfs-3g avoid this hack.

Basically we ultimately need a method of ensuring ntfs data and metadata is sync'd to disk asap to keep Wubi as robust as possible. As for performance penalty, I have also benchmarked the SYNC mode of operation and it is inline with the figure your report. This is probably OK for Wubi since most of the I/O operations are read and not writes, and I suspect robustness is more important than speed - but it will probably be an choice that the Wubi user can make.

Colin.

Revision history for this message
Martin Pitt (pitti) wrote :

Any testers?

Revision history for this message
Agostino Russo (ago) wrote :

I was on holidays, anyway resumed my testing yesterday night but I have no good news: 10X performance hit when copying an ISO with syncio vs copying the ISO without syncio...

I will have to try again today to confirm, there might be something wrong with my testing rig. Not sure yet what is different since last test (might it be that then I forgot to upgrade libntfs3g?). Can somebody else try that too?

Revision history for this message
Dave Morley (davmor2) wrote :

Confirming this issue.

I was struggling to get a speed of 2.6 mb/sec with syncio enabled.

However I was getting transfer speeds of <21 mb/sec without syncio enabled.

Revision history for this message
Colin Ian King (colin-king) wrote :
Download full text (4.4 KiB)

Ago, is Wubi using the normal loopback or dm-loop in this configuration? And what are we using the default vm sysctl's here, or are we using the Wubi vm sysctl's hacks too?

I'm surprised we are seeing a factor of 10 difference is data speeds. Here are some benchmarks I got when I first investigated this on Hardy back in July, which do indicate performance penalty, but not to such a high degree:

Test 1. Raw data I/O tests
---------------------------

Comparison of loopback and dm-loop block I/O. Block write/read tests of
a 6 GB data file on ext3 file system on the loopback.

Results are averages of 3 tests.

               sync? write read
loop no 46.2 MB/s 58.1 MB/s
loop yes 40.7 MB/s 40.0 MB/s

dm-loop no 44.1 MB/s 57.0 MB/s
dm-loop yes 49.8 MB/s 55.5 MB/s

Asynchronous writes of large files seem to be roughly the same with
loopback and dm-loop, probably because we are bottle necked on the disk
I/O performance underneath. Curiously dm-loop synchronous writes are
best performing which is totally unexpected.

Now for some real-world file transfer tests.

Test 2. Simple loopback tests
------------------------------

Loopback copy test. Test involved timing the copying a pre-cached
intrepid kernel source git tree to a loop mounted ext3 file system. The
test copied 26,438 files and 647MB of data.

              loop
              sync? time1 time2 time3 average Datarate

loop no 47.472 44.478 38.372 43.441 14.9 MB/s
loop yes 41.995 39.841 53.996 45.227 13.3 MB/s

dm-loop no 49.957 42.594 34.525 42.359 15.3MB/s
dm-loop yes 35.715 37.292 47.320 40.109 16.1MB/s

In real-world file copying tests, it appears that dm-loop performs
better than the normal loopback. Again, with dm-loop in sync mode
performs better than in async mode, which is most unexpected.

Test 3. Wubi style configuration
--------------------------------

The final tests try to simulate a Wubi configuration, with an ntfs-3g
file mounted using ntfs-3g via fuse, and inside this is an ext3
filesystem that is mounted using the loopback. In these tests, we test
with sync mode turn on and off on ntfs-3g and also on and off on the
loopback and dm-loop loop devices. Like test 2, we copy a pre-cached
intrepid kernel source git tree of 26,438 files and 647MB of data.

           loop ntfs-3g
           sync? sync? time1 time2 time3 average Datarate

loop no no 68.492 87.544 95.719 83.918 7.71 MB/s
loop yes no 183.243 225.088 n/a 204.166 3.17 MB/s

dm-loop no no 66.602 65.576 70.238 67.472 9.57 MB/s
dm-loop yes no 84.470 91.677 89.652 88.597 7.30 MB/s

loop no yes 87.237 94.472 91.491 91.06 7.10 MB/s
loop yes yes 193.071 192.259 192.041 192.456 3.36 MB/s

dm-loop no yes 78.685 80.738 89.807 83.076 7.79 MB/s
dm-loop yes yes 194.754 190.230 194.610 193.198 3.36 MB/s

With ntfs-3g mounted asynchronously, we see that dm-loop outperforms the
normal loopback b...

Read more...

Revision history for this message
Colin Ian King (colin-king) wrote :

..one more thing... my formatting of the benchmarks was completely messed up by the webform!

Revision history for this message
Agostino Russo (ago) wrote :

Colin, my results above are completely independent of Wubi. The test was done using a standard intrepid installation and mounting ntfs from there with and without syncio.

Revision history for this message
Colin Ian King (colin-king) wrote :

Hi Ago,

Here is my analysis from several tests on my server box:

1. ntfs-3g mount with syncio is indeed 10-12x slower than the default async ntfs-3g mount.

2. However, if one has a loop-back or dm-loop backed ext3 filesystem inside this then problem is less of an issue due to the buffer-caching at the ext3 layer - essentially the buffer cache at this layer will improve performance significantly.

3. Having ntfs-3g mounted as syncio is only useful with the ext3 filesystem loopbacked inside it to make sure that ext3 writes get written back through the fuse layer. At umount time of the looped back filesystem or calling sync will ensure that the files in the ext3 system are written back correctly to the ntfs-3g fuse layer.

4. Without the syncio mount option, we may see the ext3 filesytem get corrupted if the ntfs3g/ fuse writes are not fully written back to the ntfs filesystem.

I am unsure how Wubi initialises the filesystem during the Wubi install. However, if you are copying an ISO image or large amounts of data over to the ntfs-3g filesystem perhaps mounting it at first without the syncio option is fine. However, once Ubuntu is up and running from a ext3 filesystem which is loopback mounted inside ntfs-3g then a syncio mount option is probably the best way to make sure data is sync'd back correctly to reduce ext3 inconsistency and with enough buffer cache it should perform reasonably well.

Hope this makes sense.

Colin

Revision history for this message
Agostino Russo (ago) wrote :

Thanks a lot Colin, on my side I had already committed the required Wubi changes (rev 508), but was on hold because of the above, I will bring it to the attention of the Ubuntu-Installer team.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lupin - 0.22

---------------
lupin (0.22) intrepid; urgency=low

  * Disabling lupin-sysctl since its functionality has been replaced by
    syncio (LP: #204133). Lupin-sysctl is not actively removed if
    already installed since syncio also requires a change to menu.lst.

 -- Agostino Russo <email address hidden> Thu, 02 Oct 2008 23:27:48 +0100

Changed in lupin:
status: New → Fix Released
Agostino Russo (ago)
Changed in wubi:
status: In Progress → Fix Released
Revision history for this message
Steve Beattie (sbeattie) wrote :

Presumably, this should be fixed before the 8.04.2 iso respin.

Changed in ntfs-3g:
milestone: none → ubuntu-8.04.2
Revision history for this message
Brian Murray (brian-murray) wrote :

Reviewing this bug report which still needs SRU verification I don't see a clear test case.

Agostino - what needs to be done so we can verify this bug fix?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ntfs-3g - 1:1.2506-1ubuntu2~hardy1

---------------
ntfs-3g (1:1.2506-1ubuntu2~hardy1) hardy-backports; urgency=low

  * Automated backport upload; no source changes.

ntfs-3g (1:1.2506-1ubuntu2) intrepid; urgency=low

   * Ensure ntfs-3g flushed dirty write data back to device
     added ntfs-3f mount option "syncio" which opens the underlying
     filesystem used by fuse with O_SYNC to ensure dirty write data is
     written back to ntfs immediately. This allows us to remove the
     vm.dirty sysctl hacks from wubi. (LP: #204133)

ntfs-3g (1:1.2506-1ubuntu1) intrepid; urgency=low

  * Resynchronise with Debian. Remaining changes:
    - Add udeb shlibs entries for libntfs-3g28-udeb.
    - Bump debhelper build-dependency to 5.0.22 for dh_makeshlibs
      --add-udeb.
    - Create libntfs3g28-udeb and ntfs-3g-udeb.
    - Copy ntfs-3g into the initramfs. If ntfs-3g is used to mount the root
      filesystem (either directly or via a loopback mount), load the fuse
      module beforehand, and save ntfs-3g's PID (indirectly) in
      /var/run/sendsigs.omit.
    - Move ntfs-3g to /bin.
    - Move libntfs-3g.so.* to /lib. Adjust libntfs-3g.so symlink to match.
    - Add /sbin/mount.ntfs symlink to /bin/ntfs-3g.
    - Add a symlink for the mount.ntfs manual page too.
    - Pass basenames rather than full paths to pidof; create
      /dev/.initramfs/varrun if necessary (LP #150831).
    - Always add ntfs and ntfs-3g to sendsigs.omit (LP #181669).
    - Build against the external fuse libraries and add dependency on
      fuse-utils.

ntfs-3g (1:1.2506-1) unstable; urgency=HIGH

  * Urgency set to HIGH because it fixes a critical issue.
  * New upstream release (Closes: #479774).
  * Bump library package name to match current soname.

ntfs-3g (1:1.2310-1) unstable; urgency=low

  * New upstream release.
  * Bump library package name to match current soname.

 -- <email address hidden> (Christian A. Reiter) Tue, 23 Dec 2008 19:27:17 +0000

Changed in ntfs-3g:
status: Fix Committed → Fix Released
Revision history for this message
Steve Beattie (sbeattie) wrote :

Can someone who is affected by this test this out? I've tried to reproduce the problems using Agostino's recipe described in https://bugs.launchpad.net/wubi/8.04/+bug/204133/comments/21 but was unable to reproduce the problem (I successfully set up a system to boot off of a loopback mounted ext3 filesystem that resided on an ntfs filesystem). That said, I didn't see any issues while using ntfs-3g from hardy-proposed.

Revision history for this message
Brian Murray (brian-murray) wrote :

I installed Ubuntu 8.04.1 amd64 using wubi in Windows Vista and was able to boot the installed system successfully.

Revision history for this message
Agostino Russo (ago) wrote :

It might be difficult to reproduce the original bug (and not that relevant for 8.04.2 because the new syncio option enabled by the cking patch will not be used in practice in 8.04). IMO it is important to test that the new ntfs package does not introduce any regression in normal usage, when mounting with and without "-o syncio". To do that, see:

https://bugs.edge.launchpad.net/ubuntu/+source/ntfs-3g/+bug/229000

Where Martin tested another patch to ntfs-3g, and in particulare comment #5

https://bugs.edge.launchpad.net/ubuntu/+source/ntfs-3g/+bug/229000/comments/5

Revision history for this message
Steve Beattie (sbeattie) wrote :

I ran the testsuite from http://www.ntfs-3g.org/pjd-fstest.html referred to in Martin's comment in bug 229000. I saw no difference in output when using the version of ntfs-3g in the version released in hardy-updates, 1:1.2216-1ubuntu2, as opposed to the version in hardy-proposed, 1:1.2216-1ubuntu3, either with or without the sync mount option. I subsequently ran the testsuite repeatedly while also running iozone in auto mode against the version of ntfs-3g in proposed, and saw no corruption or other negative behaviors; once again, I did this twice, with the filesystem mounted with and without the sync option. As far as I can tell, this doesn't introduce any regressions.

Martin Pitt (pitti)
Changed in ntfs-3g:
status: Fix Released → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ntfs-3g - 1:1.2216-1ubuntu3

---------------
ntfs-3g (1:1.2216-1ubuntu3) hardy-proposed; urgency=low

  * Ensure ntfs-3g flushed dirty write data back to device
    added ntfs-3f mount option "syncio" which opens the underlying
    filesystem used by fuse with O_SYNC to ensure dirty write data is
    written back to ntfs immediately. This allows us to remove the
    vm.dirty sysctl hacks from wubi. (LP: #204133)

 -- Colin Ian King <email address hidden> Wed, 18 Jun 2008 10:32:31 +0100

Changed in ntfs-3g:
status: Fix Committed → Fix Released
Revision history for this message
Bratwurstler (j-sage) wrote :

This again happens in Karmic Koala. http://ubuntuforums.org/showthread.php?t=1300537

Revision history for this message
Agostino Russo (ago) wrote :

What is the output of cat /proc/cmdline?

Revision history for this message
Bratwurstler (j-sage) wrote :

It seems like upgrades from earlier wubi-installations (<9.10) are effected.

$ cat /proc/cmdline
root=UUID=E04C95C74C95993C loop=/ubuntu/disks/root.disk ro ROOTFLAGS=syncio quiet splash

Revision history for this message
GDB (gangadhar-biradar) wrote :

$ cat /proc/cmdline
root=UUID=B654EB6554EB26B9 loop=/ubuntu/disks/root.disk ro quiet splash

I am also facing the same issue. Upgraded the system to 9.10 from 9.04

Once I hit the shut down the black screen displays

Stopping Kernel oops catching service kerneloops
Stopping the winbind daemon winbind
Shutting down ALSA
Asking all remaining processes to terminate
Deconfiguring network interfaces
Deactivating swap
[25490:712192] Buffer I/O error on device loop0, logical block 3002840
This happens every time and I have to hard shut down the PC using power button.

Revision history for this message
Agostino Russo (ago) wrote :

If you upgraded from wubi jaunty to karmic, please see bug #468589

Revision history for this message
garik (garikgarik) wrote :

Hi,
I have this problem on Ubuntu 9.10 Koala installed with Wubi((((
I hibernated Ubuntu and now it is unbootable.
I have "kernel panic not syncing VFS unable to mount root fs on unknown block 8,1".
I have data hidden in ubuntu, how can I get it?
Thanks Igor

Revision history for this message
Agostino Russo (ago) wrote :

If it is an ntfs corruption issue, you can run chkdsk /r from windows on the drive containing the ubuntu installation and then try to boot again.
You can access your data by booting from a linux live CD / USB, then mounting the appropriate windows partition, then mounting the /ubuntu/root.disk file therein using the -o loop mount option.

e.g. assuming your windows partition is /dev/sda1 (change it as appropriate):

mkdir windowspartition
mkdir wubi
sudo mount /dev/sda1 windowspartition
sudo mount -o loop windowspartition/ubuntu/disks/root.disk wubi
#your files are now under the dir wubi
ls wubi
#to undo everything
sudo umount wubi
sudo umount windowspartition

Changed in wubi:
assignee: Agostino Russo (ago) → Kurupeththalage Sanath Rathna Kumara (ksrkumara)
status: Fix Released → New
Martin Pitt (pitti)
Changed in wubi:
assignee: Kurupeththalage Sanath Rathna Kumara (ksrkumara) → Agostino Russo (ago)
status: New → Fix Released
Revision history for this message
Rolf Leggewie (r0lf) wrote :

Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix".

Changed in lupin (Ubuntu Hardy):
status: New → Won't Fix
Revision history for this message
Raymond Michael Attila Erdey (rerdey) wrote :

I have the same problem in 14.04 and do not know how to rsolve it.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.