casper-rw fs not cleanly unmounted on persistent live USB shutdown

[Mike]

When shutting down gutsy tribe 2 from a persistent usb drive, I get a "failure" in umounting local filesystems. Running e2fsck on the casper-rw partition, which was formatted as ext2, shows errors on the casper-rw partition.

[Mike (mike0999)]

It is possible to make this failure go away by reverting to the edgy version of the upstart files.

I.e. by including a preferences file in the /etc/apt directory that contains the following and downgrading to these versions:

Package: upstart
Pin: version 0.2.7-7
Pin-Priority: 1001

Package: upstart-compat-sysv
Pin: version 0.2.7-7
Pin-Priority: 1001

Package: upstart-logd
Pin: version 0.2.7-7
Pin-Priority: 1001

[Brian Murray (brian-murray)]

Thanks for taking the time to report this bug and helping to make Ubuntu better. There is now a newer Tribe CD available and we were wondering if the problem exists with it. Additionally, it would be quite helpful to know Thanks in advance.

[Mike (mike0999)]

I'll try the new version.

It looks like your email got cut off. It looks like you were saying it
would be helpful to know something?

On 7/19/07, Brian Murray <email address hidden> wrote:
>
> Thanks for taking the time to report this bug and helping to make Ubuntu
> better. There is now a newer Tribe CD available and we were wondering
> if the problem exists with it. Additionally, it would be quite helpful
> to know Thanks in advance.
>
> ** Changed in: Ubuntu
> Assignee: (unassigned) => Brian Murray
> Status: New => Incomplete
>
> --
> failure to umount local filesystems - gutsy tribe 2
> https://bugs.launchpad.net/bugs/125702
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Mike (mike0999)]

I get the same error on Tribe3. I think I mentioned this, but just to be
sure, I am booting into persistent mode.

I am using a "Live USB" that I create in accordance with the Ubuntu Wiki.
It is a 4GB Drive. My casper-rw partition is a little over 2GB. I did also
try a different USB drive and got the same error with another USB drive.

I do exactly the same thing with Edgy as I do with Gutsy and I don't get the
error with Edgy. Based upon some customized iso's I created to get
persistence to work on Feisty, it looks like the error also occurred with
the Feisty iso (unless I pin the Feisty upstart files at the Edgy versions).

I have the casper-rw partition formatted as ext2. I don't have a swap
partition, so the system may be using a swap file.

The partition on which I am placing the "iso" files is formatted FAT32.

I don't have any other disks mounted - just the Live USB. Presumably the
casper-rw partition is mounted using unionfs. I don't know if this error
may somehow be related to the persistence issues, but persistence does seem
like it is working at least to some extent (e.g. I installed an application
and it was present when I rebooted; it didn't maintain my WPA password, but
it did maintain my keyring password).

If I execute the "sync" command before shutting down, I still get the same
error. After booting up again after I get the umount failure, I believe I
may have seen some errors (I am booting in debug mode). I don't remember
the error message exactly, but will try to note the details if I see it
again. I think it may have been error 110 or something like that, with
maybe some sort of message about inodes. My recollection is that the error
was associated with the casper-rw partition (I am pretty sure that that was
the case, but can double check that if I get the error again).

If it would be useful to you to have me burn an actual CD and try
persistence using the CD, I can try to do that as well if you think it will
provide useful information.

I am using an IBM X41 tablet with a mobile pentium. Obviously it has an
internal drive, but again it is not mounted.

I can't think of other information at the moment that might be useful, but
if there is any additional info I can provide, feel free to let me know.

Regards,

Mike

On 7/20/07, Michael Panepucci <email address hidden> wrote:
>
> I'll try the new version.
>
> It looks like your email got cut off. It looks like you were saying it
> would be helpful to know something?
>
> On 7/19/07, Brian Murray <email address hidden> wrote:
> >
> > Thanks for taking the time to report this bug and helping to make Ubuntu
> > better. There is now a newer Tribe CD available and we were wondering
> > if the problem exists with it. Additionally, it would be quite helpful
> > to know Thanks in advance.
> >
> > ** Changed in: Ubuntu
> > Assignee: (unassigned) => Brian Murray
> > Status: New => Incomplete
> >
> > --
> > failure to umount local filesystems - gutsy tribe 2
> > https://bugs.launchpad.net/bugs/125702
> > You received this bug notification because you are a direct subscriber
> > of the bug.
> >
>
>

[Mike (mike0999)]

My fstab file contains the following when booted up:

unionfs / unionfs rw 0 0
tmpfs /tmp tmpfs nosuid,nodev 0 0

My mtab file contains the following:

proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
tmpfs /lib/modules/2.6.22-8-generic/volatile tmpfs rw,mode=0755 0 0
varrun /var/run tmpfs rw,noexec,nosuid,nodev,mode=0755 0 0
varlock /var/lock tmpfs rw,noexec,nosuid,nodev,mode=1777 0 0
udev /dev tmpfs rw,mode=0755 0 0
devshm /dev/shm tmpfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
tmpfs /lib/modules/2.6.22-8-generic/volatile tmpfs rw,mode=0755 0 0
fusectl /sys/fs/fuse/connections fusectl rw 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev 0 0

I am not getting the error 110 error at the moment, so I am not sure if you
can rely upon that feedback.

When booting, it definitely is telling me that it is activating a swap
file. On shutdown, I get "ok" when it is deactivating the swap file. I
also get "ok" when it is unmounting temporary filesystems on shutdown.

On 7/21/07, Michael Panepucci <email address hidden> wrote:
>
> I get the same error on Tribe3. I think I mentioned this, but just to be
> sure, I am booting into persistent mode.
>
> I am using a "Live USB" that I create in accordance with the Ubuntu Wiki.
> It is a 4GB Drive. My casper-rw partition is a little over 2GB. I did also
> try a different USB drive and got the same error with another USB drive.
>
> I do exactly the same thing with Edgy as I do with Gutsy and I don't get
> the error with Edgy. Based upon some customized iso's I created to get
> persistence to work on Feisty, it looks like the error also occurred with
> the Feisty iso (unless I pin the Feisty upstart files at the Edgy versions).
>
>
> I have the casper-rw partition formatted as ext2. I don't have a swap
> partition, so the system may be using a swap file.
>
> The partition on which I am placing the "iso" files is formatted FAT32.
>
> I don't have any other disks mounted - just the Live USB. Presumably the
> casper-rw partition is mounted using unionfs. I don't know if this error
> may somehow be related to the persistence issues, but persistence does seem
> like it is working at least to some extent ( e.g. I installed an
> application and it was present when I rebooted; it didn't maintain my WPA
> password, but it did maintain my keyring password).
>
> If I execute the "sync" command before shutting down, I still get the same
> error. After booting up again after I get the umount failure, I believe I
> may have seen some errors (I am booting in debug mode). I don't remember
> the error message exactly, but will try to note the details if I see it
> again. I think it may have been error 110 or something like that, with
> maybe some sort of message about inodes. My recollection is that the error
> was associated with the casper-rw partition (I am pretty sure that that was
> the case, but can double check that if I get the error again).
>
> If it would be useful to you to have me burn an actual CD and try
> persistence using the CD, I can try to do that as well if you think it will
> provide useful information.
>
> I am using an IBM X41 tablet with a mobile pentium. Obviously it has an
> internal drive, but ...

[Mike (mike0999)]

Although it says it is activating a swap file when I am booting up, it looks
like my system may not be using swap. If I type "free" in a terminal, I get
the following:

             total used free shared buffers cached
Mem: 1026464 572388 454076 0 45980 198412
-/+ buffers/cache: 327996 698468
Swap: 0 0 0

On 7/20/07, Michael Panepucci <email address hidden> wrote:
>
> My fstab file contains the following when booted up:
>
> unionfs / unionfs rw 0 0
> tmpfs /tmp tmpfs nosuid,nodev 0 0
>
> My mtab file contains the following:
>
> proc /proc proc rw 0 0
> sysfs /sys sysfs rw 0 0
> tmpfs /lib/modules/2.6.22-8-generic/volatile tmpfs rw,mode=0755 0 0
> varrun /var/run tmpfs rw,noexec,nosuid,nodev,mode=0755 0 0
> varlock /var/lock tmpfs rw,noexec,nosuid,nodev,mode=1777 0 0
> udev /dev tmpfs rw,mode=0755 0 0
> devshm /dev/shm tmpfs rw 0 0
> devpts /dev/pts devpts rw,gid=5,mode=620 0 0
> tmpfs /lib/modules/2.6.22-8-generic/volatile tmpfs rw,mode=0755 0 0
> fusectl /sys/fs/fuse/connections fusectl rw 0 0
> tmpfs /tmp tmpfs rw,nosuid,nodev 0 0
>
> I am not getting the error 110 error at the moment, so I am not sure if
> you can rely upon that feedback.
>
> When booting, it definitely is telling me that it is activating a swap
> file. On shutdown, I get "ok" when it is deactivating the swap file. I
> also get "ok" when it is unmounting temporary filesystems on shutdown.
>
> On 7/21/07, Michael Panepucci <<email address hidden> > wrote:
> >
> > I get the same error on Tribe3. I think I mentioned this, but just to
> > be sure, I am booting into persistent mode.
> >
> > I am using a "Live USB" that I create in accordance with the Ubuntu
> > Wiki. It is a 4GB Drive. My casper-rw partition is a little over 2GB. I
> > did also try a different USB drive and got the same error with another USB
> > drive.
> >
> > I do exactly the same thing with Edgy as I do with Gutsy and I don't get
> > the error with Edgy. Based upon some customized iso's I created to get
> > persistence to work on Feisty, it looks like the error also occurred with
> > the Feisty iso (unless I pin the Feisty upstart files at the Edgy versions).
> >
> >
> > I have the casper-rw partition formatted as ext2. I don't have a swap
> > partition, so the system may be using a swap file.
> >
> > The partition on which I am placing the "iso" files is formatted FAT32.
> >
> > I don't have any other disks mounted - just the Live USB. Presumably
> > the casper-rw partition is mounted using unionfs. I don't know if this
> > error may somehow be related to the persistence issues, but persistence does
> > seem like it is working at least to some extent ( e.g. I installed an
> > application and it was present when I rebooted; it didn't maintain my WPA
> > password, but it did maintain my keyring password).
> >
> > If I execute the "sync" command before shutting down, I still get the
> > same error. After booting up again after I get the umount failure, I
> > believe I may have seen some errors (I am booting in debug mode). I don't
> > remember the error message exactly, but will try to note the details if I
> >...

[Mike (mike0999)]

Error still shows up in tribe 5.

On 7/21/07, Michael Panepucci <email address hidden> wrote:
>
> Although it says it is activating a swap file when I am booting up, it
> looks like my system may not be using swap. If I type "free" in a terminal,
> I get the following:
>
> total used free shared buffers cached
> Mem: 1026464 572388 454076 0 45980 198412
> -/+ buffers/cache: 327996 698468
> Swap: 0 0 0
>
>
> On 7/20/07, Michael Panepucci <email address hidden> wrote:
> >
> > My fstab file contains the following when booted up:
> >
> > unionfs / unionfs rw 0 0
> > tmpfs /tmp tmpfs nosuid,nodev 0 0
> >
> > My mtab file contains the following:
> >
> > proc /proc proc rw 0 0
> > sysfs /sys sysfs rw 0 0
> > tmpfs /lib/modules/2.6.22-8-generic/volatile tmpfs rw,mode=0755 0 0
> > varrun /var/run tmpfs rw,noexec,nosuid,nodev,mode=0755 0 0
> > varlock /var/lock tmpfs rw,noexec,nosuid,nodev,mode=1777 0 0
> > udev /dev tmpfs rw,mode=0755 0 0
> > devshm /dev/shm tmpfs rw 0 0
> > devpts /dev/pts devpts rw,gid=5,mode=620 0 0
> > tmpfs /lib/modules/2.6.22-8-generic/volatile tmpfs rw,mode=0755 0 0
> > fusectl /sys/fs/fuse/connections fusectl rw 0 0
> > tmpfs /tmp tmpfs rw,nosuid,nodev 0 0
> >
> > I am not getting the error 110 error at the moment, so I am not sure if
> > you can rely upon that feedback.
> >
> > When booting, it definitely is telling me that it is activating a swap
> > file. On shutdown, I get "ok" when it is deactivating the swap file. I
> > also get "ok" when it is unmounting temporary filesystems on shutdown.
> >
> > On 7/21/07, Michael Panepucci < <email address hidden> > wrote:
> > >
> > > I get the same error on Tribe3. I think I mentioned this, but just to
> > > be sure, I am booting into persistent mode.
> > >
> > > I am using a "Live USB" that I create in accordance with the Ubuntu
> > > Wiki. It is a 4GB Drive. My casper-rw partition is a little over 2GB. I
> > > did also try a different USB drive and got the same error with another USB
> > > drive.
> > >
> > > I do exactly the same thing with Edgy as I do with Gutsy and I don't
> > > get the error with Edgy. Based upon some customized iso's I created to get
> > > persistence to work on Feisty, it looks like the error also occurred with
> > > the Feisty iso (unless I pin the Feisty upstart files at the Edgy versions).
> > >
> > >
> > > I have the casper-rw partition formatted as ext2. I don't have a swap
> > > partition, so the system may be using a swap file.
> > >
> > > The partition on which I am placing the "iso" files is formatted
> > > FAT32.
> > >
> > > I don't have any other disks mounted - just the Live USB. Presumably
> > > the casper-rw partition is mounted using unionfs. I don't know if this
> > > error may somehow be related to the persistence issues, but persistence does
> > > seem like it is working at least to some extent ( e.g. I installed an
> > > application and it was present when I rebooted; it didn't maintain my WPA
> > > password, but it did maintain my keyring password).
> > >
> > > If I execute the "sync" command before shutting down, I still get the
>...

[Mike (mike0999)]

Error still present in the Gutsy beta candidate iso.

[Mike (mike0999)]

bug still present in Gutsy beta

[Scott James Remnant (Canonical) (canonical-scott)]

Bug has not been triaged: there is insufficient information here to identify the fix, only sufficient information to confirm that there is a bug.

[Mike (mike0999)]

Is there any additional information that you need from me?

[waxhell (waxhell)]

I may have this bug in Gutsy release, however in my instance, I get a seg fault.
Persistence enabled on a pen-drive. Experiencing some data corruption as a result of this (not all files are copied properly to the casper-rw partition)
Reference Bug #147117

* Umounting local filesystem
   umount2: Device or resource busy
   umount: /cdrom busy - remounted read only
   Segmentation fault

[Jones D. Le (joneslee85)]

I confirmed the waxhell's report. The writing features for home folder sometimes works sometimes don't. The system writing is not working yet.

live-initramfs of Debian does not produce the error, however persistent writing is not yet working. I think Ubuntu and Debian should merge in together and work toward a solution. I woud love to have my USB load from RAM instead of Flash memory.

[waxhell (waxhell)]

Ext3 seems to help the data corruption (probably at the expense of the drive lifetime).

I've set my drive to run e2fschk every boot, and it seems to be better.

[Mike (mike0999)]

Also, at least for the problem I experienced, reverting to the edgy
version of upstart made at least that particular problem go away
(although I don't know if that will meet your needs).

On Oct 29, 2007 10:23 PM, waxhell <email address hidden> wrote:
> Ext3 seems to help the data corruption (probably at the expense of the
> drive lifetime).
>
> I've set my drive to run e2fschk every boot, and it seems to be better.
>
>
> --
> failure to umount local filesystems - gutsy tribe 2
> https://bugs.launchpad.net/bugs/125702
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[nox216 (nox216)]

Confirmation that I'm seeing similar results with corrupted/unmounted casper partitions in the released Gutsy, using a LiveUSB install.

[Ron Z (rzwaal)]

It seems to me that the problem exists with the live cd as well. I've tried this with the AMD64 and the i386 versions of ubuntu and kubuntu on an AMD64 laptop and desktop and an intel laptop all with the same segfault unmounting the filesystem on shutdown/reboot from the cd.

The problem seems to be different on Feisty (at least for the live cd I haven't tried Feisty on my pendrive). Umount fails when trying to unmount the /cow directory. I think I read somewhere that this was a typo in the umountfs script.

I think we only notice this when using persistent mode on the pendrive because that's the only time it's an issue. It's doing the same thing from the live cd, it just doesn't matter because no files are actually written. Is this possibly fixed in Hardy Heron? I lack the necessary skills to play with early Alpha builds.

[jpolizo (jim-eighthsage)]

I believe I am running into a variant of this same problem. I installed 7.10 on a USB drive and it works great when using just the live CD mode. When I use persistence (second partition is ext2) it works great the first time I boot and then it steadily degrades after that. In one case, I had a corrupted /etc/sudoers file and when I looked at this on another machine the file had the contents of mtab stuffed right into the middle of it. In other cases it has blown up the X configuration so that X cannot start.

No matter how bad the persistent mode boot gets, it wall always boot fine in "live" mode.

If I do an fsck on the casper-rw drive, I get lots of errors after a reboot. If I fix the errors and then delete everything on the casper-rew drive, it will boot again fine in "persistent" mode though obviously that's not very persistent.

I've also tried 'sync' before a shutdown of the system but that doesn't seem to help. I also tried doing a suspend and unsuspend and that seemed to work fine with no corruption.

[jpolizo (jim-eighthsage)]

I tried persistence in the 6.10 version and I think there are problems there too although much less than in 7.10.

When the 6.10 version is shutting down, I would see I/O errors to the USB device. I ran fsck on the casper partition and it did find a number of errors. Most notably, mtab had errors which is the same file that I saw get stuffed into the sudoers file when 7.10 got corrupted. The degree of corruption was dramatically less in 6.10 than in 7.10 and I seem to be able to reboot after the fsck and have things work; the level of damage appears less but looks to still be there. I don't think this bug was introduced in 7.10, it just looks to be worse in 7.10.

[mikeXYZ (qmikeq)]

For me, same as reported by waxhell (above). Issue with: Kubuntu 7.10 on UFD, Live Persistent.
I've tried using ext3 and I get the same segmentation fault. I did not test the extent of data corruption using ext3 (vs ext2), but I suspect it will show up upon repeated re-boots.
I have also seen the issue regarding “trying to unmount the /cow directory.”
The extent of data corruption gets worse upon the second re-boot (X messes up so badly that the issue is moot on the third re-boot).

I have no problems with the straight Live Kubuntu on UFD (without persistence). Ron Z's comments on that are noted.

Specifically, as said by waxhell (2007-10-29):
“I may have this bug in Gutsy release, however in my instance, I get a seg fault. Persistence enabled on a pen-drive. Experiencing some data corruption as a result of this (not all files are copied properly to the casper-rw partition) Reference Bug #147117
Umounting local filesystem [fail]
umount2: Device or resource busy
umount: /cdrom busy - remounted read only
Segmentation fault [fail]

[mikeXYZ (qmikeq)]

As waxhell also reported, it seems to be working better with casper-rw formatted as ext3 instead of ext2. In fact, everything thus far seems fine after 4 re-boots and conducting persistent activity each time. But I do get some messages:

When I log out and re-boot from the live persistent session, I do get the following message (instead of the message I reported in my post above when I used ext2 for casper-rw):

Unmounting local filesystems ... [failed]
casper is resyncing snapshots and caching reboot files.
Please remove the disk, close the tray (if any), and press Enter to continue.
[After pressing Enter, then:]
Will now restart.

No problems, EXCEPT twice so far I got:
Error-KDesktop
The process for the file protocol died unexpectedly.
[and the Desktop was empty of my icons]

Then I pressed OK, then Refresh Desktop did not help, then Control+Alt+Backspace, and it cleared it up so everything was back on the Desktop (all the persistent stuff I had put there).

Finally, if anyone is interested in the GRUB boot stanza I'm using:

title Kubuntu 7.10 LIVE Persistent
root (hd0,0)
kernel /casper/vmlinuz boot=casper ramdisk_size=1048576 root=/dev/ram rw quite splash persistent
initrd /casper/initrd.gz

I don't know about wear-out issues (with either ext2 or ext3), but that is a separate issue.

[Richard Musil (richard-musil)]

I concur to what jpolizo wrote. I spent the whole day playing with this setup (i.e. Gutsy on 1GB USB as per http://www.pendrivelinux.com/2007/09/28/usb-ubuntu-710-gutsy-gibbon-install/) and noticed following details when booting into persistent:

1) some settings are stored, some not (like new user). Even for Appearence->Fonts some font settings get stored (almost all sizes) and some not (AA setting).
2) mtab was consistently corrupted after boot
3) some files are created with access mode 0000 (i.e. .ICEauthority-c, or some .gconf files)
4) modes for those files with 0000 can sometimes be changed sometimes I got I/O error
5) when the system is shutting down, at certain moment it displays the message "Remove the disk and press ENTER". When removing USB at this point I observed immediate errors on block I/O (on console) and casper-rw got completely messed afterwards.
6) even after forementioned message there is some activity on flash (led blinks) before shutdown.

All these factors basically makes the system degrade so fast, it does not even survive few reboots without loosing some critical data.

[jpolizo (jim-eighthsage)]

One possibly important thing I failed to mention is that I'm testing on a laptop with a dual-core Centrino.

[Ron Z (rzwaal)]

I ended up following Mikes? advice and used ext3 instead of ext2 for both
the casper-rw and the home-rw partitions. I also followed the instructions
for removing the eject cd message and the instructions here (
http://www.atworkonline.it/~bibe/ubuntu/custom-livecd.htm) to customize the
installation. I updated all the software and adjusted the packages. I've
been using this setup for almost a month without problems on both my 8Gig
verbatim and my 2G SanDisk Cruzer. I use both almost daily and just checked
them and they're error free.

I've made a 1G partition for the squash filesystem (I've loaded quite a few
extras) and on the 8G stick a 700M casper-rw partition and the rest of the
device is for my home-rw partition. On the 2G stick I have a 300M casper-rw
and the rest (700M) for my home partition.

Hope this helps. (I should check my email a little more often:-)

Ron

On Jan 9, 2008 2:40 PM, jpolizo <email address hidden> wrote:

> I believe I am running into a variant of this same problem. I installed
> 7.10 on a USB drive and it works great when using just the live CD mode.
> When I use persistence (second partition is ext2) it works great the
> first time I boot and then it steadily degrades after that. In one
> case, I had a corrupted /etc/sudoers file and when I looked at this on
> another machine the file had the contents of mtab stuffed right into the
> middle of it. In other cases it has blown up the X configuration so
> that X cannot start.
>
> No matter how bad the persistent mode boot gets, it wall always boot
> fine in "live" mode.
>
> If I do an fsck on the casper-rw drive, I get lots of errors after a
> reboot. If I fix the errors and then delete everything on the casper-
> rew drive, it will boot again fine in "persistent" mode though obviously
> that's not very persistent.
>
> I've also tried 'sync' before a shutdown of the system but that doesn't
> seem to help. I also tried doing a suspend and unsuspend and that
> seemed to work fine with no corruption.
>
> --
> failure to umount local filesystems - gutsy tribe 2
> https://bugs.launchpad.net/bugs/125702
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[aselya1 (aaron-selya)]

I also wanted to confirm that this problem still exists in 8.04 alpha 1 and alpha 3 for Ubuntu and Xubuntu. Also, some random daily-builds in the middle. I haven't checked Kubuntu or any daily-builds since. Are there any log files we can view/post to make triaging this easier? Can anyone figure out why Edgy's udevs work?

[Christopher Barth (cbarth)]

I pinned upstart at edgy and downgraded by adding edgy repositories to sources.list, but my 7.10 hangs on halt
From init1 I get hang or usr prompt (I can call for shutdown as many times as I want; system won't halt), from x I get the shutdown splash with moving bar than flashing cursor

How do have e2fsck run every boot like waxhell's installation? I assume we set -y or -p in an init somewhere but I don't see a every mount stanza in e2fsck.conf

Also, has anyone got permanence to work with some of the upstart alternatives?

[Mike (mike0999)]

It always works for me that way, with both feisty and gutsy.

On Jan 18, 2008 6:29 AM, Christopher Barth <email address hidden> wrote:
> I pinned upstart at edgy and downgraded by adding edgy repositories to sources.list, but my 7.10 hangs on halt
> >From init1 I get hang or usr prompt (I can call for shutdown as many times as I want; system won't halt), from x I get the shutdown splash with moving bar than flashing cursor
>
> How do have e2fsck run every boot like waxhell's installation? I assume
> we set -y or -p in an init somewhere but I don't see a every mount
> stanza in e2fsck.conf
>
> Also, has anyone got permanence to work with some of the upstart
> alternatives?
>
>
> --
> failure to umount local filesystems - gutsy tribe 2
> https://bugs.launchpad.net/bugs/125702
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Mike (mike0999)]

Maybe something different about our systems.

On Jan 20, 2008 10:56 AM, Michael Panepucci <email address hidden> wrote:
> It always works for me that way, with both feisty and gutsy.
>
>
> On Jan 18, 2008 6:29 AM, Christopher Barth <email address hidden> wrote:
> > I pinned upstart at edgy and downgraded by adding edgy repositories to sources.list, but my 7.10 hangs on halt
> > >From init1 I get hang or usr prompt (I can call for shutdown as many times as I want; system won't halt), from x I get the shutdown splash with moving bar than flashing cursor
> >
> > How do have e2fsck run every boot like waxhell's installation? I assume
> > we set -y or -p in an init somewhere but I don't see a every mount
> > stanza in e2fsck.conf
> >
> > Also, has anyone got permanence to work with some of the upstart
> > alternatives?
> >
> >
> > --
> > failure to umount local filesystems - gutsy tribe 2
> > https://bugs.launchpad.net/bugs/125702
> > You received this bug notification because you are a direct subscriber
> > of the bug.
> >
>

[Mike (mike0999)]

It's been so long since I looked at this, I don't remember for sure,
but I think there are actually 3 upstart files that need to pinned at
the edgy level. I used aptitude to downgrade, so whenever I pinned
one, the other two were downgraded automatically. But, you may want
to make sure all three are being downgraded. Don't know if that could
be the problem.

On Jan 20, 2008 10:57 AM, Michael Panepucci <email address hidden> wrote:
> Maybe something different about our systems.
>
>
> On Jan 20, 2008 10:56 AM, Michael Panepucci <email address hidden> wrote:
> > It always works for me that way, with both feisty and gutsy.
> >
> >
> > On Jan 18, 2008 6:29 AM, Christopher Barth <email address hidden> wrote:
> > > I pinned upstart at edgy and downgraded by adding edgy repositories to sources.list, but my 7.10 hangs on halt
> > > >From init1 I get hang or usr prompt (I can call for shutdown as many times as I want; system won't halt), from x I get the shutdown splash with moving bar than flashing cursor
> > >
> > > How do have e2fsck run every boot like waxhell's installation? I assume
> > > we set -y or -p in an init somewhere but I don't see a every mount
> > > stanza in e2fsck.conf
> > >
> > > Also, has anyone got permanence to work with some of the upstart
> > > alternatives?
> > >
> > >
> > > --
> > > failure to umount local filesystems - gutsy tribe 2
> > > https://bugs.launchpad.net/bugs/125702
> > > You received this bug notification because you are a direct subscriber
> > > of the bug.
> > >
> >
>

[waxhell (waxhell)]

Christopher:

I do an tune2fs -c 1 <device> to force a check everytime the volume is mounted.

My setup still is without errors, but I don't use my persistent thumbdrive that often (maybe once every two weeks or so).

[Christopher Barth (cbarth)]

Thanks all, pinning preferences at first post fixes the primary issue in whatever tribe I would get if I ran update last week.

I found my new problem with ttys is related to a video bug since everything works fine when I plug ubuntu into an intel915 but totally fails with my ati9600. Must of updated something else at the same time. Tracking it down; going to try folding fbcon into initramfs-update next. Slow going when ttys and X are both down on my primary PC.

[Scott James Remnant (Canonical) (canonical-scott)]

This bug still baffles me.

Upstart is barely involved in shutting down the machine, so it should make no difference which version is running.

[Mike (mike0999)]

Just a thought - and I am the first to admit my limited knowledge - but maybe it is not completely the code that is shutting down the system that is causing the issue. Maybe the more recent version of upstart is somehow creating a system state that is not created by the earlier version of upstart. and it is this system state that is not handled correctly by the code that shuts down the system. I.e. the code that shuts down the system could be identical, but maybe it is trying to handle a state that does not exist when the earlier version of upstart is used. I have no idea whether that is possible . . . just thinking out loud. I.e. it is an interaction, not just the upstart code.

[Ron Z (rzwaal)]

Here is another thought. I'm not confident at all that the problem is with
shutting down the system at all. I've noticed that files created on the
pendrive when running from the pendrive are often corrupt.
One of the things I use pendirive linux for is downloading podcasts while on
the road. Often the files end up scrambled together randomly. It's not the
podcatching software causing the problem. The files seem to be intact when
they are downloaded. The file scrambling seems to happen when transferring
files from the pendrive to the mp3 player, memory card or other pendrive.
Also, configuration files set up on the device often seem to be corrupt (ie.
the config files for frostwire, amule and some other apps were corrupt on my
pendrive. Copying working files from my harddrive fixed the problem.
I recently tried to build an Ubuntu pendrive for a colleague by copying the
fat32 files from my pendrive over to his. The files appeared to be the
correct size but were corrupt. I ended up having to boot to Windows and
transfer the files from there to get it to work. Then I had to copy my
working config files for several apps separately to get them to work.
Perhaps the whole problem is that files on usb devices are not being handled
properly at all when running from the pendrive?

On Wed, Mar 5, 2008 at 11:21 PM, Mike <email address hidden> wrote:

> Just a thought - and I am the first to admit my limited knowledge - but
> maybe it is not completely the code that is shutting down the system
> that is causing the issue. Maybe the more recent version of upstart is
> somehow creating a system state that is not created by the earlier
> version of upstart. and it is this system state that is not handled
> correctly by the code that shuts down the system. I.e. the code that
> shuts down the system could be identical, but maybe it is trying to
> handle a state that does not exist when the earlier version of upstart
> is used. I have no idea whether that is possible . . . just thinking
> out loud. I.e. it is an interaction, not just the upstart code.
>
> --
> failure to umount local filesystems - gutsy tribe 2
> https://bugs.launchpad.net/bugs/125702
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[mikeXYZ (qmikeq)]

Ron Z, that's not the case for my experience with this bug (corrupt files already there somehow)--it's clearly a shutdown issue for me & have seen this elsewhere, too.

btw,
As for the flash drive, if I have a working, bootable flash drive (e.g., live persistent Kubuntu 7.10), I clone it easily with a one-liner dd command:
dd if=/dev/sdc of=/dev/sdd conv=noerror,notrunc
(sdc is the working UFD to be cloned; sdd is the empty, identical UFD; you can specify any bs=Block_Size other than the default bs=512 (bytes); etc., as you probably already know :) )
That copies everything, including the boot mechanism (e.g., GRUB in MBR of sdc pointing at the installed OS on sdc) => sdd is ready to boot.

[Richard Musil (richard-musil)]

mikeXYZ: If it is shutdown issue, what about files with 0000 access mode? I believe these are created on the fly.

[Ron Z (rzwaal)]

Mike,
Thanks for the info. I tried cloning a drive as per your instructions below
and it worked flawlessly. Still, copying the files using cp -rf seems to
corrupt them during the copy process. Also, I blew away the casper-rw
partition after cloning and ended up with the same issues I had before (that
is with the config files).
I think I'm conflating several unrelated issues here and assuming they share
a common cause.
Still I'm finding the situation with the file corruption the most irritating
of all. Using ext3 works for me and mypen drive has held up OK so far.

On Sat, Mar 29, 2008 at 11:15 AM, mikeXYZ <email address hidden> wrote:

> Ron Z, that's not the case for my experience with this bug (corrupt
> files already there somehow)--it's clearly a shutdown issue for me &
> have seen this elsewhere, too.
>
>
> btw,
> As for the flash drive, if I have a working, bootable flash drive (e.g.,
> live persistent Kubuntu 7.10), I clone it easily with a one-liner dd
> command:
> dd if=/dev/sdc of=/dev/sdd conv=noerror,notrunc
> (sdc is the working UFD to be cloned; sdd is the empty, identical UFD; you
> can specify any bs=Block_Size other than the default bs=512 (bytes); etc.,
> as you probably already know :) )
> That copies everything, including the boot mechanism (e.g., GRUB in MBR of
> sdc pointing at the installed OS on sdc) => sdd is ready to boot.
>
> --
> failure to umount local filesystems - gutsy tribe 2
> https://bugs.launchpad.net/bugs/125702
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[drewster1829 (arruud-hotmail)]

I am getting the exact same errors with my persistent 8.04 LiveUSB drive. Using ext3 on the casper-rw partition helped a little, but I still get a string of buffer I/O errors on shutdown.

Since Edgy Eft is no longer supported (as of 4-25-08), it seems rather hard to find the versions of the 3 upstart packages above. I have the Edgy LiveCD, but I have no idea how to revert the package on the LiveUSB stick, or if it's even possible using the Live CD.

Isn't this bug really an error somewhere in the shutdown script? I wish I knew how to fix it, as it's been an issue for quite some time now.

[Steve Dodd (anarchetic)]

drewster: you could boot from the Live CD, then install dpkg-repack and use it reconstruct the packages you need.

[Mike (mike0999)]

drewster:

I think you can also boot up into your persistent system, add the preferences file to /etc/apt as described above, change your sources.list repositories to edgy rather than hardy, and then run sudo apt-get install [list the upstart files that were pinned]. That should cause the files to downgrade to the edgy version.

Right after that, change your sources.list repositories back to hardy. I am pretty sure I was able to get that to work. I don't recall for sure, but the system may have hung on me after the downgrade. But, when I rebooted, it worked.

Steve's approach may be more of a sure thing (and the approach I describe is subject to the corruption issues that we are attempting to address), but probably worth giving that a try because it is easy to do and might work for you.

[drewster1829 (arruud-hotmail)]

I must be getting a different bug (though some of the symptoms are similar). I've never had the unmount error or segmentation fault, but I always seem to get about a dozen or so "Buffer I/O error"s on shutdown (after the message about removing the disc and pressing Enter...even if I don't press enter, I eventually get the errors...of course I don't unplug the USB drive, but if I do, I get the same result), resulting in dramatic corruption when it was formatted ext2.

Using ext3 for casper-rw buys me some time, but I still get the errors (I'm not sure if there's any file corruption--no evidence of any, yet). Also, it adds "aborting journal on device sdb3" errors.

Steve: Thanks for your help. I ended up doing this (had to download the deb for dpkg-repack manually...the Edgy repositories are no longer available), along with booting the persistent system and installing "dpkg-dev" so I could use dpkg-scanpackages to make a local source repository for the Edgy upstart debs I made, as well as using the preferences file at top to downgrade the packages. However, it didn't help with the buffer I/O errors. (I assume this fixed the umount error, which I've never had...maybe the bug is gone with Hardy, or is only applicable on some installations?)

Mike: I tried doing that initially, but the Edgy repositories are no longer available (I'm assuming because Edgy is no longer supported). That's why I had to find a way to do it with the LiveCD of Edgy (I was able to find a download of the iso still), and use dpkg-repack to make the three debs for upstart.

So, in my case (which apparently is a different bug), the old upstart packages didn't help. Thank you, Steve and Mike, though, for helping me downgrade those packages!

[Richard Musil (richard-musil)]

Since it seems to be still alive issue, I would like to add recent experience.

I have tried to do the same setup as before using new 8.04.1 version. Using the same 1GB stick I used before led to the same problems (with corruption and missing files). Then I got 8GB stick, brand new, and did the install, but instead of having like 250MB for casper-rw I set up like 2GB for it. And so far the thing works OK, no corruption, no messed modes. It even updates correctly from automatic update.

Maybe I was hitting some limit with 1GB stick before, I do not know. The rw filesystem was however not full when the problems occured. So it may related to some underlying dependencies or something not directly related to filesystem usage, but its size in general.

[Steve Dodd (anarchetic)]

I created a persistent setup on a USB stick using the 'official' USB installer in Intrepid. I haven't had any corruption problems (yet), but the ext3 journal on the casper-rw (loopback) filesystem still seems to get replayed on start up, which presumably means it wasn't cleanly unmounted (or the blocks weren't flushed to disk in time.)

Scott: is there any specific information you need? Any debug we can insert anywhere?

[Steve Dodd (anarchetic)]

Just checked that again, and the journal does get replayed, so the rw fs isn't being umounted cleanly. I also noticed block device I/O errors on shutdown.

Digging into this a bit deeper, some observations:

* Whatever this line from /etc/rc0.d/S40umountfs is supposed to do ..

PROTECTED_MOUNTS="$(sed -n '0,/^\/[^ ]* \/ /p' /proc/mounts)"

.. it seems to end up matching the whole of /proc/mounts on a casper boot, possibly because the mount point "/" appears twice.

* The casper-rw filesystem is mounted on /cow, which isn't accessible once the boot has completed. Remounting / (the unionfs / aufs) ro doesn't remount /cow ro (I wouldn't expect it to.)

Given that casper sets up /cow, presumably this is primarily a casper bug - regardless of the fact that the edgy -> gutsy changes to upstart seemed to make it worse.

I hope nobody minds me retitling this bug slightly.

[Steve Dodd (anarchetic)]

Here's a patch, based on casper 1.152 (the version in intrepid.) I don't know if this is the best way to achieve this, but it seems to work for me - this problem can be tested in Virtualbox using a live CD image and a hard disk partition labelled 'casper-rw'.

[Steve Dodd (anarchetic)]

I've rebuilt the intrepid ISO and this patch seems to work. I'd upload an rdiff but it seems to be ~200Mb - anybody got any ideas about how to make a squashfs that can be sanely rdiffed against the official live CD image? I've tried with and without -nolzma using squashfs-tools 1:3.3-1ubuntu2 ..

[Steve Dodd (anarchetic)]

Humph. Looks we need to sync and sleep as well. New patch attached ..

[Aleksey Makarenko (magalex28)]

I think, that Alex Roper in this https://bugs.launchpad.net/ubuntu/+source/casper/+bug/371477 line-up of the same problem. You can explain the details, what can we do to solve this problem? I understand that we need to fix the scripts in filesystem.squashfs on our USB-flash drives? Please, explain the details...
Thank you. Sorry for my bad English...

[Steve Dodd (anarchetic)]

Hi Aleksey,

Yes, I think this is the same problem. This fix requires changing two files, one in the initrd.gz, one in the main filesystem. Fortunately you can make this second change in the casper-rw filesystem, so there's no need to rebuild filesystem.squashfs ..

The first _is_ a bit of a pain because it means unpacking and repacking the initrd. The quick and dirty way to do that (i.e., as root), is to copy the initrd.gz to /tmp and do something like (untested):

cd /tmp
gunzip initrd.gz
mkdir ird
cd ird
cpio -i <../initrd
patch scripts/casper <<EOM
diff -Nur casper-1.152.orig/scripts/casper casper-1.152/scripts/casper
--- casper-1.152.orig/scripts/casper 2008-10-25 23:46:45.000000000 +0100
+++ casper-1.152/scripts/casper 2009-04-03 00:57:25.000000000 +0100
@@ -396,6 +396,8 @@
     mount -t ${cow_fstype} -o ${cow_mountopt} ${cowdevice} /cow || panic "Can not mount $cowdevice on /cow"

     mount -t ${UNIONFS} -o noatime,dirs=/cow=rw:$rofsstring ${UNIONFS} "$rootmnt" || panic "${UNIONFS} mount failed"
+ [ -d "${rootmnt}/cow" ] || mkdir "${rootmnt}/cow"
+ mount /cow "${rootmnt}/cow" -o move

     # Adding other custom mounts
     if [ -n "${PERSISTENT}" ]; then
@@ -592,3 +594,5 @@
     exec 2>&7 7>&-
     cp casper.log "${rootmnt}/var/log/"
 }
+
+# vi:ts=4:et:
EOM
find | cpio -H newc -o >../newird
cd ..
gzip newird

And then overwrite the initrd.gz on your live USB/CD with the generated 'newird.gz'.

The second file is easier, boot the live CD/USB with persistence enabled and then (as root) do:

patch /etc/init.d/casper <<EOM
diff -Nur casper-1.152.orig/debian/casper.init casper-1.152/debian/casper.init
--- casper-1.152.orig/debian/casper.init 2008-10-25 23:46:45.000000000 +0100
+++ casper-1.152/debian/casper.init 2009-04-10 14:49:09.000000000 +0100
@@ -75,6 +75,17 @@
  prompt=
     fi

+ # remount cow ro
+ OPTS=`grep '^aufs / aufs ' /proc/mounts | awk '{print $4}'`
+ if [ -n "$OPTS" ]; then
+ OPTS=`echo "$OPTS" | sed -e 's/xino=.*,/noxino,/' -e \
+ 's/cow=rw/cow=ro/' -e 's/^rw/ro/'`
+ mount / -no remount,$OPTS
+ mount /cow -no remount,ro
+ sync # you'd think mount would do this ..
+ sleep 5 # could probably get away with less
+ fi
+
     for path in $(which halt) $(which reboot) /etc/rc?.d /etc/default $(which stty); do
         cache_path "$path"
     done
@@ -113,3 +124,5 @@
         exit 3
         ;;
 esac
+
+# vi:ts=4:et:
EOM

I've not yet tested this fix with jaunty, but it seemed to work for me with intrepid.

[Steve Dodd (anarchetic)]

Damn, some of the lines seem to have wrapped.. see attached text file.

[Steve Dodd (anarchetic)]

Meh, you'll also need to make those patch commands:

patch ... <<'EOM'

(Note the single quotes around the delimiter..)

[Aleksey Makarenko (magalex28)]

Thank you very much for the help, Steve!
I would try. I will write as act in Jaunty.
I hope soon I can sleep peacefully ... :)

[Aleksey Makarenko (magalex28)]

OK! I have done as you wrote.
And now I have only two errors in completing the work and such an e2fsck output:

root@aleksey-laptop:/home/aleksey# e2fsck /dev/sdb2
e2fsck 1.41.4 (27-Jan-2009)
casper-rw was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
casper-rw: 559/181056 files (2.5% non-contiguous), 24354/722925 blocks
root@aleksey-laptop:/home/aleksey#

This is normal? Can I be calm for my data?

[Aleksey Makarenko (magalex28)]

  Oh... I'm sorry... "casper-rw was not cleanly unmounted" only appears after the first reboot. Now everything is OK. I'm still have two errors on shutdown/reboot, but after it "casper-rw clean".
  Ha-ha! It's working! :) Thank you, Steve! Great job!

[Scott James Remnant (Canonical) (canonical-scott)]

Well done for your work Steve, so this definitely looks like a Casper bug? In which case I'll mark the Upstart task Invalid and leave the casper one open

[Steve Dodd (anarchetic)]

FWIW, I've justed tested and I can confirm this problem is still present in Jaunty, as Aleksey and bug #371477 suggest.

[Steve Dodd (anarchetic)]

Here's a new version of the patch, this time leveraging the existing SHOWMOUNTS functionality that I didn't notice the first time around. This has the advantage that it works with multiple stacked .squashfs's. It's still aufs-specific, but's that fine for jaunty and I gather the hope for karmic is that there will be union mounts supported directly within the VFS ..

[Steve Dodd (anarchetic)]

Marking this confirmed as a number of people have seen this problem. I hope that doesn't breach ettiquette ..

[soldonz (webdum)]

I am kinda new with linux and am a bit confused by the new and old version fo the patch and how they should be applied.

Would you kindly provide an instruction on how to apply the latest patch or maybe just provide a script that we can just download it and run it as root?

[Steve Dodd (anarchetic)]

(I've already emailed soldonz, but for anybody else who wants to fix a 9.04 live USB/CD setup..)

I've uploaded a patched version of the initrd to a hosting service, I don't know how long they keep stuff:

http://www.datafilehost.com/download-2e60bf8e.html

The md5 is 0263a21a04c56bf5d24c8f6c16f9acec.

The /etc/init.d/casper file also needs patching, but this can done from within the persistent system itself.

[soldonz (webdum)]

Thanks Steve,

I just tried the patch, now there are only a couple io error messages shown at shutdown screen, compare to a full page of io error messages.

Here are the steps I have taken:

1. create a ubuntu 9.04 live USB drive from windows using the batch file from pendrivelinux.com (using 3GB persistent file)
2. replace /casper/initrd.gz with the patched initrd.gz, provided by Steve
3. restart the computer and boot with the USB drive.
4. install patch software (sudo apt-get install patch)
5. apply the second part of Steve's patch with the added quote. (see attached file)
6. reboot

the following error messages were shown at shutdown screen.

------------------------------------------------------------------
[ 226.987862] end_request: I/O error, dev sdb, sector 31447
[ 226.987915] Buffer I/O error on device sdb1, logical block 31384
------------------------------------------------------------------

the patch definitely cleared up most of the error messages but not all of them in my case.

[soldonz (webdum)]

I just gave it another go, this time using the casper file that Steve sent me over the email. The problem is now fixed completely. (not sure what happened to my patch on casper file, it said it was done successfully)

thank you so much, Steve.

It surprise me how few instances of this problem have been reported on the internet. I suspect that a lot of people sees it as a bad sector problem with their usb drive. I, for one, spent hours doing surface scan, low level format, trying to solve this issue.

If it wasn't for Steve's post on bug 371477, I would never have found this thread and solve this problem.

If this bug occurs in every ubuntu persistent install, then it may be a good idea to give it a bit more attention? the pendrivelinux.com people should make a note of it if the problem does indeed occur everytime.

[Steve Dodd (anarchetic)]

On Fri, Jun 19, 2009 at 02:11:45AM -0000, soldonz wrote:

> I just tried the patch, now there are only a couple io error messages
> shown at shutdown screen, compare to a full page of io error messages.

I still see those sometimes - I don't think they're harmful. The
important thing to check is that your casper-rw file is getting
unmounted cleanly. If you look in your kernel log (dmesg | less) you
shouldn't see warnings about recovery being required any more (assuming
you used ext3 for the casper-rw file, which is the default if you use
the live USB creator.)

Steve

[Steve Dodd (anarchetic)]

On Fri, Jun 19, 2009 at 04:14:02AM -0000, soldonz wrote:

> If this bug occurs in every ubuntu persistent install, then it may be a
> good idea to give it a bit more attention?

Realistically, it's too late to do anything about Jaunty (9.04), and the
mechanism used on the live CDs for union mounts is supposed to be
changing in Karmic (9.10), so I guess there's no point in incorporating
the current patch as it stands. Once the new system is in place I'll try
to adapt the patch and get it incorporated before release ..

> the pendrivelinux.com people should make a note of it if the problem
> does indeed occur everytime.

If you know anybody at that project, feel free to point them here! It's
worth noting that, if you use ext3 for the casper-rw file, this bug
doesn't seem to cause data loss in practice, but even so it seems a bit
cavalier not to umount the users filesystem before shutting down!

[Gabriel Molina-Terriza (ventisca)]

Two days ago I made my first persitent USB with Jaunty, so this thread comes up handy for me. I am trying the fix right now, but wouldn't it be nice if you post also the casper file that soldonz seems to have tested all right? (I mean with respect to the patch that seems to have some minor errors still)

Thanks a lot in any case for the work!

[Steve Dodd (anarchetic)]

As I said, I think there is still sometimes a residual I/O error or two, but they don't seem to be harmful - casper-rw seems to get umounted OK (check the kernel log to make sure, or fsck the file when booted from a different system.) Attached is the full replacement /etc/init.d/casper for 9.04 - make sure you're running in persistent mode when you replace it or the change won't be .. persistent ;-)

[Gabriel Molina-Terriza (ventisca)]

Thanks again Steve. I don't see any difference from my casper and the new one... but... I'll tell you once I reboot if it works :-)

Thank you once more!

[rene7705 (rene7705)]

Ok, i've spent 3 days now trying to get persistent storage on an Ubuntu USB installation to work.
It fails on karmic too....

Not even when using the initrd.gz + /etc/init.d/casper update listed in this thread, do i get it to work :((

The basic install and startup of Ubuntu from USB works, but saving _anything_ (including only a change of desktop background), results in a failure to boot up after restart.

The error consistently seems to be these write-errors on shutdown of the USB installation:
* possibility 1:
 Buffer : I/O error @somesector (repeated slightly more than a dozen times on different sectors)
* possibility 2:
 end_request : I/O error (also slightly more than a dozen times on different sectors)

I tried:
* Creating with usb-creator-gtk (Administration -> USB Startup Disk Creator), on Ubuntu 9.04 jaunty, to a good 4Gb USB key, using the latest .iso for 9.04 to install:
 * Using persistent storage at all: fail.
 * Using a persistent space of only 300Mb: fail.
 * Disabling HAL polling before reboot on first change of running Ubuntu USB : fail
 * (Tag 1) Updating the initrd.gz before booting from the usb-key with the one provided by Steve in this thread (comment #60), then when booted perform the update described in comment #66:
  Only 2 write-errors on shutdown of USB installation, but fails to boot next time round.

* Creating with usb-creator-gtk on Ubuntu 9.10 Karmic, to the same 4Gb USB key, using either 9.04 iso for install or even the 9.10-beta.iso:
 * Fail, error remains.
 * Using (Tag 1): Fail.

* Creating with "Portable Linux" (http://rudd-o.com/new-projects/portablelinux):
 * Using ubuntu 9.04 install iso: Fail.

[Steve Dodd (anarchetic)]

@rene:

Weird. How is the boot failing - any error message? You might need to boot with "quiet" and "splash" removed from the kernel command line to see what's going on .. adding "debug" or "debug=y" to the kernel command line will give even more info (one will debug to the screen, the other to a file in the ramdisk, can't remember which is which.)

I booted my 4Gb persistent jaunty install today, so I promise it's not a figment of my imagination this does work!

Oooh .. another thought: are you booting on a machine that uses LVM or kernel RAID? If so you might be hitting bug #385305 ..

[rene7705 (rene7705)]

Hmm... here's a funny one:
First time after applying the patch @ runtime of karmic USB: fail.
But the second time i tried it, it does boot, with settings from last time remembered! :D

I'm still not sure about the particulars, or if i need to worry about it failing to boot from USB in the future..
But for now, thanks heaps for the patch..

I was experimenting just now with restoration of a Norton Ghost14 image of a tweaked Windows XP (but created in VirtualBox) to a real partition on my boot harddrive.
Turns out you can restore it, but i couldn't get it to boot up, not even in safe mode.

For reasons too embarassing to go into, i didn't have any ubuntu live CDs when i tried this, so this USB installation is kinda saving my neck here.. Thanks again, steve.

[rene7705 (rene7705)]

Hey, i've been twiddling with the usb key a bit more, and i noticed that if i change anything on the persistent filesystem (aka make changes while running it, then reboot), i have to physically unplug the usb briefly, then plug it in again, or it won't boot up (freezes on fetching bootloader of usb key).

I can live with that, of course, but mentioning it might help someone with this problem.

[Steve Dodd (anarchetic)]

Aaaah, yes, that rings a bell. /etc/init.d/casper does an "eject" on the 'cdrom' device - even if the cdrom device is actually a thumb drive. Some thumb drives apparently just disable themselves in response to that, until they are un/re-plugged. Debian mention this in their equivalent script, and work-around it - might be worth someone back-porting the fix. Not sure that it should matter whether you use persistent mode or not though.

[hernad-b-out-ba (hernad-b-out-ba)]

> ... even if the cdrom device is actually a thumb ...

Thank you Steve Dodd, you are a life saver :)

That resolve a huge issue with rebooting my usb livecd remix.

[soldonz (webdum)]

I just did a fresh install of ubuntu 9.10 on my USB drive and there was no more error message about bad sectors.. etc.
Can someone confirm that this issue has been addressed?

[James (marsrover7)]

I cannot. I get no errors that I can see on a regular boot, but running a check on the casper-rw partition shows that it gets corrupted after nearly every boot. I first noticed it when it kept gradually loosing disk space and ran a check on the partition.

I'm running 10.04.1 Lucid

[doperative (doperative)]

James wrote on 2010-12-13: I cannot. I get no errors that I can see on a regular boot, but running a check on the casper-rw partition shows that it gets corrupted after nearly every boot ..

Something similar here, after a few boots I keep getting errors when running fsck on the USB device. I do see lots of "i/o errors' just before shutdown, so it looks like it isn't being cleanly unmounted. Right now home-rw is marked full and there is a corrupt "d????????? ? ? ? ? ? .gvfs" directory in the home dir.

[doperative (doperative)]

update: Would flush have anything to do with the problem, I notice a lot of flushes on boot that disappear after a while, and this one seems to be stuck.

$ps -ef

root 375 2 0 13:20 ? 00:00:00 [flush-8:16]

[doperative (doperative)]

updated update: Would selecting shutdown instead of reboot have anything to do with it? I notice casper-rw is mounted twice, once as aufs and again as /media/casper-rw. If I unmount /media/casper-rw it doesn't appear to have any affect.

Filesystem Size Used Avail Use% Mounted on

aufs 2.2G 1.8G 330M 85% /

/dev/sdb2 2.2G 1.8G 330M 85% /media/casper-rw

[doperative (doperative)]

update to the updated update: I have a brand new 8GB USB device, fresh install of Lubuntu, I still get these errors. Running fsck gives:

Block bitmap differences:

Free inodes count wrong

Inode bitmap differences:

home-rw was not cleanly unmounted, check forced.

[Frank Wang (yafrank)]

The same casper-rw file system corruption happens here in the Lucid 10.04.2. I was first tricked to believe there were bad blocks, then realized I was wrong after more test and Google to this thread. I know Steve's patch is not for the Lucid but I still tried, hoping it will work. If the initrd.lz is patched, system hangs at boot. The later one against /etc/init.d/capser alone in "persistent" boot won't help either. I still get corrupted casper-rw partition after shutdown or reboot.
I also tried the backstore of Sysresccd-2.0.1, which is similar to the casper-rw file for persistent storage. It has the backstore file system corruption problem as well. Don't have time to test other live distro CDs. Is it a common problem for live USB?

[Jayson Legaspi (legaspi-jayson)]

up for mr frank... but on the 11.04 distro, were having the same problem on boot

[wodny (z-launchpad-wodny-org)]

Ubuntu 10.04.2 LTS (Lucid)
Linux 2.6.35-25-generic

I've also observed this problem. And just like others - I thought it was a hardware problem at first.

I've experimented with a couple of configurations and it seems that the current one decreased the number of filesystem errors.

So the theory is that USB flash drives like to do some block juggling even after signalling they have finished and they lose power halfway to data really being flushed.

Ext3 doesn't really help here. Both Ext2 and Ext3 can be marked clean but have errors. Errors may appear in data not in metadata. Then even fsck won't see them. For example MD5 would.

The problem seems to be computer and USB flash drive dependant.

I've decided to try the configuration with a separate "casper-rw" Ext2
partition on the USB flash drive as casper initrd scripts allow it. It is used instead of a loop-mounted "casper-rw" file on FAT.

"Showmounts" boot option enabled.

I remount filesystems to "ro" before shutdown in the order below:
- /cdrom,
- / (prepend noxino, rw->ro),
- /cow.
I do sync, some sleep.
At the end I send raw SCSI sync command (sg3-utils) - not every USB flash drive understands this.
I've also disabled the eject command (as Debian does for USB drives).

I attach scripts I've used.

I think that a kernel/hardware guru is needed here.

[Stéphane Graber (stgraber)]

I just made a few changes to that part of casper which "may" fix this bug in the process.
As I don't have the environment to easily reproduce and test the fix, I'm attaching the new /etc/init.d/casper in the hope that someone can test it and see if that fixes your problem.

[Stéphane Graber (stgraber)]

Marking the bug as incomplete until I get someone to test the new /etc/init.d/casper I attached above.
Tomorrow's daily builds of Precise should also be shipping that init script if that's easier for you to test with that, just ensure casper is at version 1.296.

[wodny (z-launchpad-wodny-org)]

Have You considered using the tip describing the noxino trick and making the root filesystem ro during shutdown?

[James Kingdon (james-kingdon)]

Looks like I've run into the equivalent problem after doing a full install onto a USB stick (I got frustrated with live/persistent approach). After any shutdown -h the system reports that the root fs was not cleanly unmounted and does an fsck & reboot. I'll try adding a sync/sleep to /etc/init.d/umountroot and see if that helps.

Out of curiosity, why remount read only as opposed to just unmounting?

Regards,
James.

[wodny (z-launchpad-wodny-org)]

You can still use tools from the RO partition while it should give the same type of write-free safety as unmounting,

[Schugy (ich-schugy)]

Is it adequate to put version 1.296 onto the casper-rw partition or do we have to create a new initrd?
I've tried to run version 1.296 manually with sudo sh /etc/init.d/casper stop but the error remains. I've verified that this is the new file (bigger).

Grub is configured as follows:

menuentry "Kubuntu persistent 11.04 32 Bit" {
    loopback loop /kubuntu-11.04-desktop-i386.iso
    linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=/kubuntu-11.04-desktop-i386.iso persistent noeject noprompt
    initrd (loop)/casper/initrd.lz

I've also tried Kubuntu 11.10 with this script:

menuentry 'Kubuntu 11.10 32 Bit persistent'{
 set root='(fd0)'
 linux (fd0)/vmlinuz root=UUID=310870bd-add2-4584-85b6-75c4e039a65e boot=casper locale=de_DE bootkbd=de console-setup/layoutcode=de persistent file=/cdrom/preseed/kubuntu.seed
 initrd (fd0)/initrd.lz}

Still no luck.

[Schugy (ich-schugy)]

Have tried http://cdimage.ubuntu.com/daily-live/current/precise-desktop-i386.iso (md5sum 19c7877726a0d1b20512647834d5ef85) looped in grub2. integrity check was successfull.
This one doesn't unmount casper-rw cleanly too. Verified the /etc/init.d/casper in this daily image is version 1.296.

[Stéphane Graber (stgraber)]

Ok, thanks for your tests. I'll try to spend some time on this issue so we can have it fixed for 12.04.

[Ben Greear (greearb)]

I built a custom 12.04 with some added kernel patches and added a few packages (and removed others).

It generally seems to work fine, but at some point, I started seeing errors about /dev/loop1.

It seems this is the casper-rw file, and it is not checked by fsck on startup. Why not?

I put the usb stick in a different machine, manually ran fsck on the casper-rw file, and
it found and fixed a bunch of errors. Now the usb key boots up fine again, but I'm worried
that this will happen again...

[Ben Greear (greearb)]

Maybe this will do the trick?

https://bugs.launchpad.net/ubuntu/+source/casper/+bug/996668

[Ben Greear (greearb)]

Well, I think I finally have this working...mainly due to lots of help from the aufs maintainer and mailing
list.

Will attach the /etc/init.d/umountrootfs and casper initrd scripts (for Ubuntu 12.04).
You have to re-spin the live-cd initrd to get the casper changes in place. The
umountrootfs just needs to be copied to /etc/init.d/umountrootfs (on the live-cd
image you are re-spining).

[Ben Greear (greearb)]

Well, I think I finally have this working...mainly due to lots of help from the aufs maintainer and mailing
list.

Will attach the /etc/init.d/umountrootfs and casper initrd scripts (for Ubuntu 12.04).
You have to re-spin the live-cd initrd to get the casper changes in place. The
umountrootfs just needs to be copied to /etc/init.d/umountrootfs (on the live-cd
image you are re-spining).

[Ben Greear (greearb)]

Here is the /etc/init.d/umountroot file.

The main trick to properly unmounting aufs root is this:

mount -n -o remount,noxino / || echo "Remount noxino / failed: $?"
# This next fails, but over-all, things work, so leaving it in in case
# it has some partial benefit.
mount -n -o remount,mod:/cow=ro / || echo "Remount /cow(ro) failed: $?"
mount -n -o remount,ro /cow || echo "Remount /cow (ro) failed: $?"
mount -n -o remount,ro /

[wodny (z-launchpad-wodny-org)]

The noxino idea is not new. I've mentioned this in comment #85 almost half a year ago. And it itself was based on an earlier comment.

What do You mean that those 3 lines fail? Have You tried doing it in right order, also mentioned in this thread?
  /cdrom -no remount,ro
  / -no remount,noxino,<the rest of options copied>
  /cow -no remount,ro
It's been even tested.

You mention the mailing list - would You provide a link to the thread?

And is this discussion still relevant? I thought Ubuntu has switched to overlayfs.

[Ben Greear (greearb)]

On 05/17/2012 05:20 PM, wodny wrote:
> The noxino idea is not new. I've mentioned this in comment #85 almost
> half a year ago. And it itself was based on an earlier comment.
>
> What do You mean that those 3 lines fail? Have You tried doing it in right order, also mentioned in this thread?
> /cdrom -no remount,ro
> / -no remount,noxino,<the rest of options copied>
> /cow -no remount,ro
> It's been even tested.
>
> You mention the mailing list - would You provide a link to the thread?
>
> And is this discussion still relevant? I thought Ubuntu has switched to
> overlayfs.

I didn't read the whole thread..but Ubuntu 12.04 is still busted. After
my changes, including noxinfo, it works for me.

I hear they are moving away from aufs to overlayfs, but I guess that
is post 12.04.

Here's the mail archives from sourceforge:

http://sourceforge.net/mailarchive/forum.php?thread_name=4FB5807D.6000307%40candelatech.com&forum_name=aufs-users

Thanks,
Ben

--
Ben Greear <email address hidden>
Candela Technologies Inc http://www.candelatech.com

[wodny (z-launchpad-wodny-org)]

SourceForge has eaten the link to the ubuntu-devel-discuss thread as it considered it an email address. Reposting with a shortened link would be useful. I would have posted there, but without the MessageID it would end up outside of the thread.