casper-rw larger than 2GB not created (because dd is called with bs=final-size-of-persistfile)

[jruss]

Binary package hint: usb-creator

I am using usb-creator 0.1.15 from the jaunty 9.04 beta DVD. I get "dd: invalid number" errors in .usb-creator.log file. I thought it was a limitation of vfat file system, but I tested it by using "dd if=/dev/zeros of=/dev/null" and it seems like dd won't allow block sizes >= 2G. Usb-creator finishes without notifying the user that the was an error and no casper-rw file was created. The usb drive will boot, but is not persistent.

The problem is in line 134 of /usr/share/usb-creator/install.py:
        popen(['dd', 'if=/dev/zero', 'bs=%s' % persist, 'of=%s/casper-rw' % target, 'count=1'])

In the past the block size was set to 1 byte, but that led to problems: https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/313364. Now things have swung the opposite direction towards creating a single (count=1) large block equal to the size of casper-rw. However this isn't allowed by dd for all block sizes and should be corrected.

This bug https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/321544 has patches that seem to be a reasonable approach to address these issues in addition to the 4GB file size limit of vfat.

[Travis Whitaker (pi-boy-travis)]

Confirmed. I have tried this on multiple machines with multiple, MD5 verified images, with multiple USB drives. The resulting drive does boot, but there is no casper-rw file at all. Perhaps usb-creator should offer the option to create and use an ext2 partition for persistence. This would fix both the 2GB limit by dd, and the 4 GB limit by vfat.

[Travis Whitaker (pi-boy-travis)]

USB persistence is an important feature. This bug should probably be at medium, or maybe even high priority. I find it to be an essential tool when demonstrating Ubuntu to new users.

[jruss (jruss)]

The 2 GB limit is irrelevant of file system. dd doesn't know about or care about file systems. So creating another partition with another file system would have no affect on dd. Block size is not the same as volume/partition size. The volume size is the block size times the number of blocks. Having very large block sizes is a bad idea for a persistent drive. There is a reason dd doesn't allow block sizes larger than 2 GB, we shouldn't try to get around that.

There is a simple patch in the bug report I linked to that will change it to use 1KB block size. Usb-creator would work correctly with that patch and the persistent drive would be created faster and it would work better for its intended purpose. I think creating a 2nd partition on the drive introduces a lot more complexity and gives little benefit. Remember the persistence file isn't for your data. You can store that in a folder on the regual vfat partition. The persistence file is just for changes in settings, software updates, etc. Four GB should be plenty for the vast majority of users. Also being able to access data on multiple operating systems is often important for flash drive users. A 2nd partition will not be accessible in windows regardless of file system.

[Tony Gambone (tonygambone)]

A workaround is to create a smaller casper-rw with the GUI and then make it larger. It's still constrained by the 4GB vfat limit, though. For example, to increase it by 1GB:

dd if=/dev/zero bs=1024k count=1024 >> /media/disk/casper-rw
e2fsck -f /media/disk/casper-rw
resize2fs /media/disk/casper-rw

I increased mine to 3GB this way, with no apparent problems.

[jruss (jruss)]

Usb-creator is just a script. It is quite easy to just apply the patch from https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/321544 to the text file. If you don't feel comfortable doing that. Then you can manually just find and change 2 lines. It is quite easy. Here are the relevant lines from the patch. Just find them in the usb-creator script and replace the '-' version with the '+' version.

- popen(['dd', 'if=/dev/zero', 'bs=1', 'of=%s/casper-rw' % target, 'count=%s' % persist])
+ popen(['dd', 'if=/dev/zero', 'bs=1024', 'of=%s/casper-rw' % target, 'count=%s' % (persist/1024)])

- persist = arg
+ persist = int(arg)

After this you can run usb-creator like normal and you will be fine with any persistence file < 4GB. This is what I did and I everything worked great for me.

[Travis Whitaker (pi-boy-travis)]

Thanks to jruss and mogrify for the info.

If I change those lines, will it interfere if I install a later version of usb-creator?

Thanks again for the help!

[Tokenekie (tokenekie)]

To expound on what Pi Boy said, here is a comment I left that might have been overlooked:

https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/321544/comments/10

[jruss (jruss)]

Sorry I missed that comment, I responded in that thread: https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/321544/comments/11

To repspond to Pi Boy any changes you make to usb-creator will just be overwritten during the update. It will replace the script with the new one in the updated package, or maybe it would give a prompt asking if you want to keep the file, but I doubt that. You are usually only prompted during an update if it is overwriting a know user config file, which the usb-creator script is not.

[Miam Miam (supermiammiam-deactivatedaccount)]

Is this bug fixed ?
I am at colledge and a lot of people use live usb key for working.
Most of them had this problem because they chose an extra space bigger than 2 GB.
That is a pity ! because the first impression hey had on ubuntu. It something that doesn't work properly !!!

[Kurt Pfeifle (pfeifle)]

I confirm this bug to still be present in Karmic (daily-live from 10th of July).

As an aside: this daily-live also reports conflicting versions about usb-creator. See also bug #398124 :

* "dpkg -l usb-creator":
  reports "0.1.16"

* "usb-creator --version":
  reports "0.0.11"

[Travis Whitaker (pi-boy-travis)]

Here's an easy workaround:

-Download the 0.2.2 tarball: https://launchpad.net/ubuntu/karmic/+source/usb-creator/0.2.2/+files/usb-creator_0.2.2.tar.gz
-Extract it and find the install.py in the scripts folder
-Replace your install.py located in /usr/share/usb-creator with the new install.py
-Laugh heartily

I just tried it, and it works like a charm!

[Evan (ev)]

This was fixed earlier in the Karmic cycle. usb-creator now spawns dd in the following manner:
            dd_cmd = ['dd', 'if=/dev/zero', 'bs=1M', 'of=%s' %
                      os.path.join(str(self.target), 'casper-rw'),
                      'count=%s' % self.persist]