Persistence Slider Wrong; Takes more than 3 hours to produce largest persistence file

[John S. Gruber]

Binary package hint: usb-creator

Does not affect Intrepid

usb-creator fails to create a persistence file as large as requested, producing the message dd: writing "`/media/disk/casper-rw': File too large". Creating the largest file can take 3 hours 25 minutes. This consumes most of one core in my two core system, mostly in system state.

Running 0.1.11 on jaunty alpha 3

Analysis:

The casper-rw file will contain the persistence file system and is created on a FAT file system since that is what is compatible with SYSLINUX (and its sibling ISOLINUX used for live-cds). http://en.wikipedia.org/wiki/FAT_file_system states that the maximum file size for a FAT file system is 4GB minus 1 byte and this is the filesize actually created by the package. However in its user interface usb-creator unrealistically offers persistence sizes larger than this when used on larger usb memory sticks.

In the bazaar trunk of usb-creator at revision 54 (and for Intrepid) the blocksize was 1M. After that it became possible to specify the size in bytes. In revision 60 the blocksize (bs) was set to 1 byte. For the largest persistent filesize the dd command executes 4 billion output operations when using this blocksize.

Since the casper-rw file will be made into an ext3 file system, any bytes greater than n*filesystemblocksize will be wasted. For an ext3 file system this filesystemblocksize will be 1K, 2K, or 4K. For a file of several gigabytes one would expect the blocksize to be 4K bytes.

Thus the maximum file size offered by usb-creator should be 4G minus 4K. This should get rid of the error message and produce more realistic expectations for those using usb-creator.

The persistence size offered by the user can be divided by 1024 and that number of 1024 byte blocks can be written. Any smaller granularity will be wasted as the EXT file system will necessarily ignore it. I suggest using 1024 here to give maximum precision for small persistence file sizes with 1K blocksizes. Using a dd blocksize of 1024 reduces the persistence file creation time down to just 6 minutes so further increases in dd blocksize to 2K or 4K are probably unnecessary. If there is a use case for creating a persistent file with one byte accuracy a second dd command can be added with a blocksize of 1 byte using the seek parameter to finish the last odd 1-1023 bytes (persistence-size%1024).

On the FAT file system it may suffice to seek and write to the last byte of the persistence file, but this doesn't save much time because of all the writes to the FAT tables for allocation, and it could produce hard to find errors should the FAT file system ever be replaced with one which supports files with holes (as software evolves). I tried this dd command as a test but persistence file creation still took several minutes.

Note that with the blocksize change any persistence size passed manually to the included install command will have to be in bytes, dd command shorthand of, say, 3K or 6M or 4G will again create an type error.

I built and tested a package sucessfully with the proposed patches I'm including.

[John S. Gruber (jsjgruber)]

See https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/313364 which reports the long creation time issue.

[jruss (jruss)]

I am using the jaunty 9.04 beta live DVD to create a persistent USB stick. Usb-creator is trying to create a single large block and is failing as a result. Block sizes >= 2G aren't allowed by dd. Here is the error in my .usb-creator.log when I tried to create a 3.9 GB file:

['dd', 'if=/dev/zero', 'bs=4162095690', 'of=/media/disk/casper-rw', 'count=1']
dd: invalid number `4162095690'

Are these patches in 9.04 beta? If not should I start a separate bug report to address this? This isn't just an inconvenience it causes usb-creator to fail silently without reporting any errors to the user. The USB will boot but is not persistent because there is no casper-rw file.

[Travis Whitaker (pi-boy-travis)]

@jruss No, the patches aren't in the final release. I am experiencing this same issue.

[Tokenekie (tokenekie)]

jruss,

Thanks for the clarification in your comment in bug #302485. I took a look at your patch your provided here but I ran into a few issues I have questions about.

The patch you wrote was intended for usb-creator 0.1.11 however I am still experiencing the same issues with version 0.1.16. I took a look at the lines you were trying to modify in install.py and found that the original line in 0.1.11:

popen(['dd', 'if=/dev/zero', 'bs=1', 'of=%s/casper-rw' % target, 'count=%s' % persist])

was different in 0.1.16. I'm very unclear as to what popen does. I took a look at "subprocess" but I'm not that well versed in python/ubuntu to understand what the class Popen actually does.

In any case, The following line of code is what appears in the 0.1.16 version of usb-creator. What should we replace this line to?

popen(['dd', 'if=/dev/zero', 'bs=%s' % persist, 'of=%s/casper-rw' % target, 'count=1'])

[jruss (jruss)]

I didn't actually apply these patches directly. I updated mine (0.1.15) manually. Read the man page of dd. The size of the file that dd creates is equal to the product of the blocksize and the count. In the 0.1.11 the block size is 1 byte and the count is 'persist' (ie the number of bytes you wanted your persistence file to be). Having a block size of 1 byte was problematic and that prompted this bug report. Apparently later someone decided to fix it but swung things to the opposite extreme. So you see in 0.1.16 that now the blocksize is set to 'persist' (the size you want) and the count is set to 1. So instead of a huge number of blocks that are too small, now it is trying to make a single block that is too large (dd won't allow > 2GB). To fix this you set the block size to something appropriate like in the 1KB-1MB range for most common cases, and then you calculate the count so that it gives you the filesize you requested.

If you look at John Gruber's patch at the top of this bug report you see this change:

popen(['dd', 'if=/dev/zero', 'bs=1024', 'of=%s/casper-rw' % target, 'count=%s' % (persist/1024)])

He set the block size to 1024 (ie 1 KB) and the count is set to persist/1024 (ie the size of your persitence file divided by the blocksize). That is why I referred to this thread in my bug report. This patch for the old problem in 0.1.11 is correct and what is currently in 0.1.16 is incorrect.

[Evan (ev)]

This is fixed in Karmic. usb-creator calls dd with a block size of 1MB.

[Oded Arbel (oded-geek)]

1M blocks work very very bad for me - it overloads the CPU but creation takes forever on a 4G thumbdrive through a non-highspeed USB connection. Worse - I can't stop the process because stopping the USB creator or even killing dd doesn't clear the kernel buffers that are used in the transfer. If I try to "unmount" the drive, I get a dialog box telling me the system is still writing data to the drive. Eventually - more then 6 hours after starting the process - I had to yank the drive out.

I think a block size of a few kilobytes will be much more useful.