No support for zsync at Ubuntu ISOs

[psl]

Binary package hint: zsync

This is not a real bug but I don't know where to enter this improvement.

ISO images of Ubuntu are distributed in many ways. RAW ISO, Jidgo support, etc. I miss support for zsync.

zsync is similar to rsync but is designed for client download, it doesn't create no extra load on the server, and no rsync is required to run on the server. zsync uses standard HTTP protocol and only needs a special file with metadata at web server, file that has pre-calculated checksums for rsync algorithm.

This ticket is about request to add files with zsync extension for every Ubuntu ISO you have. It will enable download of Ubuntu ISO images from web servers in efficient way with zsync client when user has older ISO image.

Ubuntu current generates support files for torrent and jidgo as I know.

Example:
ubuntu-8.04.1-dvd-amd64.iso
ubuntu-8.04.1-dvd-amd64.iso.torrent
ubuntu-8.04.1-dvd-amd64.iso.jidgo

Ubuntu should generate zsync files too, like:
ubuntu-8.04.1-dvd-amd64.iso.zsync

Information about pre-calculated checksums for the rsync algorithm stored in "zsync file", this file is downloaded by zsync client first and is used to download iso file in efficient way, like rsync does. File with zsync meta data is creted with "zsyncmake" utility.

zsync technical and design details:
http://zsync.moria.org.uk/

[psl (slansky)]

This is way I think that zsync files can be created:

for FILE in *.iso; do
   echo "Processing "$FILE"
   zsyncmake -v -u "$FILE" -f "$FILE" -o "${FILE}.zsync" "$FILE"
done

[psl (slansky)]

Sample files to show that zsync files are "small", about 1.5MB for CD and 6.5MB for DVD:

-rw-r--r-- 1 root root 3371372544 2006-08-08 01:03 dapper-dvd-i386.iso
-rw-r--r-- 1 root root 6584882 2008-11-23 15:41 dapper-dvd-i386.iso.zsync
-rw-r--r-- 1 root root 731744256 2006-06-01 18:16 ubuntu-6.06-desktop-i386.iso
-rw-r--r-- 1 root root 1429379 2008-11-23 15:44 ubuntu-6.06-desktop-i386.iso.zsync

[psl (slansky)]

I would like to highlight one interesting point about ZSYNC. It can support jidgo downloads. I use jidgo often and it happens time from time that you cannot get several packages to successfully finish ISO image. Recommended way in this case is to use rsync to get missing pieces of a file. But it is a painful and time consuming process to find working RSYNC server and to locate images on the server. ZSYNC can offer the same service like RSYNC but it is easier to use.

The same apply to torrent downloads. It can happen that you cannot finish your download because there is no peer offering your missing part of a file. You have to wait or you can use RSYNC to finish download. I think most people will just wait or download the whole image with WGET or similar utility. ZSYNC can offer valuable help here. I am not sure if it is problem to download Ubuntu images with torrent client but it is general observation from downloading other files with torrent client. One missing part of a file can degrade all your effort and you have to wait and wait and sometimes just give up and delete unfinished file.

[psl (slansky)]

Ubuntu 8.04.2 is out. It was great opportunity to test zsync in real life. To test upgrade of Ubuntu iso images from 8.04.1 to 8.04.2 with zsync. Opportunity that was missed :-(

[psl (slansky)]

I prepared a ZSYNC file to test download of Ubuntu 8.04.2 images.

ubuntu-8.04.2-server-i386.iso.zsync was created in this way:

FILE="ubuntu-8.04.2-server-i386.iso"; zsyncmake -v -u "$FILE" -u "http://releases.ubuntu.com/8.04.2/$FILE" -f "$FILE" -o "$FILE.zsync" "$FILE"

To test zsync update, copy old ISO image to some temporary directory at your disk. I use ubuntu-8.04.1-server-i386.iso in my example to build ubuntu-8.04.2-server-i386.iso. Download ubuntu-8.04.2-server-i386.iso.zsync to the same directory and run this command:

zsync -i ./ubuntu-8.04.1-server-i386.iso ./ubuntu-8.04.2-server-i386.iso.zsync

With support from zsync, you have to download only 50% of the ISO file.

In the case you don't have old ISO image,you can try to download while ISO image with this command:

zsync ./ubuntu-8.04.2-server-i386.iso.zsync

In this case, zsync doesn't add real value compared with wget and other similar tools.

In the case that official Ubuntu server will support ZSYNC, you can use this (only sci-fi now):

zsync -i ./ubuntu-8.04.1-server-i386.iso http://releases.ubuntu.com/8.04.2/ubuntu-8.04.2-server-i386.iso.zsync

And you can use "-u" option to overwrite URL embedded in the zsync file with server you prefer:

zsync -i ./ubuntu-8.04.1-server-i386.iso -u http://ftp.sh.cvut.cz/MIRRORS/ubuntu-releases/releases/releases/8.04.2/ubuntu-8.04.2-server-i386.iso ./ubuntu-8.04.2-server-i386.iso.zsync

[psl (slansky)]

More ZSYNC files for Ubuntu 8.04.1 and 8.04.2 can be downloaded from this location:

http://psl.ic.cz/zsync/

To upgrade ISO image ubuntu-8.04.1-server-i386.iso to ubuntu-8.04.2-server-i386.iso, you can run this command:

zsync -i ./ubuntu-8.04.1-server-i386.iso http://psl.ic.cz/zsync/ubuntu-8.04.2-server-i386.iso.zsync

[psl (slansky)]

Link to the article at brainstorm. Article is related to to topic "ZSYNC support for Ubuntu ISO images":

http://brainstorm.ubuntu.com/idea/7590/

[psl (slansky)]

Another example. Use "server" image ubuntu-8.04.2-server-i386.iso to create "jeos" image. jeos-8.04.2-jeos-i386.iso file has 105644032 bytes, only 13738663 were downloaded from HTTP server, it is only 13% of the original iso size because these iso images "share" several files.

Details:

$ zsync -i ubuntu-8.04.2-server-i386.iso http://psl.ic.cz/zsync/jeos-8.04.2-jeos-i386.iso.zsync
#################### 100.0% 0.0 kBps DONE

reading seed file ubuntu-8.04.2-server-i386.iso: ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************Read ubuntu-8.04.2-server-i386.iso. Target 87.0% complete.
downloading from http://psl.ic.cz/zsync/jeos-8.04.2-jeos-i386.iso:
#################--- 87.0%bad status code 404
#################--- 87.0% 0.0 kBps aborted

failed to retrieve from jeos-8.04.2-jeos-i386.iso
downloading from http://cdimage.ubuntu.com/jeos/releases/8.04.2/release/jeos-8.04.2-jeos-i386.iso:
#################### 100.0% 482.0 kBps DONE

verifying download...checksum matches OK
used 91918336 local, fetched 13738663

$ ls -ltr jeos-*
-rw-r--r-- 1 psl psl 105644032 2009-01-27 09:48 jeos-8.04.2-jeos-i386.iso
-rw-r--r-- 1 psl psl 180811 2009-04-11 15:47 jeos-8.04.2-jeos-i386.iso.zsync

[Colin Watson (cjwatson)]

Sorry we didn't notice this bug until now; the ubuntu-cdimage product is actually the right place to reach the cdimage.ubuntu.com maintainers, not ubuntu-website. Lars filed this separately as bug 408501 and I added this support last week.