BitTorrent support for smart

[Rudd-O]

[04:33] <Rudd-O> the point of smart where I see bt useful
[04:33] <Rudd-O> is in update downloading
[04:34] <Jonathan_R> give me an example
[04:34] <Rudd-O> so when an update is released, sure it's downloaded from the official mirror, but then seeded using some sort of torrent
[04:34] <Rudd-O> ok so you have a mirror
[04:34] <Rudd-O> and the mirror usually generates the metadata and stuff
[04:35] <Rudd-O> it could conceivably generate a trackerless torrent that is seeded over the web, and smart would download only selected files from that torrent (or, in case of failure, well, it's web-seeded so...)
[04:35] <Rudd-O> that way a channel ("repo" in yum parlance) can be bittorrent-enabled and reduce the bandwidth across the web
[04:35] <Rudd-O> fewer mirrors would be needed

The way I see this could be implemented is:

Enable the deb repo generator and the yum createrepo to create a Web-seeded, trackerless torrent. That is step 1. So when the repos are regenerated in the mirrors, they are auto-creating the new .torrent file that will be placed in the appropriate directory of the repo.

Step 2 would be to make smart interface with the high-quality python libraries for bittorrent, so it can download only select files (the ones that it needs to perform the system update) from the Web-seeded trackerless torrent.

Step 3 would be to be a good citizen. Seed until a predefined percentage or some time has passed, then quit and remove the files from disk.

This idea would, of course, need to be an opt-in option and naturally also would need the user to specify a maximum upload bandwidth (or perhaps auto-envelope the upload bandwidth based on the latency to the download server, if pings are too high, cut the upload bandwidth in half, until the bandwidth sits at a level where it doesn't impact network performance, or maybe in the other direction, start at 1 KB and multiply it by 2 until network performance decreases, then slash by 2 and remember the value.).

And, of course, this would not need any modification to smart's repo code, except for detecting whether a .torrent exists on the repo directory (and if that's the case, enable bittorrent).

If this is implemented correctly, this would lead to major bandwidth savings on Internet links in companies, and blindingly fast updates across machines in the local network, because LAN speeds are much, much faster than Internet speeds.

[Anders F Björklund (afb)]

Wouldn't you need one such .torrent file for each .rpm or .deb ?
I mean it's useful for .iso files, but not so very for smaller ones.

But yeah, there are some variants on this BT theme floating around.
Like debtorrent/apt-torrent, or http://brainstorm.ubuntu.com/idea/7792/

[Anders F Björklund (afb)]

http://debtorrent.alioth.debian.org/ has some Python code for this

[Rudd-O (rudd-o)]

Not really, you would not need one .torrent per file. You would only need one .torrent per repository, because a torrent can keep an entire directory of files, and clients can "check out" only certain files from the entire torrent. So, as long as the .torrent that the client has cached locally is newer than the latest repo manifest in the server, everything is good and the .torrent is valid.

[Rehan Khan (rasker)]

Therein lies the rub. If you have a swarm where people are only getting a few files from the torrent the number of seeds would be quite low and the number of peers would be relatively high. This may result in relatively low performance of the torrent. Torrent's are more suited to large static un-changing blobs of data (video, iso's etc) for this reason.

Also for the very small files the ratio of setup traffic/data transferred would be very high and thus it would be in-efficient.

An alternative would be to use an edonkey/emule p2p protocol which performs better for small files. Still it has it's own issues.

Another method would be to use torrents to synchronise mirrors which is a more practical use.

R

[Rudd-O (rudd-o)]

El Miércoles 21 Enero 2009, Rehan Khan escribió:
> Therein lies the rub. If you have a swarm where people are only getting
> a few files from the torrent the number of seeds would be quite low and
> the number of peers would be relatively high. This may result in
> relatively low performance of the torrent. Torrent's are more suited to
> large static un-changing blobs of data (video, iso's etc) for this
> reason.

I have torrented large collections of files selectively with no performance
impact. And since the torrent is Web-seeded, there is no negative performance
impact -- only positive. Furthermore, the latest batch of critical must-ahve
updates (or rather, the popular cluster of torrent chunks containing them)
will ALWAYS have seeds because everybody requests those. And what you care
about is optimizing most-popular use case, not *everything*.

>
> Also for the very small files the ratio of setup traffic/data
> transferred would be very high and thus it would be in-efficient.

Why? Or, put it in another way, what is impeding this implementation to put
only large (say above 10MB) files inside the torrent file list, and use the
traditional mechanism for small files?

> An alternative would be to use an edonkey/emule p2p protocol which
> performs better for small files. Still it has it's own issues.

True. True. I just referred to BT because it's stable and has many python
implementations.

>
> Another method would be to use torrents to synchronise mirrors which is
> a more practical use.

That's also conceivable but it still means people have to download from the
*mirrors* instead of from nearby computers (an optimization that becomes
possible if using BT).

--

 Manuel Amador (Rudd-O) <email address hidden>
 Rudd-O.com - http://rudd-o.com/
 GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

The trouble with the rat-race is that even if you win, you're still a rat.
  -- Lily Tomlin