dynamically change segment size based on download speed

[Pseudonym]

The default segment size of 1 MiB doesn't work very well on high-bandwidth connections, and most users are wary of tweaking the segment size in experts only. Therefore, it would be nice if DC++ would automatically request larger segments if it can gain from it.

Preliminary algorithm:
DC++ should keep track of how large the previous segment was and how long it took to download. Segment size should start at what the default is now. If the transfer took less than one minute, double the segment size. If it took longer than 5, drop back to the default size.

[Steven Sheehy (steven-sheehy)]

"If the transfer took less than one minute, double the segment size. If it took longer than 5, drop back to the default size."

Where did you come up with these numbers? That seems like a bit of a hack. Instead, I recommend we use proven algorithms like slow start and/or AIMD (or variations thereof) that are used by the TCP protocol. Basically, what I'm suggesting is we make this dynamic segment size more dynamic and flexible than what has been suggested.

[Big Muscle (bigmuscle)]

StrongDC++ uses following:

 if(userSpeed > 0) {
  // get the speed of average chunk
  int64_t averageChunkSpeed = downloads.size() > 0 ? getAverageSpeed() / downloads.size() : DEFAULT_SPEED;
  if(averageChunkSpeed == 0) averageChunkSpeed = DEFAULT_SPEED;

  // set maxSize according to user's lastSpeed
  double x = max(1.0, (double)userSpeed / (double)averageChunkSpeed);
  maxSize = (int64_t)((double)maxSize * x);

  // chunk is still too small for this user
  if(maxSize / userSpeed <= 5) {
   maxSize *= 2;
  }
 }

where:
userSpeed is last user's speed saved in QueueManager::putDownload
DEFAULT_SPEED is minimum speed to start increasing segment size when it is the first segment or current file speed is 0 B/s (StrongDC++ has this value set to 25600 = 25 kB/s)

To ensure this method would be optimal, DC++ should change download speed measuring. Because segments finished below 1 second have download speed 0 B/s and then the code is not usable for it.

[Jacek Sieka (arnetheduck)]

Steven, how is slow start applicable? We're not requesting multiple segments at a time waiting for a
response so there's no way to count how many responses we got...

Was it the growth bit you were after, starting with a small segment and growing exponentially until
some condition is met and then grow linearly a bit more? Then we need a good condition, probably
some target time that you want a segment to take, trading off round-trip loss of efficiency for good
behavior for the last segments...it could be a function of the file size, or constant, possibly
user-settable. '

We'd need some bounds as well, TTH leaf size is a natural lower bound. For upper bound, maybe
time-left-for-file-at-current-speed / 2 (using speed of last block to calculate the actual size
bound), to avoid waiting for a slow source at the end....

[Big Muscle (bigmuscle)]

To avoid waiting for a slow source at the end I invented feature called "Slow chunks overlapping".

When QueueItem::getNextSegment would return Segment(0,0) (which means "no other segment to download"), it tries to find the slowest running segment and returns it as it would be needed segment. There are some conditions for it as that new user must finish that segment at least 2x faster (so you must store user's last download speed) etc... It requests this chunk from that user and we got a slot, it checks whether original (slow) segment is still running. If yes => disconnect it, if no => disconnect "fast" user.

You can check following links to see how it's exactly implemented:

line 204 : http://strongdc.svn.sourceforge.net/viewvc/strongdc/trunk/client/QueueItem.cpp?view=markup
line 931: http://strongdc.svn.sourceforge.net/viewvc/strongdc/trunk/client/QueueManager.cpp?view=markup

[Jacek Sieka (arnetheduck)]

I'll have a look, but I'd prefer doing something smart to avoid throwing away those bytes...anyway, a first version of this is committed now...

[Big Muscle (bigmuscle)]

It throws away only bytes needed to align overlapping segment to TTH block size (same when segment is disconnected manually)

[Jacek Sieka (arnetheduck)]

ok, implemented with 2-minute target segment time which falls off towards the end of the file...if it creates problems I'll consider overlapping as well...