Comment 7 for bug 401094

Michael Terry wrote:
>> Depending on how fast the network is, duplicity will be able to overlap
>> its marshaling with the previous file being uploaded. We don't build up
>> a queue, so if it takes longer to marshal the files for the next volume,
>> say a bunch of small files, then you have dead time in the network.
>
> Right, but he said it maxes out at 300. Asynchronous shouldn't have
> anything to do with max speed, but rather throughput. emilio, can you
> clarify if the problem is max speed or throughput?

If this is a throughput issue, then 300 may be the max we can get
without async. If this is a burst issue (need to know how measured),
then much is dependent on the protocol used. Once its in the backend,
its pretty much out of duplicity's control.

>> As to CPU, IO does not use much, only the marshaling process and
>> encryption, and 50% sounds about right in bursts.
>
> emilio has a dual-core, so 50% is really 100% of one core. I meant that
> since the marshaling isn't threaded, even if asynchronous were turned
> on, we'd still only use 100% of one core. (Though we'd be using it more
> often -- i.e. even when uploading something. Another max vs. throughput
> issue)

Sorry, I'm used to looking at 'top' which shows each CPU. Kind of
surprised that some of it is not offloaded to the other CPU since there
are multiple sub-processes running. Wonder what's up with that.

> So again, unless emilo can clarify about the max/throughput question, I
> don't think asynchronous is the answer here. We still want both network
> and CPU maxes to be higher, even though asynchronous would help with
> overall throughput.

Asynchronous will help with throughput, but if done with a queue, the
result will still be the max that the backend can do. If its done with
parallel connections, we can get a big gain at the cost of more space
needed.