Add TransferManager, TransferWorker(s)

[Jason Gerard DeRose]

In 0.4 we added most of the infrastructure for moving files around between dmedia peers and services (tracking where files are stored, and providing download backends). Now we need to add (and remove) downloads from a queue, and have a download manager dispatch individual downloads from that queue.

So the first design issue is: what does it mean to queue a download, where is this information stored? I'm strongly leaning toward queuing downloads for a given device in that device's "dmedia/machine" record, something like this:

{
  "_id": "D3VTOIKRTUJBWRTPCZ57TWER",
  "type": "dmedia/machine",
  "queue": [
    "4YRXSQX6PBAJTF5A7QALZFQZC75MY3J6",
    "KJXQX3ZTXS7R6TATM3RNTSO4F76VA7H5"
  ]
}

The reason for keeping the queue in CouchDB is there are really cool use cases where you might want to queue a download from a remote device - like while at work you queue a download on your TV from your phone so it's ready when you get home. Well, actually you just modify the appropriate "dmedia/machine" record in the CouchDB running locally on your phone, and then at the next sync your TV would see the change and start downloading. FYI, we think dmedia should be on phones and TVs (of course).

As mentioned in lp:714041, we'll add a convenience D-Bus API to make it easy for local applications to queue downloads. But the download manager will just be listening for changes on the CouchDB changes feed, so it doesn't care how or where the download was queued. Although we may want to prioritize the downloads differently depending on whether a local applications needs a given file *now*.

When running with D-Bus integration, we probably want to expose information about queued and active downloads over D-Bus so individual applications can display, say, a progress bar for a specific download that the application is concerned with.

As the needed pieces become clearer, I may split this bug into smaller pieces.

[Jason Gerard DeRose (jderose)]

Okay, gonna give this a go. First of all, in terms of the semantics the TransferManager deals with, there are two jobs that can be performed:

  download(file_id, remotestore_id)

  upload(file_id, remotestore_id)

In both cases, `file_id` is the _id of the "dmedia/file" doc, and `remotestore_id` is the _id of the "dmedia/store" doc describing the store or service the file will be downloaded from, or uploaded to.

When an upload is started, a decision has to be made about which FileStore to source the file from (in case a file is on multiple local FileStore). When a download is started, a decision has to be made about which FileStore to save the file into. Like with imports, we eventually want to be able to save a file to multiple FileStore simultaneously, but for now we'll just deal with saving to one FileStore.

I'm gonna get uploading to/downloading from Amazon S3 working first as that would be really handy for us while at UDS-O. But there's a lot of smarts that will need to be built into the TransferManager over time. For example, when running on a desktop/mobile device, we want to know when an Internet connection is actually available, which I believe you can get from some DBus signal or another.