Add new core API/entry-point sans import functionality

[Jason Gerard DeRose]

The current dmedia DBus service manages import operations and implements the Pro File Import UX [1]. Although starting with file imports was very helpful for developing the schema because it forced me to deal with large batches of unruly, real-world files (like our 500 gigs of UDS-N video, audio, and photos), importing doesn't belong in our bare-bones, common service.

Lots of apps will never use the import functionality (say media consumption apps like the Novacut player), so we want the import service to be an optional component... which will help keep the dependencies low for a minimal dmedia install. We also don't want to needlessly burn up memory all the time... the importer is something better to start when needed, kill when not. So as already discussed in lp:724707 "Split into simpler, more MAC friendly services" [2], the current DBus service will be renamed to `dmedia-import-service` and we'll add a new `dmedia-service`.

(One more note: the import functionality is worthless on a server/VM you're never physically near, so this separation also makes sense for clusters running dmedia and Novacut servers.)

This new `dmedia-service` will be the DBus bridge to the core API this bug here is concerned with. The core DBus API is discussed some in lp:714041 "Add file path resolution, file download methods to D-Bus AP" [3].

However, we want this bug's core API/entry-point to be completely independent of DBus and any mainloop (so no GObject). This core API/entry-point needs to be flexible and agnostic enough so that it's suitable for DBus with GTK mainloop, DBus with Qt mainloop, and headless servers with no DBus at all. I think the approach I took with the ImportManager and ImportWorker proved worthwhile, so I'm going to reuse and continue to enhance the existing `dmedia.workers` stuff.

So what will this core API/entry-point do?

First of all, the core entry-point will handle the basic dmedia startup:

 * Figure out CouchDB environment (desktopcouch, or system wide? oauth?)
 * Create the 'dmedia/machine' record if needed
 * Initialize all the local FileStore
 * Initialize the CouchDB views if needed
 * Initialize the WebUI app if needed

Second, there will be the core API functions, like:

 * Add and remove FileStore
 * Resolve file _id into full filesystem path
 * Enqueue and dequeue files for download/upload

And third, there will be a number of background tasks the core entry-point will start and manage:

 * Launch the dmedia HTTP server for localnet P2P transfer, advertise with Avahi
 * TransferManager will dispatch queued downloads/uploads to TransferWorker(s)
 * Reclamation - as space is needed, will delete the stalest files first
 * Watch for {'origin: 'user'} files with low durability, upload them to peers/services

Lastly, what will the core API/entry-point *not* do? It will do no importing, no transcoding... importing will be a separate service, transcoding will be one-off processes that communicate directly with CouchDB.

[1] https://wiki.ubuntu.com/AyatanaDmediaLovefest
[2] https://bugs.launchpad.net/dmedia/+bug/724707
[3] https://bugs.launchpad.net/dmedia/+bug/714041

[Jason Gerard DeRose (jderose)]

Little update on this.

This turned into a wild refactor. It was difficult to complete this feature without also completing:

  Gut `metastore`, rename to `views` - https://bugs.launchpad.net/dmedia/+bug/753180

and

  Split into simpler, more MAC friendly services - https://bugs.launchpad.net/dmedia/+bug/724707

It's possible, but would take a lot more time, take writing quite a bit of temporary band-aid code in order to land the changes separately. As this needs to be an extra high velocity month, I'm just tearing through all 3 bugs in one fail swoop.

I apologize to those that are following along with the dmedia development that this wont be an easy merge to review. On the upside, this is exactly why it's great to have a comprehensive test suite... you can confidently do aggressive refactoring. Plus, this is bringing about some significant improvements in dmedia, and adding the missing pieces needed for other applications to integrate with dmedia.

[Benoit Chesneau (bchesneau)]

Did you had a look in teh way couchdbkit does for views ? It is
getting them from the fs like couchapp.

- benoît

On Mon, Apr 11, 2011 at 12:36 AM, Jason Gerard DeRose
<email address hidden> wrote:
> Little update on this.
>
> This turned into a wild refactor.  It was difficult to complete this
> feature without also completing:
>
>  Gut `metastore`, rename to `views` -
> https://bugs.launchpad.net/dmedia/+bug/753180
>
> and
>
>  Split into simpler, more MAC friendly services -
> https://bugs.launchpad.net/dmedia/+bug/724707
>
> It's possible, but would take a lot more time, take writing quite a bit
> of temporary band-aid code in order to land the changes separately.  As
> this needs to be an extra high velocity month, I'm just tearing through
> all 3 bugs in one fail swoop.
>
> I apologize to those that are following along with the dmedia
> development that this wont be an easy merge to review.  On the upside,
> this is exactly why it's great to have a comprehensive test suite... you
> can confidently do aggressive refactoring. Plus, this is bringing about
> some significant improvements in dmedia, and adding the missing pieces
> needed for other applications to integrate with dmedia.
>
> --
> You received this bug notification because you are a member of Novacut
> Community, which is subscribed to DMedia.
> https://bugs.launchpad.net/bugs/753260
>
> Title:
>  Add new core API/entry-point sans import functionality
>

[Jason Gerard DeRose (jderose)]

Ooo, a CouchDB celebrity has graced this bug report :)

No, haven't looked at couchdbkit yet. I believe desktopcouch has a similar feature (load design docs from fs), but I avoided using it as dmedia needs to run both on the desktop and on headless servers.

Would couchdbkit be appropriate for servers also?

Thanks for the tip!

[Jason Gerard DeRose (jderose)]

Hehe, ended up fixing https://bugs.launchpad.net/dmedia/+bug/710405 in this to. Four for the price of one.

[Jason Gerard DeRose (jderose)]

3658 line diff, not as bad as I thought it would be:

https://code.launchpad.net/~jderose/dmedia/core-api/+merge/57148