D-Bus guru? Add D-Bus skeleton to dmedia

[Jason Gerard DeRose]

The backend components of dmedia will run as an on-demand session daemon. Seems the best way to do this is through D-Bus. I'm not very experienced with D-Bus or its best practices, so if someone could stub this out, it would be greatly appreciated. This should be implemented in Python.

The first thing needed in the dbus API is a function to recursively import media files from a location (e.g., a Compact Flash memory card inserted in a card reader). The Python function would look like this:

def recursive_import(directory, extensions):
    pass

Where *directory* is the directory to start the import from (e.g., '/media/EOS_DIGITAL'), and *extensions* is a list of file extensions to consider (e.g., ['mov', 'jpg', 'cr2', 'wav']).

If the dmedia daemon isn't already running, it should start and perform the import. Applications will get progress information over D-Bus.

There are 3 types of long-running background processes the dmedia daemon will handle:

 * Importing files (importing ~ 32GBs or more will be typical, full-content-hash done on all new files)
 * Uploading and downloading files from dmedia peers, S3, UbuntuOne, etc.
 * Rendering proxy files (lower-resolution versions for responsive editing and fast Internet upload/download)

There is also one (presumably) short-running background process: bi-directionally synced the dmedia CouchDB database with UbuntuOne, dmedia peers, etc. But this will be handled by desktopcouch rather than the dmedia daemon.

I think it would be great to use an Application Indicator to see what long-running dmedia jobs are running, control them. And it would be great to use NotifyOSD to alert the user when a batch job completes (import completes, all new files synced to cloud, etc).

If this could be stubbed out according to D-Bus/Python best practices, I can do the work integrating it with dmedia. FYI, dmedia will go through some heavy refactoring in the next month, so it's best if you leave the stub disconnected so to speak... just add a new module (like dmedialib/bus.py) for the skeleton, but don't have it use the unstable dmedia API just yet. As always, must have tests and reasonable docstrings.

If you need more information, please comment on this bug or ping jderose on #novacut.

I'm targeting this for dmedia 0.2 (December 30th), but I would be happy to land at least the stub in 0.1 (November 25) if someone completed this in time.

[Jason Gerard DeRose (jderose)]

Okay, I'm getting started on this. I'm planing on using the `multiprocessing` module for dmedia as we really need to take advantage of multi-core well, make sure user interfaces are very responsive, stay nice and async. It's going to be simple, share-nothing with message passing over queues. When a dbus method is called, any "heavy work" needed will be put in the InQueue and handled by a worker processes. When a job completes, worker process puts result in OutQueue, which main process can emit as a dbus signal, whatever. Or something roughly like this.

Anyone have experience with a similar setup with dbus and multiprocessing? Any advice, gotchas to avoid? Is this basically a reasonable approach?

I'm not very dbus savvy, so if anyone has guidance on this, I'm all ears. In the dbus documentation, it has an example of exporting an object for each open file in a word processor:

"""
The object path. Applications can export many objects - for instance, example.com's word processor might provide an object representing the word processor application itself and an object for each document window opened, or it might also provide an object for each paragraph within a document.
"""

In the future we want dmedia to be able to do parallel imports, so from the above, it would seam reasonable to export an object for each currently running import operation... that way applications can get the status of a specific import, cancel or pause a specific import.

From the get-go I want dmedia (and Novacut) to integrate property with application indicators, NotifyOSD, and dbusmenu, so guidance there would also be deeply appreciated.

[Jason Gerard DeRose (jderose)]

This is now merged into trunk. The dmedia service provides a fairly complete API for controlling import operations (including have multiple import operations happening in parallel). There is more work do to, but this was a good stopping point and the test coverage is quite comprehensive.

There is a client.Client class that provides a more convenient API for Python applications than direct dbus does. Its docstring explains the dbus API well. See:

http://bazaar.launchpad.net/~dmedia/dmedia/trunk/annotate/head%3A/dmedialib/client.py?start_revid=117

Next will probably be to make David Green's import GUI stub do stuff using Client, so adventurous testers can import through a means other than the command line. It will be just a quick throw-away GUI to be replaced by the properly designed one later... but still a good way to make sure the plumbing is getting appropriately tested in the meantime.

And then the dmedia script needs to be ported to Client also, use the dbus service.