Add support for running the daemon the "Twisted Way"

[Duncan McGreggor]

Currently, the daemon is run via a shell script that performs some basic actions and then executes some Twisted (Python) code. The reactor.run() method is called directoy, instead of using either a Twisted plugin or, alternatively, a Twisted application.

We need to (at the very least) set up the daemon code so that a Twisted application is instantiated and then run as part of a Twisted service. This will provide greater stability, improved logging support, easier command-line control, and graceful exiting.

[Duncan McGreggor (oubiwann)]

The deamon code is very monolithic right now, and won't lend itself to this very easily, as things are. I have some plans about how to accomplish this, and will create a blueprint for it, once that's turned on for the project.

[Jono Bacon (jonobacon)]

Thanks for fleshing out a blueprint on this Duncan; this is definitely something that we will want to have a solid plan for first. I would like if we can scope this work out to specifically getting the existing daemon API running as a twistd service although in future a complete evaluation of the current daemon API would be useful.

When the blueprint is set up if you could associate it with this bug that would be great. Thanks!

[Duncan McGreggor (oubiwann)]

Hrm, so... I've started working on this in a branch for utilizing twisted.application (for use with twistd), but... there's so much that's not actually done in such a way that we can use Twisted in the way it's meant to be used. I'm fixing only what is strictly necessary for this ticket, but there are some pretty extensive changes necessary. For the stuff I'm not changing right now, I'm leaving TODO breadcrumbs throughout the code... and as of now, there are several loaves' worth...

In short, I may have to put this work in a separate branch.

To be continued...

[Jono Bacon (jonobacon)]

On 18 March 2012 21:15, Duncan McGreggor <email address hidden> wrote:
> Hrm, so... I've started working on this in a branch for utilizing
> twisted.application (for use with twistd), but... there's so much that's
> not actually done in such a way that we can use Twisted in the way it's
> meant to be used. I'm fixing only what is strictly necessary  for this
> ticket, but there are some pretty extensive changes necessary. For the
> stuff I'm not changing right now, I'm leaving TODO breadcrumbs
> throughout the code... and as of now, there are several loaves' worth...
>
> In short, I may have to put this work in a separate branch.

Can you elaborate a little more on how the current code could not be
used with twistd?

   Jono

--
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon

[Duncan McGreggor (oubiwann)]

On Mon, Mar 19, 2012 at 12:22 AM, Jono Bacon <email address hidden> wrote:
> On 18 March 2012 21:15, Duncan McGreggor <email address hidden> wrote:
>> Hrm, so... I've started working on this in a branch for utilizing
>> twisted.application (for use with twistd), but... there's so much that's
>> not actually done in such a way that we can use Twisted in the way it's
>> meant to be used. I'm fixing only what is strictly necessary  for this
>> ticket, but there are some pretty extensive changes necessary. For the
>> stuff I'm not changing right now, I'm leaving TODO breadcrumbs
>> throughout the code... and as of now, there are several loaves' worth...
>>
>> In short, I may have to put this work in a separate branch.
>
> Can you elaborate a little more on how the current code could not be
> used with twistd?

Sure.

The way it's currently written, it's very difficult to tell what is
deferred and what is not. In fact, there's actually quite a lot of
blocking code occurring in methods that should be returning deferreds.
Even more, there are methods that *should* be returning deferreds (or
using inlineCallbacks) when they're not. The (non-)functional result
is a daemon that only appears to be async.

One of the benefits of using twistd is not only that you get a daemon
out of the deal, you also get appropriate process management and
signal handling. It's not clear to me that this application will
behave as expected, handle errors appropriately, and take advantage of
the benefits of twistd without first fixing the code.

There may be more problems; I won't be able to tell until I've
finished trying to put this stuff in a twisted app...

That being said, I'm actually *trying* to get it to run with twistd.
For more details, you should check out the branch... there's a LOT of
comments in there now ;-)

[Duncan McGreggor (oubiwann)]

Separate thread here: I think I've got a good idea of how to handle the dbus stuff in a twisted daemon now. But first:

Jono, is there any specific behaviour you're going to want the SessionBus (or DBusGMainLoop) to enact upon startup or shutdown of the service?

If not, that's okay -- we can leave that functionality as a stub.

However, if so, let me know so that ^C (and more importantly, system shutdown) can do the appropriate things for the session bus or the dbus mainloop.

I'm also going to add a TimerService to the Twisted app so that the SCRIPT_DELAY hack isn't necessary anymore.

It looks like there's no other services or client activity going on in the project (yet!) so those two services should be all that's needed.

[Duncan McGreggor (oubiwann)]

Hrm, maybe one more service... one that wraps Twisted's application service as well as the DBUS object exporter service.

We'll see...

[Duncan McGreggor (oubiwann)]

Success! I've got a local twistd version of the daemon running :-)

(Horray for long conference calls...)

[Duncan McGreggor (oubiwann)]

Okay, this puppy is ready for review.