IPython.kernel.client cannot be imported inside an engine

[Dejan Pecevski]

Importing IPython.kernel.client in an engine doesn't work.
The twisted reactor is started for the controller and engine
automatically when they start and for the client when
IPython.kernel.client is imported. The problem is that when executing import
IPython.kernel.client script in the engine, the engine already has the reactor running,
and the import tries to start the reactor again. This causes
a the ipengine<->ipcontroller communication to fail on
subsequently executed commands.

A typical use case when one needs to import IPython.kernel.client in the
engines, is when one wants to use one of the engines as a client connected
to another ipcontroller, and it seems impossible to do it,
if one cannot import the client in one of the engines.

[Brian Granger (ellisonbg)]

Fixing this will require that the core be thread safe and that the reactor runs in a different thread from user code. This should happen this summer.

[Vishal Vatsa (vvatsa)]

Just wondering, if the problem is that there is already another reactor
running so can't we just use the asyncclient.

I ran this StringTask:
"from IPython.kernel import asyncclient ; doc_str = asyncclient.__docformat__"

I am sure using asyncclient has its own set of complications.

-vishal

2009/4/14 Brian Granger <email address hidden>:
> Fixing this will require that the core be thread safe and that the
> reactor runs in a different thread from user code.  This should happen
> this summer.
>
> ** Changed in: ipython
>   Importance: Undecided => Medium
>
> ** Changed in: ipython
>       Status: New => Confirmed
>
> ** Changed in: ipython
>     Assignee: (unassigned) => Brian Granger (ellisonbg)
>
> --
> IPython.kernel.client cannot be imported inside an engine
> https://bugs.launchpad.net/bugs/300218
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: Confirmed
>
> Bug description:
> Importing IPython.kernel.client in an engine doesn't work.
> The twisted reactor is started for the controller and engine
> automatically when they start and for the client when
> IPython.kernel.client is imported. The problem is that when executing import
> IPython.kernel.client script in the engine, the engine already has the reactor running,
> and the import tries to start the reactor again. This causes
> a the ipengine<->ipcontroller communication to fail on
> subsequently executed commands.
>
> A typical use case when one needs to import IPython.kernel.client in the
> engines, is when one wants to use one of the engines as a  client connected
> to another ipcontroller, and it seems impossible to do it,
> if one cannot import the client in one of the engines.
>

[Brian Granger (ellisonbg)]

There are a couple of issue:

* The reactor is already running and kernel.client tries to start it.

* The sync code in kernel.client has to be run in a different thread
from the main reactor. When you use kernel.client in IPython, it
actually starts the twisted reactor in a thread. The make the client
work in the engine, we need to run the users code (where the client
will be used) in a different thread from the reactor.

* To do this, parts of the engine have to be made threadsafe.

I haven't though about using the asyncclient though in the engine. It
may work, but as you guess, there may be complications.

Cheers,

Brian

On Fri, Apr 24, 2009 at 11:35 AM, Vishal Vatsa <email address hidden> wrote:
> Just wondering, if the problem is that there is already another reactor
> running so can't we just use the asyncclient.
>
> I ran this StringTask:
> "from IPython.kernel import asyncclient ; doc_str = asyncclient.__docformat__"
>
> I am sure using asyncclient has its own set of complications.
>
> -vishal
>
> 2009/4/14 Brian Granger <email address hidden>:
>> Fixing this will require that the core be thread safe and that the
>> reactor runs in a different thread from user code.  This should happen
>> this summer.
>>
>> ** Changed in: ipython
>>   Importance: Undecided => Medium
>>
>> ** Changed in: ipython
>>       Status: New => Confirmed
>>
>> ** Changed in: ipython
>>     Assignee: (unassigned) => Brian Granger (ellisonbg)
>>
>> --
>> IPython.kernel.client cannot be imported inside an engine
>> https://bugs.launchpad.net/bugs/300218
>> You received this bug notification because you are a member of IPython
>> Developers, which is subscribed to IPython.
>>
>> Status in IPython - Enhanced Interactive Python: Confirmed
>>
>> Bug description:
>> Importing IPython.kernel.client in an engine doesn't work.
>> The twisted reactor is started for the controller and engine
>> automatically when they start and for the client when
>> IPython.kernel.client is imported. The problem is that when executing import
>> IPython.kernel.client script in the engine, the engine already has the reactor running,
>> and the import tries to start the reactor again. This causes
>> a the ipengine<->ipcontroller communication to fail on
>> subsequently executed commands.
>>
>> A typical use case when one needs to import IPython.kernel.client in the
>> engines, is when one wants to use one of the engines as a  client connected
>> to another ipcontroller, and it seems impossible to do it,
>> if one cannot import the client in one of the engines.
>>
>
> --
> IPython.kernel.client cannot be imported inside an engine
> https://bugs.launchpad.net/bugs/300218
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: Confirmed
>
> Bug description:
> Importing IPython.kernel.client in an engine doesn't work.
> The twisted reactor is started for the controller and engine
> automatically when they start and for the client when
> IPython.kernel.client is imported. The problem is that when executing import
> IPython.kernel.client script in the engine, the engine already h...