Using multiprocessing module crashes parallel iPython
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
IPython |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
I have a parallel scientific code that runs on clusters. It currently uses my own (badly designed) parallel communication engine and I'm trying to transition to iPython's TaskClient interface. One part of my code uses the subprocess module to wrap a call to a different piece of software. Unfortunately, this seems to be causing problems for me. I'm running on OS X 10.6.2, with python 2.6.4 and ipython 0.10 as supplied by MacPorts. The following code snippet will reproduce the problem I'm having.
-----
#!/usr/bin/env python
from IPython.kernel import client
tc = client.TaskClient()
@tc.parallel()
def remote_test(input):
import subprocess
# I'm obviously not wrapping ls, but I have the same problem with the real binary I'm trying to call
subprocess.
return input
work = range(100)
results = remote_test(work)
-----
The output of this program is:
-----
/opt/local/
import sha
/opt/local/
self.tub = Tub()
/opt/local/
class FCTaskControlle
Traceback (most recent call last):
File "./test.py", line 15, in <module>
results = remote_test(work)
File "/opt/local/
return self.mapper.
File "/opt/local/
task_results = [self.task_
File "/opt/local/
taskid, block)
File "/opt/local/
return twisted.
File "/opt/local/
result.
File "/opt/local/
raise self.type, self.value, self.tb
OSError: [Errno 4] Interrupted system call
-----
The ipcontroller log file looks like:
-----
2010-02-04 12:42:36-0800 [-] Running task 82 on worker 1
2010-02-04 12:42:36-0800 [Negotiation,
2010-02-04 12:42:36-0800 [Negotiation,
2010-02-04 12:42:36-0800 [Negotiation,
2010-02-04 12:42:36-0800 [Negotiation,
2010-02-04 12:42:36-0800 [-] distributing Tasks
2010-02-04 12:42:36-0800 [-] Running task 84 on worker 0
2010-02-04 12:42:36-0800 [Negotiation,
2010-02-04 12:42:37-0800 [-] distributing Tasks
2010-02-04 12:42:37-0800 [-] Running task 85 on worker 1
2010-02-04 12:42:37-0800 [Negotiation,
2010-02-04 12:42:37-0800 [-] distributing Tasks
2010-02-04 12:42:37-0800 [-] Running task 86 on worker 0
2010-02-04 12:42:37-0800 [Negotiation,
2010-02-04 12:42:38-0800 [-] distributing Tasks
2010-02-04 12:42:38-0800 [-] Running task 87 on worker 1
-----
Notice that a few of the tasks actually complete, while the majority fail with the strange interrupted system call error.
Changed in ipython: | |
status: | New → Confirmed |
After some research, I believe that this is an issue that arises because of Twisted, which doesn't play nice with mulitprocessing or popen.