KeyboardInterrupt in pylab mode causes ValueError

[TheFlyingPengwyn]

When started with pylab mode, any KeyboardInterrupts in a running function throw a ValueError as the MAIN_THREAD_ID appears to be incorrectly defined. Can someone reproduce this please?

My version is that packaged with ubuntu 8.04 and is (pasted from the intro ipython text):
Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30)
IPython 0.8.1 -- An enhanced Interactive Python.

Apologies if this has already been debugged - I had a look through the Trac list and couldn't find anything definite.

Here is a sample run (I hit ctrl-c while the asdf() func is running):

In [1]: def asdf():
   ...: while(True):
   ...: a = 1;
   ...:
   ...:

In [2]: asdf()
---------------------------------------------------------------------------
<type 'exceptions.ValueError'> Traceback (most recent call last)

/home/pengwyn/<ipython console> in <module>()

/home/pengwyn/<ipython console> in asdf()

/var/lib/python-support/python2.5/IPython/Shell.py in sigint_handler(signum, stack_frame)
    289
    290 if CODE_RUN:
--> 291 _async_raise(MAIN_THREAD_ID,KeyboardInterrupt)
    292 else:
    293 KBINT = True

/var/lib/python-support/python2.5/IPython/Shell.py in _async_raise(tid, exctype)
    270 ctypes.py_object(exctype))
    271 if res == 0:
--> 272 raise ValueError("invalid thread id")
    273 elif res != 1:
    274 # """if it returns a number greater than one, you're in trouble,

<type 'exceptions.ValueError'>: invalid thread id

[Ville M. Vainio (villemvainio)]

Can you reproduce this with 0.8.4?

Additionally, what backend are you using? At least I didn't get the error with 0.8.4 + tk backend.

[Ville M. Vainio (villemvainio)]

Can you reproduce this with 0.8.4?

Additionally, what backend are you using? At least I didn't get the error with 0.8.4 + tk backend (but that is to be expected)

[Ville M. Vainio (villemvainio)]

Also, it doesn't repro on wx and Qt4 backends, on windows.

[TheFlyingPengwyn (dcocks)]

Thanks for checking Ville. I tried using 0.8.4 and there was no change (exact same code above generated exact same error).

I'm using the GTKAgg backend for matplotlib at the moment. I'll have a go changing to a few different backends tomorrow and see if i can cause something different to happen.

[TheFlyingPengwyn (dcocks)]

Sorry about the long delay - I had completely forgotten about this bug report!

One quick question - why wouldn't you expect the Tk backend to not error? Does that backend run in the same thread?

I tested with all the backends that I could and found the problem only occurs in the GTK backends.

Specifically I found:

GTK - raised ValueError
GTKAgg - raised ValueError
GTKCairo - raised ValueError

Backends that worked fine (i.e. raised KeyboardInterrupts - i didn't check any plotting however)
TkAgg
Agg2
Agg
Cairo
GDK
Pdf
PS
SVG
Template

And backends that i don't have the required modules for, or no support for:
FltkAgg
QtAgg
Qt4Agg
WX
WXAgg
CocoaAgg
EMF

Anything not listed in not present in my matplotlib.backend.all_backends list.

[Fernando Perez (fdo.perez)]

I can't reproduce the problem either, I just tested on Linux with Wx, GTK, Qt and Qt4, and in all cases I get the correct KeyboardInterrupt reported.

[Andrew P (app26)]

I can reproduce this problem on RHEL 5 using fresh build of python 2.6, ipython 0.9.1, with either WX, WXAgg or GTK, GTKAgg backends.

$ ipython -pylab

.../python2.6/site-packages/ipython-0.9.1-py2.6.egg/IPython/Magic.py:38: DeprecationWarning: the sets module is deprecated
  from sets import Set

Python 2.6 (r26:66714, Oct 2 2008, 16:43:05)
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.

In [1]: for i in xrange(1000000) : pass
   ...:
^C
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)

/data/store/app26/dla/dla_dust/<ipython console> in <module>()

.../python2.6/site-packages/ipython-0.9.1-py2.6.egg/IPython/Shell.pyc in sigint_handler(signum, stack_frame)
    320
    321 if CODE_RUN:
--> 322 _async_raise(MAIN_THREAD_ID,KeyboardInterrupt)
    323 else:
    324 KBINT = True

.../python2.6/site-packages/ipython-0.9.1-py2.6.egg/IPython/Shell.pyc in _async_raise(tid, exctype)
    301 ctypes.py_object(exctype))
    302 if res == 0:
--> 303 raise ValueError("invalid thread id")
    304 elif res != 1:
    305 # """if it returns a number greater than one, you're in trouble,

ValueError: invalid thread id

[Nathaniel Smith (njs)]

I also see this with both 0.8.4 and 0.9.0.

I suspect the problem is that the bug only surfaces on 64-bit platforms; I'm running on x86-64, and thread ids are very long integers:

In [2]: thread.get_ident()
Out[2]: 140479552612064

And indeed, if I try to pass that directly to PyThreadState_SetAsyncExc, then it doesn't work somehow:

ipdb> ctypes.pythonapi.PyThreadState_SetAsyncExc(140479552612064, ctypes.py_object(KeyboardInterrupt))
0

("0" indicates an invalid thread id.) But, if I explicitly cast it to a long (the documented type for this function, see http://docs.python.org/c-api/init.html#PyThreadState_SetAsyncExc ), I get:

ipdb> ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(140479552612064), ctypes.py_object(KeyboardInterrupt))
1
[and then after hitting Enter a few times...]
KeyboardInterrupt Traceback (most recent call last)
[...]

So it looks like the fix is just to wrap a call to ctypes.c_long() around MAIN_THREAD_ID in the async_raise code.

[Nathaniel Smith (njs)]

Above, as a patch against lp:ipython head.

[Brian Granger (ellisonbg)]

I have committed the fix to the keyboardint64bit branch and will post if for review

[Brian Granger (ellisonbg)]

Could the folks who were seeing this bug try the fix in this branch:

bzr branch lp:~ellisonbg/ipython/keyboardint64bit

Thanks!

[TheFlyingPengwyn (dcocks)]

Great detective work Nathaniel!

I tried the fix Brian and it appears to work perfectly, using the same test loop as my original post.