Improve robustness and debuggability of test suite

[Fernando Perez]

This bug is a follow-up of:

https://bugs.launchpad.net/ipython/+bug/362142

which is now closed and split into several more specific reports. See the above for extra background.

The main points brought up by Brian:

Our test suite is still quite fragile and difficult to debug when things go wrong (case in point, your recent bug report about the history being dumped to the screen). If we can get the dpendency issues worked out (to isolate twisted and nose) I think most of this fragility is localized to the nose extension that handles the one time IPython shell. I think our test infrastructure has to be completely independent of our main IPython code (the testing infrastucture should never do "from IPython import ...").

At some point, we figured out how we could enable the writing of IPython doctests without the nose extension (do you remember how this can be done?). As I remember we talked about using this idea along with pexpect to isolate the running IPython to another process.

Jorgen noted:

I think using pexpect requires some research if you want to keep the the
tests running on windows.

Brian's reply:
Yes, this is a huge problem - pexpect doesn't work on Windows...

We should check with the Sage folks, I know at one point, they were working on a windows port of pexpect called wexpect.

Twisted does have many of the same capabilities that pexpect has for working with external processes. That might be our best bet.

[Fernando Perez (fdo.perez)]

Some notes on the above:

- I talked with William Stein (sage) a few days ago: no major progress from them on using pexpect for windows, instead they are moving away from pexpect as much as possible by building library wrappers for everything they can (like maxima and gap).

- I don't see us abandoning nose anytime soon. Trial has a few critical problems:

* it would make testing depend on twisted for us. Nose is a tiny, self-contained download, but Twisted is a much larger dependency. Too much for something as critical as testing.

* It has no discovery mechanism for doctests.

* My earlier efforts on doing ipython doctests without nose were only partly successful. In the last few weeks that I've been working a lot on this, I realized we really need quite a bit of machinery to be able to validate full ipython sessions, just doing syntax conversion to get a unittest out of a docstring (the part I was able to succeed on) isn't sufficient for the whole test suite.

I don't think it's really possible to expect our test suite to pass without an ipython shell running somewhere. We are more than just plain python, and to test all of that we really need a true ipython somewhere that we can beat on.

Having said all this, I think things now aren't really that bad. Having cleaned up the test machinery a lot recently, it's working quite well, debugging is possible with --pdb or other mechanisms, etc. The only thing that is hard is debugging a big doctest with lots of output, but that's simply how doctests are, not something specific to ipython (and for that reason, a bad idea in general if you want your tests to be debuggable).

I'm all for making our testing machinery as simple/robust as humanly possible, because I fully realize the importance of the test support in the health of a project. But having spent literally upwards of a hundred hours on this problem alone, I'm not sure right now I'm capable of making it much simpler without writing something like nose again, from scratch.

One ray of light may come from python itself: I think for 2.7 they are improving unittest, so perhaps we could find better support there once it's out.

I'm open to revisiting this question when possible, but I'm not terribly optimistic on *me* making lots of improvements beyond what's right now in my trunk-dev branch (which actually works quite well, it must be said). But I'd love to see someone who comes with a fresh eye on this problem make progress on it!