> With recent work in my trunk-dev branch, pretty much all the points in > the original report are dealt with.  The solutions aren't super-clean, > and we still have lots of room for improvement on iptest (mainly its > options handling and verbosity), but by and large the core problems > reported above are fixed. Yes, I think some of the original issues have been addresses, but not all of them. But we should probably create, new more focused tickets about these isssues. These issues may overlap with other iptest tickets you have created... Issue 1 ====== It is still super easy for a single developer to break the test suite by making their new module depend on something (twisted, wx, etc.). For that dev, they always have the dep. installed, so they never see the failure, so all their new tests will run fine. This is particularly insidious (still) for twisted because iptest does not have a clean way of declaring test dependencies and run methods for individual files. We are trying to do this by subpackage, but that is quite inflexible - what if only a single file in a subpackage depends on twisted - then the rest of the subpackage can't be tested using nose. We need a simple, declarative API for "declaring" the testing related information on a per subpackage or per module basis. This includes: * Dependencies * Method of running the test (nose+iptest/trial) * Thing like "test this subpackage this way, but these two modules are different" We run into this with frontend, which uses twisted in a few places, but not everywhere. I don't think this needs to be in place for 0.11, but because some of these issues are quite subtle, I don't want us to forget them. Thus, I think it makes sense to keep a ticket around for this. Issue 2 ====== Our test suite is still quite fragile and difficult to debug when things go wrong (case in point, your recent bug report about the history being dumped to the screen). If we can get the dpendency issues worked out (to isolate twisted and nose) I think most of this fragility is localized to the nose extension that handles the one time IPython shell. I think our test infrastructure has to be completely independent of our main IPython code (the testing infrastucture should never do "from IPython import ..."). At some point, we figured out how we could enable the writing of IPython doctests without the nose extension (do you remember how this can be done?). As I remember we talked about using this idea along with pexpect to isolate the running IPython to another process. Can we keep a ticket for this and fill in some of the details that we talked about. I honestly don't remember what solution we came up with, but it was somehow related to the new parametric testing you came up with. Again, I don't think this is 0.11 critical. > Brian, what do you think?  We're never going to get off of the > nose/twisted pair of tools because twisted.trial is very limited in > critical areas compared to nose, yet needed for managing the reactor. > So unless twisted.trial were to massively improve (extensible doctest > management, full test discovery of non-unittest tests, etc), it's not a > realistic option. > > I'm inclined to close this bug now, leaving the remaining iptest > problems to their own bug reports so we can act on them with more > specificity. > > OK with that? > > -- > Our method of running the test suite (iptest) is broken > https://bugs.launchpad.net/bugs/362142 > You received this bug notification because you are a member of IPython > Developers, which is subscribed to IPython. > > Status in IPython - Enhanced Interactive Python: In Progress > > Bug description: > Currently we run the IPython test suite using a custom command line program "iptest" that calls nosetests with the right options.  The problem with this is that the entire approach is very fragile.  A couple of problems exist: > > * The integration with twisted.trial seems to be buggy at best.  But, we need to use twisted.trial to test things in kernel and frontend. > * We are getting lots of odd errors/failures that are impossible to track down. > * Different tests have very different and at times conflicting dependencies.  For instance, we have code that uses wx, qt, twisted, etc.  All of these things should not be tested in the same process. > > We need a new way of running tests that: > > * Run's certain tests in subprocesses. > * Isolates tests based on dependencies. > * Runs twisted using tests using trial. > * Allow packages/modules to declared their deps without being imported so it can be decided how they will be run. > > The difficulty with this is that our test dependencies are not well isolated into package or even modules.  A perfect example of this is that frontend.asyncfrontendbase uses twisted.  Other tests on frontend could easily use wx and twisted.  Another problem is that the twisted using parts of our code can't use nose things like decorators or the additional testing syntax that nose supports.  Argh! > > > -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo