Test framework re-compiles for each test case
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mars |
Fix Released
|
High
|
Matt Giuca |
Bug Description
The test framework is rather slow (especially when heavy analysis is performed) because it re-compiles each module from scratch for each test case. This results in N^2 performance for a given test module (if there is roughly one test case per function, the framework will run N tests, and for each test, will fully compile ~N functions). The result is that it is much faster to have each test case in its own module than lump them all together (which is undesirable).
The cause is that each runtime test and unit test calls `mars -c` which compiles the program, then runs a single command. The compilation testing calls `marsc`, and an additional call is made to `mars -c :ta` to get a list of all functions for unit testing. These can all be combined into a single compilation by using pipes.
Replace the initial call to `marsc` (the compilation testing) with a call to `mars -i` which performs compilation, and starts the interactive mode. This should check for errors, and if there are none, result in a Popen-like object which retains a persistent connection with the Mars prompt. All subsequent calls to run_mars should be replaced to a method call on this object, which enters a command into the prompt and checks its feedback without terminating the process. This should not result in interference between test cases (unless they read and write interactive variables, which they should not).
Related branches
Changed in mars: | |
status: | Fix Committed → Fix Released |
Woohoo! The current test suite takes 25 seconds to run on my laptop. The fast-testing branch, 2 seconds! I expect an even greater speedup on the (private) heavy-analysis branches.