Test framework: Skipped runtime errors are still "unexpected" even if compile error is expected
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mars |
Fix Released
|
Medium
|
Matt Giuca |
Bug Description
If the MTC file includes under the "compile" section "expect: compile_error", this indicates that the test is supposed to compile successfully but currently doesn't, and not to be noisy about it (report the compile failure as an expected fail).
However, if the compile fails, all of the runtime tests are reported as "skipped" (which generates an 'x' in the visual output, and does not count as a pass/expected fail). For a compile error which has been expected, all of the skips should also be expected. Therefore, the 'x' should instead be a grey '.', and the skips should count towards the pass/expected fail total, not the skipped total (such that you can still get 100% pass/expected fail).
For example, current output:
mytest: .xxx (one red dot, three grey xs)
1/4 cases passed (or expected fail)
3 cases skipped
Expected output:
mytest: .... (one red dot, three grey dots)
4/4 cases passed (or expected fail)
Related branches
Changed in mars: | |
status: | Fix Committed → Fix Released |
Fixed in trunk r1103.