Quis custodiet ipsos custodes?

[Paul F. Dietz]

Recently we may have become too dependent on the random tester to find bugs introduced during development. The random tester is not guaranteed to cover all of SBCL's code, or even all the compiler's code. Bugs can hide in its blind spots, as the recent multiple-value-bind & if bug in MCCLIM showed.

I propose a test evaluation strategy based on mutation of SBCL itself. Parts of the compiler will be mutated to introduce bugs. Then, our tests will be used to try to find them. Any mutation that creates a real bug, but that is not found, is a bug in the test generator.

[Stas Boukarev (stassats)]

I can't really say that the mcclim thingy is totally surprising, I had
known about the class of problems that it might cause at the time, and
reversed a test case not from mcclim but from sbcl code. So it can't
be said that it was some obscure interaction of intermingling parts,
just a lapse of concentration.

The tn-ref-type was something that I hadn't considered, but in
hindsight it wasn't something that hard to imagine either.

So random testing shouldn't replace thorough thinking.

[Paul F. Dietz (paul-f-dietz)]

The compiler has about 57K lines. Of that, maybe 45K could be mutated. At 2 min/build (say) this would be two months to build with one mutant per line. So this doesn't seem entirely absurd as an ongoing project. It would help if I had coverage information on the compiler so I could know which parts are not being touched by tests.