assess checks for new failure patterns automatically

[Robert Collins]

By mining check (and gate) runs that are possibly nondeterministic tests (e.g. passed unittests, fail in tempest) we can identify possible new flaky patterns automatically.

Basic process:
 - matches known flaky pattern: discard
 - diff against a recent successful run to build a candidate pattern
 - first occurrence of candidate pattern? hold on shelve
 - second occurrence of candidate pattern, especially in a different project, and definitely not in the same changeset? flag possible new failure pattern for human review