Comment 2 for bug 1269782

| Unfortunately, there is currently no way to reliably identify the first
| case because the name of the test does not have to relate to the
| filename of the YAML file.

| The filter cannot tell the difference - there are simply no results
| which match the filter.

As a user of this stuff I have no idea what a "filter" is...

| It may be possible to collate the lava_test_shell data in the result
| bundle in such a way as to create a list of test definitions for that
| job and then annotate each test_id if the job failed.

| This could help the filter distinguish between tests which were not
| requested and tests which were requested but failed to run.

...I think the above matches what I'd thought was going on here. I had thought that we were scheduling a bunch of existing testsuites that LAVA knows about to run.