Detection for tests that fail due to error in test code needed

[Chris Gregan]

There is currently no way to detect when at Plainbox test dies or causes a system failure so bad that the test run must be recovered. It would be helpful for test cases quality maintenance to know when test processes quit in an ungraceful manner. If this could be flagged so it could, minimally, be displayed in the report would be great, with eventual plans to add support to C3.

[Daniel Manrique (roadmr)]

A simple way to achieve this is to set the comment to "this test was skipped because it crashed the test run". We already have a place where the outcome of the last attempted test is set to "fail" when the user selects "continue" as a resume option.

[Zygmunt Krynicki (zyga)]

This needs to be done at the framework level. What we've briefly disscussed with Chris is to have a set of flags associated with each test result. The framework could detect when the job was cancelled / interrupted (by killing the process) and when we didn't really observe the process exiting (so probably the machine rebooted). We should expose this in the API so that apps can do whatever is appropriate (render pretty reports, set comments, etc)

[Chris Gregan (cgregan)]

1) write a job that kills plainbox (IIRC there's one in stubbox or somewhere else)
2) run it and see what happens to plainbox
3) try stuff that misbehaves in other ways

One thing it boils down to is the exit status (not just exit code, exit status also has type of exit and more) we should get that data and use it.
The other thing is the pre-run/post-run session meta data update
Code that runs external programs is in plainbox.vendor.extcmd

[Zygmunt Krynicki (zyga)]

The replacement extcmd implementation can now detect exactly how a process has finished. We don't currently make use of that bue we can simply store that in the job result object. We still need to figure out what's needed to push this data to C3 and into test reports.

[Zygmunt Krynicki (zyga)]

Replacement extcmd almost here, I'll defer this until the outcome metadata API lands so adding new outcomes is easy.

[Zygmunt Krynicki (zyga)]

Next time please read the merge request. This works regardless of which runner is used. It can be a part of the next release.