test run don't match test installed in kernel CI jobs

[Alexander Sack]

guess symptom of us not getting back proper failure reports if something goes wrong ... anyway, looking at the snowball kernel CI view seems that random "run" results get not reported, while more tests are getting installed.

http://validation.linaro.org/lava-server/kernel-ci-views/per_board/snowball_sd

pandaboard feels a bit better:
http://validation.linaro.org/lava-server/kernel-ci-views/per_board/panda

In what cases don't we see results? How can this problem be addressed?

[Andy Doan (doanac)]

looking at:

 http://validation.linaro.org/lava-server/scheduler/job/27529/log_file

The problem appears to be that we were unable to get the test results from the target device (see end of log file). At least one of the bundles we pulled didn't have valid JSON for some reason. Here's the guilty snippet of code:

                try:
                    tar = tarfile.open(result_path)
                    for tarinfo in tar:
                        if os.path.splitext(tarinfo.name)[1] == ".bundle":
                            f = tar.extractfile(tarinfo)
                            content = f.read()
                            f.close()
                            device_bundles.append(json.loads(content))
                    tar.close()
                except:
                    logging.warning(traceback.format_exc())
                    status = 'fail'
                    err_msg = err_msg + " Some test case result appending failed."
                    logging.warning(err_msg)

I think we could improve this code in a couple of ways:

1) put exception handling in the "for tarinfo in tar" loop so that the failure of handling one bad bundle doesn't cause of us to not parse all bundles

2) log the actual bundle that failed to parse (and maybe dump its contents)?

[Andy Doan (doanac)]

I just had another idea to make this fix better. In the event of an exception that's happening while doing the json.loads, we could add that content as an attachment to the "test_results" bundle so that we could analyze what the actual failure was.

I'll be doing some minor dispatcher fixes this week and will try to slip this in with those.