snappy-device-agents does not properly return exit status for failed tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Testflinger |
Fix Released
|
Medium
|
Paul Larson |
Bug Description
when passing test_cmds, it can go in one of two ways:
1. a list of commands to run
When passed in like this, we want to try to run all of them, but we don't know what to do with failed steps. Does a failed step mean the whole run failed? or was it just checking something and it's ok that it failed? We don't know. So in cases like that, we've always handled it by sending back a non-zero default value of "4". Why 4? because it's something non-zero, and makes it a little more obvious that it came from testflinger if you go searching for why. This turned out being a horrible way to do things because there were too many unknowns
2. the new way - pass a text block to test_cmds and run the whole thing as a script. In this case, we don't worry about individual tests, just the whole rc of the script. So you can set -e if you want, or not. In *THIS* case, we should exit with the rc of the script, but we don't. It looks like this was just an oversight when this feature was added.
To reproduce this:
Submit a test something like this:
job_queue: test
test_data:
test_cmds: |
echo foo
exit 1
Then check the status:
$ testflinger-cli results <job_id> |jq -r .test_status
It should be 1, but instead it will be 0
Related branches
- Paul Larson: Approve
-
Diff: 16 lines (+2/-2)1 file modifiedsnappy_device_agents/__init__.py (+2/-2)
Changed in testflinger: | |
status: | New → In Progress |
Changed in testflinger: | |
status: | In Progress → Fix Committed |
Changed in testflinger: | |
status: | Fix Committed → Fix Released |
This is relatively minor, as you should never rely on testflinger to tell you if your test passed or failed, but it is very annoying and unexpected behavior.