snappy-device-agents does not properly return exit status for failed tests

Bug #1698440 reported by Paul Larson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Testflinger
Fix Released
Medium
Paul Larson

Bug Description

when passing test_cmds, it can go in one of two ways:
1. a list of commands to run
When passed in like this, we want to try to run all of them, but we don't know what to do with failed steps. Does a failed step mean the whole run failed? or was it just checking something and it's ok that it failed? We don't know. So in cases like that, we've always handled it by sending back a non-zero default value of "4". Why 4? because it's something non-zero, and makes it a little more obvious that it came from testflinger if you go searching for why. This turned out being a horrible way to do things because there were too many unknowns

2. the new way - pass a text block to test_cmds and run the whole thing as a script. In this case, we don't worry about individual tests, just the whole rc of the script. So you can set -e if you want, or not. In *THIS* case, we should exit with the rc of the script, but we don't. It looks like this was just an oversight when this feature was added.

To reproduce this:
Submit a test something like this:
job_queue: test
test_data:
  test_cmds: |
    echo foo
    exit 1

Then check the status:
$ testflinger-cli results <job_id> |jq -r .test_status

It should be 1, but instead it will be 0

Related branches

Revision history for this message
Paul Larson (pwlars) wrote :

This is relatively minor, as you should never rely on testflinger to tell you if your test passed or failed, but it is very annoying and unexpected behavior.

Changed in testflinger:
importance: Undecided → Medium
assignee: nobody → Paul Larson (pwlars)
Paul Larson (pwlars)
Changed in testflinger:
status: New → In Progress
Paul Larson (pwlars)
Changed in testflinger:
status: In Progress → Fix Committed
Paul Larson (pwlars)
Changed in testflinger:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.