checkbox should warn the user if a test case is not properly formatted
Bug #1187216 reported by
Jeffrey Chang
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Checkbox (Legacy) |
Fix Released
|
Medium
|
Daniel Manrique |
Bug Description
Checkbox should warn the user, if any field in a test case is not in proper format.
Currently, test case with format problem will be skipped automatically, see bug 1185759.
We should let the user know more info about the problem.
Related branches
lp:~roadmr/checkbox/1187216-check-job-format
- Jeff Lane : Approve
- Daniel Manrique (community): Needs Resubmitting
-
Diff: 85 lines (+32/-6)3 files modifiedcheckbox-old/checkbox/tests/test_message_files.py (+29/-3)
checkbox-old/debian/changelog (+2/-0)
checkbox-old/jobs/fingerprint.txt.in (+1/-3)
tags: | added: core |
tags: | added: ce-qa-concern |
Changed in checkbox: | |
status: | In Progress → Fix Committed |
Changed in checkbox: | |
status: | Incomplete → Confirmed |
Changed in checkbox: | |
status: | Confirmed → Triaged |
Changed in checkbox: | |
importance: | High → Medium |
tags: |
added: checkbox-core removed: core |
affects: | checkbox → checkbox-legacy |
Changed in checkbox-legacy: | |
status: | Triaged → Fix Released |
To post a comment you must log in.
Hello!
We've had some discussions on the most desirable behavior for this.
The most obvious choice is for checkbox to warn at runtime of any erroneous jobs. The problem here is that if we warn the user mid-run about a bad job, there's little the user can do: for instance, he may not have knowledge of what the proper format is, or where to locate the job files. Worse, he'd have to abort the run in order to change the job, since checkbox doesn't reread jobs from disk if they change.
The other option is checking the jobs at build time. This simply warns the developers if a job has an incorrect format, and refuses to build a checkbox package with bad jobs. The developers *are* familiar with the format and can fix the problem so that the package always contains valid jobs. The problem with this option is that if someone other than checkbox developers wants to write tests, they will not be checked.
A possible compromise is including a job checking script with checkbox, that can be run both by developers and any other test writers. However I strongly believe that this would have to be separate from checkbox itself. The reason, as stated above, is that once the run is started, there's little we can do to fix a bad job, so for checkbox to warn the user of something that already failed and can't be fixed feels very unfriendly.
I'll get to work on something to address this need.