Standardize return codes
Bug #1025633 reported by
Para Siva
This bug affects 3 people
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| UTAH |
Fix Released
|
Wishlist
|
Javier Collado | ||
Bug Description
The following job reports 'Success' despite some test failures with return code 1.
http://
Please see the following log file, which reports a few failures,
10.189.
Related branches
lp:~javier.collado/utah/returncodes
- UTAH Dev: Pending requested
-
Diff: 391 lines (+75/-35)11 files modifieddebian/changelog (+3/-0)
examples/run_install_test.py (+4/-3)
examples/run_test_bamboo_feeder.py (+4/-3)
examples/run_test_cobbler.py (+4/-3)
examples/run_test_vm.py (+4/-3)
examples/run_utah_tests.py (+5/-4)
utah/client/common.py (+15/-8)
utah/client/result.py (+1/-4)
utah/client/runner.py (+4/-4)
utah/client/tests/test_runner.py (+2/-2)
utah/run.py (+29/-1)
lp:utah
- Max Brustkern: Pending requested
-
Diff: 10666 lines (+6316/-1397)85 files modifiedCOPYING (+674/-0)
Makefile (+1/-1)
client-Makefile (+4/-1)
client.py (+15/-0)
conf/utah/uqt-vm-tools.conf (+1/-1)
debian/changelog (+101/-0)
debian/control (+17/-3)
debian/rules (+21/-2)
debian/utah.install (+2/-0)
debian/utah.postinst (+10/-2)
distribute_setup.py (+3/-1)
docs/source/conf.py (+42/-23)
docs/source/reference.rst (+17/-0)
examples/run_install_test.py (+43/-18)
examples/run_test_bamboo_feeder.py (+123/-0)
examples/run_test_cobbler.py (+55/-25)
examples/run_test_vm.py (+38/-14)
examples/run_utah_tests.py (+51/-31)
examples/utah_logs.sh (+3/-1)
examples/utah_syslog.sh (+1/-1)
setup.py (+20/-6)
tests/test_parser.py (+60/-0)
tests/test_preseed.py (+476/-33)
utah/__init__.py (+15/-0)
utah/client/__init__.py (+15/-0)
utah/client/common.py (+70/-32)
utah/client/examples/test_state_partial_all_failed.yaml (+90/-0)
utah/client/examples/test_state_partial_inprogress.yaml (+90/-0)
utah/client/examples/test_state_partial_run_all.yaml (+90/-0)
utah/client/exceptions.py (+15/-0)
utah/client/phoenix.py (+40/-25)
utah/client/result.py (+48/-22)
utah/client/runner.py (+66/-58)
utah/client/state_agent.py (+15/-0)
utah/client/testcase.py (+73/-51)
utah/client/tests/__init__.py (+14/-0)
utah/client/tests/common.py (+43/-0)
utah/client/tests/manual_privileges.py (+17/-2)
utah/client/tests/test_common.py (+22/-7)
utah/client/tests/test_jsonschema.py (+21/-49)
utah/client/tests/test_misc.py (+15/-0)
utah/client/tests/test_phoenix.py (+25/-10)
utah/client/tests/test_result.py (+22/-5)
utah/client/tests/test_runner.py (+22/-7)
utah/client/tests/test_state_agent.py (+134/-68)
utah/client/tests/test_testcase.py (+46/-31)
utah/client/tests/test_testsuite.py (+45/-23)
utah/client/tests/test_vcs_bzr.py (+15/-0)
utah/client/tests/test_vcs_dev.py (+15/-0)
utah/client/tests/test_vcs_git.py (+15/-0)
utah/client/tests/test_yaml.py (+51/-36)
utah/client/testsuite.py (+31/-13)
utah/commandstr.py (+15/-0)
utah/config.py (+192/-18)
utah/exceptions.py (+15/-12)
utah/group.py (+15/-0)
utah/iso.py (+277/-118)
utah/isotest/data/file_list_di (+23/-23)
utah/isotest/data/file_list_ubiquity (+21/-21)
utah/isotest/iso_static_validation.py (+85/-60)
utah/orderedcollections.py (+90/-0)
utah/parser.py (+211/-0)
utah/preseed.py (+669/-119)
utah/process.py (+20/-5)
utah/provisioning/__init__.py (+15/-0)
utah/provisioning/baremetal/__init__.py (+15/-0)
utah/provisioning/baremetal/bamboofeeder.py (+301/-0)
utah/provisioning/baremetal/cobbler.py (+124/-130)
utah/provisioning/baremetal/exceptions.py (+15/-0)
utah/provisioning/baremetal/power.py (+90/-0)
utah/provisioning/exceptions.py (+15/-0)
utah/provisioning/inventory/__init__.py (+15/-0)
utah/provisioning/inventory/exceptions.py (+15/-0)
utah/provisioning/inventory/inventory.py (+20/-5)
utah/provisioning/inventory/sqlite.py (+87/-52)
utah/provisioning/provisioning.py (+337/-107)
utah/provisioning/vm/__init__.py (+15/-0)
utah/provisioning/vm/exceptions.py (+15/-0)
utah/provisioning/vm/libvirtvm.py (+124/-70)
utah/provisioning/vm/vm.py (+16/-0)
utah/publish.py (+146/-0)
utah/retry.py (+15/-0)
utah/run.py (+142/-34)
utah/timeout.py (+21/-4)
utah/url.py (+182/-17)
| tags: | added: qa-manual-testing quantal |
Revision history for this message
| Max Brustkern (nuclearbob) wrote | #1 |
| Changed in utah: | |
| status: | New → Triaged |
Revision history for this message
| Javier Collado (javier.collado) wrote | #2 |
| Changed in utah: | |
| importance: | Undecided → Medium |
| assignee: | nobody → Javier Collado (javier.collado) |
Revision history for this message
| Javier Collado (javier.collado) wrote | #3 |
Revision history for this message
| Max Brustkern (nuclearbob) wrote | #4 |
| Changed in utah: | |
| importance: | Medium → Wishlist |
| summary: |
- Utah reports Success despite some post installation test failures + Standardize return codes |
| Changed in utah: | |
| status: | Triaged → Fix Committed |
| status: | Fix Committed → Fix Released |
To post a comment you must log in.
Right now, we try to report success if all jobs were run successfully, regardless of their outcome. If we want to put in a check for the job success as part of the test script, I think we should decide what output codes mean "test failure" and what output codes mean "test error" and report those consistently. The other option would be to validate the test results as part of the jenkins job; we could grep all the yaml files for returncode, and fail if a non-zero value comes up.
Relatedly, jobs which threw exceptions when provisioning a machine also reported success earlier today. That should now be fixed in the tree, and I'll be ordering a new build once I've got all the merges done.