Intermediate results should not use pass/fail, it confuses LAVA
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Linaro PMWG QA |
Fix Released
|
High
|
sanjay singh rawat |
Bug Description
Despite pm-qa has fewer than 100 test scripts spread across multiple test suites, LAVA reports over than 250 test cases from this suite.
The reason for this mis-interpretation is down to the fact that LAVA has a results parsing method by which it uses a reg-ex to interpret each log outputs from the test suites and matches them with 'pass/fail' tokens.
In pm-qa suites the scripts output log files even for intermediate test steps with a pass/ fail verdict which prompts LAVA to pick up as separate test cases.
The solution is to use say OK/NOT OK tokens for all intermediate test steps in each script and have only a final pass/fail verdict from each script in pm-qa.
This would let LAVA results parser to ignore any intermediate steps to be reported as separate test cases while results parsing.
Changed in linaro-power-qa: | |
assignee: | nobody → Hongbo Zhang (hongbo-zhang) |
importance: | Undecided → High |
status: | New → Confirmed |
Changed in linaro-power-qa: | |
milestone: | none → 2013.04 |
summary: |
- pm-qa test scripts need to use only pass/fail verdicts as a single final - verdict rather than intermediate results which makes LAVA to interpret - these as multiple separate test cases + Intermediate results should not use pass/fail, it confuses LAVA |
Changed in linaro-power-qa: | |
assignee: | Hongbo Zhang (hongbo-zhang) → sanjay singh rawat (sanjay-rawat) |
Changed in linaro-power-qa: | |
status: | Confirmed → Fix Committed |
Changed in linaro-power-qa: | |
status: | Fix Committed → Fix Released |
- sent patch for the issue
- need to merge and relase