Inconsistent result aggregation and reporting using lava-test-case
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
LAVA Server |
Fix Released
|
High
|
Senthil Kumaran S |
Bug Description
The following repo contains an example of the issues I see:
http://
The example runs some tests in a similar way that the Java/JTREG tests run
and for each test it invokes 'lava-test-case $TESTCASE_NAME --result pass|fail"
to record the result.
Looking at the result bundle[1] the number of tests reported doesn't match the
number of times lava-test-case is invoked.
The test-runner script runs in two ways:
while IFS=' ' read -r tc rem; do
echo "$FUNCNAME: TC=<$tc>, REMAINDER=<$rem>"
lava-test-case $(basename $tc .java) --result $result
done < $filename
Using this loop I see only 18 results reported (16 pass, 2 fail).
If I change this loop to not invoke lava-test-case immediately:
local ofile=/
echo "#!/bin/bash" > $ofile
while IFS=' ' read -r tc rem; do
echo "$FUNCNAME: TC=<$tc>, REMAINDER=<$rem>"
echo "lava-test-case $(basename $tc .java) --result $result" >> $ofile
done < $filename
[ -e $ofile ] && chmod 755 $ofile
[ -e $ofile ] && $ofile
then I see 35 results reported (32 pass, 3 fail) - which is the correct number
based on the (synthesized) tests that are executed. summary.txt in the repo
contains 35 lines.
Is this expected behaviour? The JTREG tests are using the first form of the loop and this mechanism was working[2] on Jan 22nd 2014 but has been broken since then.
[1] Examples of invocations against KVM and the RTSM model using the job descriptions
checked into:
http://
are recorded here:
https:/
https:/
summary: |
- Inconsistent result aggregation and reporting using lava_test_case + Inconsistent result aggregation and reporting using lava-test-case |
description: | updated |
Changed in lava-server: | |
assignee: | nobody → Senthil Kumaran S (stylesen) |
importance: | Undecided → High |
status: | New → Confirmed |
Changed in lava-server: | |
status: | Confirmed → In Progress |
Changed in lava-server: | |
status: | In Progress → Fix Committed |
Changed in lava-server: | |
status: | Fix Committed → Fix Released |
Any progress on this Senthil?