Total test case numbers changing in PM-QA test suite on Samsung Arndale Linux Linaro ubuntu.

Bug #1304373 reported by Botao
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Linaro PMWG QA
Fix Released
Critical
sanjay singh rawat

Bug Description

On Samsung Arndale with Linux Linaro ubuntu image, the number of total test case in pm-qa test suite is changing, on different image types. Inconsistence behaviour observed:
############################
hardware pack + nano rootfs:

7 April 2014:

https://validation.linaro.org/dashboard/streams/private/team/linaro/ci-linux-linaro-tracking-ll/bundles/d39f8d7c2254befc5a992e75d69730bcd23e6de2/

133 in total, all passed.

6 April 2014:

https://validation.linaro.org/dashboard/streams/private/team/linaro/ci-linux-linaro-tracking-ll/bundles/896617916191cfee01fd6a42bacbe18c16826081/

229 in total, 200 passed.
############################

hardware pack + server rootfs:

25 March 2014 (Linaro 14.03 monthly release image):

https://validation.linaro.org/dashboard/streams/private/team/linaro/pre-built-arndale/bundles/39d9e0aeef38715149bc752250d47e72d07d7c18/

263 in total, all passed.
############################

I also made my own LAVA job submission with the hardware pack #365 but replaced nano rootfs to server rootfs #651:

http://snapshots.linaro.org/ubuntu/images/server/651/linaro-saucy-server-20140406-651.tar.gz

This time it shows the same result as LAVA job 119566, that 7 April one:

https://validation.linaro.org/dashboard/streams/anonymous/botao-sun/bundles/a20e2529857ed4dd7f0a6ead06e13c59c95cf13e/

These behavours confused me and I have 3 questions here:

1. How many test cases in total of pm-qa test suite should be performed on Samsung Arndale Linux Linaro ubuntu?

2. The monthly release image for Linaro 14.03 is that pre-built image #618:

http://snapshots.linaro.org/ubuntu/pre-built/arndale/618/arndale-saucy_server_20140325-618.img.gz

which in LAVA it shows that all 263 test cases passed. Why there is a huge number (~50%) test cases dropped in the following test run? And even 29 test cases failed on that 6 April run?

3. I have walked through the git log to try to identify the root cause on both test definition and pm-qa repositories, but there is no commit related to the test cases dropping between 25 March 2014 and 7 April 2014. Is there an mechanism inside pm-qa test code to automatically decide how many test cases should be run on a specific hardware pack + rootfs combination? But if so, after I changed the rootfs from nano to server, why there are still only 133 test cases be executed?

##################################################
The images mentioned above can be downloaded here:
http://snapshots.linaro.org/kernel-hwpack/linux-linaro-tracking-ll-arndale/365/hwpack_linaro-arndale_20140407-1551_b365_armhf_supported.tar.gz
http://snapshots.linaro.org/ubuntu/images/nano-lava/651/linaro-saucy-nano-lava-20140406-651.tar.gz
http://snapshots.linaro.org/ubuntu/images/server/651/linaro-saucy-server-20140406-651.tar.gz

Amit Kucheria (amitk)
Changed in linaro-power-qa:
importance: Undecided → Critical
assignee: nobody → sanjay singh rawat (sanjay-rawat)
Revision history for this message
sanjay singh rawat (sanjay-rawat) wrote :

I checked the logs, the issue is cpuhotplug_07 subtest runs infinitely (bug 1304251) which from the logs
- hangs and exits (no test count is less)
- or returns 139 (SIGSEV) and continue running other tests (so test count increasing).
I had posted a patch for bug 1304251 which is under review, this will help to solve the issue.

Revision history for this message
sanjay singh rawat (sanjay-rawat) wrote :

patch is merged, will help to solve the issue in next release test cycle.

Changed in linaro-power-qa:
status: New → Fix Committed
Changed in linaro-power-qa:
status: Fix Committed → Fix Released
Revision history for this message
Milosz Wasilewski (mwasilew) wrote :
Revision history for this message
Botao (botao-sun) wrote :

This issue still exists by observing the dashboard:

https://validation.linaro.org/dashboard/image-reports/linux-linaro-arndale

Changed in linaro-power-qa:
status: Fix Released → In Progress
Revision history for this message
Botao (botao-sun) wrote :

From build #680, the total test case number can be stayed at 421:

https://validation.linaro.org/dashboard/image-reports/linux-linaro-arndale

Changed in linaro-power-qa:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.