Ordering of tests

[Jeff Lane ]

It would be nice to be able to create a meta job that can call other jobs in a particular sequence. The use case is for using existing jobs as pre and post action tests. Recently , this became apparent with the consumer tests and suspend/resume.

For example a meta job could look like this (horribly pseudocoded)

plugin: shell
name: Suspend One Time
_description: This test will check network and resolution, suspend and resume the system, then recheck network and resolution to ensure they work after a sleep state.
command:
 from network.txt run internet_test job
 from video.txt run get_resolution job
 sleep_test --options
 from network.txt run internet_test job
 from video.txt run compare_resolution job
 if any fail
   return 1
 else
   return 0

[Brendan Donegan (brendan-donegan)]

Agreed. In general, checkbox needs to provide a way to give more control over the ordering of tests.

[Jeff Lane  (bladernr)]

This has been talked about and hashed out and rehashed out and so on for so long now. I'd love to see this in checkbox for 12.04 LTS finally.

[Daniel Manrique (roadmr)]

There are now various ways to get checkbox to run jobs in a particular order. Say you want this order:

A
B
C

If you want job B to *not* run unless job A passed (and it means *passed*, so even skipped or untested jobs will count as failures for dependency purposes) then on B, you specify depends: A.

If you want job C to run *after* A and B, even if those were failed or skipped due to dependencies or other circumstances, you put the ordering in the whitelist, and Checkbox will execute jobs in whitelist order.

Note that dependency ordering takes precedence over the whitelist; checkbox will not run a job before its dependencies. So if B depends on A, and the whitelist looks like this:

B
A
C

Checkbox will do the right thing, overriding the whitelist and running in A, B, C order.

I'll mark this as Fix Released as the desired functionality can be expressed in terms of ordering in the whitelist.

[Jeffrey Chang (modern911)]

On 12.04 with chekbox 0.13.7~ppa0.12.04, checkbox-oem 0.44-0-precise1~rev564, and attahed witelist file.
Test order is not like I specified in my whitelist file.
USB/keyboard always runs before usb/mouse.

[Jeffrey Chang (modern911)]

I'm running above whitelist file with checkbox-oem-gtk

And run test.wl with checkbox-gtk, test order was not as specified in whitelist.
The strange thing is, input/mouse will run before input/keyboard, but usb/keyboard will run before usb/mouse.