Some test cases from the 12.04.4 whitelist were not started

[Po-Hsu Lin]

While testing the 12.04.5 with 12.04.4 whitelist, some test cases were not started at all
e.g.
graphics/maximum_resolution
graphics/rotation
graphics/glxgears
graphics/cycle_resolution

Seems that they're not in the "desired_job_list".

[Zygmunt Krynicki (zyga)]

Same session in json format

[Zygmunt Krynicki (zyga)]

Looking at what was in desired job list:

1) graphics/maximum_resolution - we have "2013.com.canonical.certification::graphics/generator_maximum_resolution" instead but it was *not* selected so it didn't run.
2) graphics/rotation - we have "2013.com.canonical.certification::graphics/generator_rotation" but it was also not selected.
3) 2013.com.canonical.certification::graphics/display actually is an existing job, it *was* selected, it ran, it passed.
4) graphics/cycle_resolution - we have "2013.com.canonical.certification::suspend/cycle_resolutions_after_suspend" and it was selected but it didn't run.

[Po-Hsu Lin (cypressyew)]

Interesting, I think maybe the root cause is that we didn't put those generator jobs into our whitelist

[Kristin Chuang (kristinchuang)]

Same issue met in QAs' tests, add ce-qa-concern tag.
Also modify original description's "graphics/display" to "graphics/glxgears" after double checking with Sam on the correct test case.

[Po-Hsu Lin (cypressyew)]

The root cause is that we convert all these graphic jobs to be generated by the corresponding generators for hybrid graphic test in 14.04, therefore they can't be found in the graphic job file.

An easy workaround for this is to use the old graphic job file (the rev before we made changes for hybrid graphic tests), which is available here:
http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/view/2906/providers/plainbox-provider-checkbox/jobs/graphics.txt.in

[Daniel Manrique (roadmr)]

The affected jobs (these were converted to be generated by local jobs so they could generate tests for all available cards) are:

graphics/driver_version
graphics/resolution-change
graphics/resolution
graphics/screen-resolution
graphics/minimum_resolution
graphics/maximum_resolution
graphics/modes
graphics/color_depth
graphics/fresh_rate
graphics/graphic_memory
graphics/cycle_resolution
graphics/rotation
graphics/compiz_check
graphics/unity-support
graphics/glxgears
graphics/3d_window_open_close
graphics/3d_window_suspend_resume
graphics/multi_3d_windows_open_close
graphics/3d_window_move
graphics/screenshot
screenshot.jpg
graphics/screenshot_fullscreen_video
screenshot_fullscreen_video.jpg

The solution I'm proposing simply restores the old versions of these tests to a graphics-legacy.txt job file. This is seamlessly included in the graphics category thanks to local job magic (see local.txt for this). The job names *should* not conflict with the generated ones, as all generators are prefixed with generator and all the generated jobs are prefixed with numbers.

This should make legacy whitelists work again.

Though the per-card jobs are mostly useful for hybrid graphics testing, they can also replace the old jobs (seen above) for single-card testing. Add to your whitelist:

graphics/generator_.*
then for the tests you need, add the first-card version only, eg.
graphics/1_screenshot_.* # and so on