Jobs instantiated from a template won't be available in test case selection page

[Po-Hsu Lin]

The core issue is that none of the applications run the right jobs "early" before coming up with the tree of jobs. Traditionally each application ran local jobs (differently) and used that as a way to discover all generated jobs.

Now, with templates, resource jobs are the key so to have a correct and predictable experience in each application plainbox should offer some new APIs that list all of the jobs that should be started "early" and applications should just follow that list instead of trying to understand this themselves.

If testing with checkbox-cli, start with client-cert test plan, and with a template job ready in the graphics.txt.in job file, e.g.:

unit: template
template-unit: job
template-resource: graphics_card_resource
requires: device.category == 'VIDEO'
plugin: shell
id: graphics/{index}_driver_version_{product}
estimated_duration: 0.500
_summary: Test X driver/version for {product}
_description: Parses Xorg.0.Log and discovers the running X driver and version for the {product} graphics card
command: graphics_driver

It won't be available to testers with the following jobs name in the client-cert.whitelist:
graphics/1_driver_version_.*

But it will be available with:
graphics/{index}_driver_version_{product}

It looks like that the {index} and {product} were not translated into actual output that obtained from the graphics_card_resource job when running with normal procedure in checkbox-cli.

However, it can be started with plainbox:
plainbox run -i '2013.com.canonical.certification::graphics_card_resource' -i '2013.com.canonical.certification::graphics/1_driver_version_.*' --dont-suppress-output

(Note that the graphics_card_resource job is a job that exist in the graphics.txt.in, not from the resource job test plan)

And if you run the graphics_card job from the __resource__, the graphic resource job will be executed, but the driver_version job still skipped:
plainbox run -i '2013.com.canonical.certification::__resource__' -i '2013.com.canonical.certification::graphics_card' -i '2013.com.canonical.certification::graphics/1_graphic_driver_*' --dont-suppress-output

testing branch available: bzr branch lp:~cypressyew/junk/template-testing

Test steps:
1. Fetch the testing branch, run checkbox-cli in venv.
2. Select client-cert test plan only, and see if the driver_version job is available or not.

[Po-Hsu Lin (cypressyew)]

I can't put the graphics_card_resource job into the client-cert whitelist, since it will only be executed after selecting all the test cases that you want.

It must be executed before entering the test case selection stage, otherwise the template jobs won't be available there.
From the __resource__ job, I think the graphics_card_resource job was executed there before entering the test case selection page, but I don't know why those job are not correctly displayed.

[Zygmunt Krynicki (zyga)]

I have figured out an API that can solve this problem for plainbox. I'm not sure where it will be hanging from (either SessionState or SessionDeviceContext). The new function will compute a set of jobs to run in the "discovery" phase. This will also add phases to the session meta-data. I would like to have the following phases: "loading", "discovery", "execution". This will also help address other issues where interrupting checkbox-ng in the discovery phase won't really resume correctly.

The discovery function computes a set of jobs by following this rough algorithm:

I was typing it in gedit for proper indenting but then gedit crashed like it's 1985 and no, I didn't have my buffer back. WTF gedit :-(

[Zygmunt Krynicki (zyga)]

I'm going to add APIs that offer something like this:

def get_discovery_jobs():
    """ Get a set of jobs that need to run to discover all jobs. """
    jobs = set()
    for job in Job.objects.filter(plugin='local',
                                  partial_id__not_like='__%__'):
        jobs.add(job)
    for template in Template.objects.filter(template_unit='job'):
        jobs.add(Job.objects.get(id=template.resource_id))
    return jobs

def get_discovery_jobs(unit_list):
    """ Get a set of jobs that need to run to discover all jobs. """
    job_id_map = {
        unit.id: unit # XXX: broken if units have clashing IDs
        for unit in unit_list
        if unit.Meta.name == 'job'
    }
    jobs = set()
    for unit in unit_list:
        if unit.Meta.name == 'job':
            job = unit
            if job.plugin != 'local':
                continue
            # NOTE: This is a convention
            job_id = job.partial_id
            if job_id.startswith('__') and job_id.endswith('__'):
                continue
            jobs.add(job)
        elif unit.Meta.name == 'template':
            template = unit
            if template.template_unit != 'job':
                continue
            jobs.add(job_id_map[template.resource_id])
    return jobs

(the first version is more readable but depends on hypothetical django-like API for working with units)

[Zygmunt Krynicki (zyga)]

Moving to 0.22 as it's not realistic for 0.21 anymore

[Sylvain Pineau (sylvain-pineau)]

Checkbox-cli2 fixes the bug. It's a transitional tool before we can get support for launchers with session assistant.

I'm closing the bug as we have a working solution now.