Option to group multiple jobs together in job trees

[James E. Blair]

Currently a single job can be used determine whether a group of jobs should subsequently run if that job succeeds. It would be nice to be able to group multiple jobs together and only run the next group if all of the jobs succeed. For example:

gate:
    - stage1:
        - pep8
        - requirements
        - stage2:
            - python26
            - python27
            - devstack

Would only run python26,27, and devstack if both pep8 AND requirements succeed. More complicated logic can also be supported:

publish:
    - stage1:
        - tarball
        - stage2:
            - pypi-upload
        - stage2b:
            - something-else
            - stage3:
                - something-further

Means:
    run tarball
    if tarball succeeds:
        run pypi-upload and something-else
        if something-else succeeds:
            run something-further

This changes the current semantics of dictionary keys in the job tree from referring to a job to referring to a job group. It is therefore incompatible with the current syntax. For backwards compatability, we should add a flag to the config file that
indicates the new format should be used; default it to false, then remove it in a later version.

[Antoine "hashar" Musso (hashar)]

I love that idea and have a perfect use case for it. A first stage to run linting / fast tests, a second stage to run the unit & integration tests :-]  That would save a bit of time!

[Antoine "hashar" Musso (hashar)]

For MediaWiki core the gating layout is:

    gate-and-submit:
      - mediawiki-core-merge:
        - mediawiki-core-phpcs-HEAD
        - mediawiki-core-jslint
        - mediawiki-core-jsduck
        - mediawiki-core-lint:
          - mediawiki-core-phpunit-databaseless:
            - mediawiki-core-install-sqlite:
              - mediawiki-core-qunit
              - mediawiki-core-phpunit-api
              - mediawiki-core-phpunit-misc
              - mediawiki-core-phpunit-parser

the -lint job is simply making sure the submitted PHP files do pass the linting which is almost always true. It then triggers the unit/integration tests.

The mediawiki-core-jslint runs the Javascript lint, if that fails, that does not prevent the tests to run :/

If I understand it correctly, your proposal would let me have a first stage made of typos/linting checks. Then if and only if all jobs are successful, a second staged is triggered.

So my layout would be something like:

gate-and-submit:
    stage-lint:
     - mediawiki-core-phpcs-HEAD
     - mediawiki-core-jslint
     - mediawiki-core-jsduck
     - mediawiki-core-lint
     - stage-fasttests:
        - mediawiki-core-phpunit-databaseless:
        - mediawiki-core-install-sqlite:
        - mediawiki-core-qunit
        - mediawiki-core-phpunit-api
        - mediawiki-core-phpunit-misc
        - stage-slowtests:
           - mediawiki-core-phpunit-parser

[Fredrik Medley (fredrik-medley)]

See also https://review.openstack.org/231124/

I have the use case of generic dependency graphs, i.e. acyclic graphs of dependencies, which the Zuul configuration layout.yaml cannot handle. Consider the following configuration:
gate:
  - A:
    - B:
      - D
      - E
    - C:
      - E

Zuul currently starts two E jobs, one when B has finished and one when C has finished. The information passed from Zuul, in our case to Jenkins, does not contain any information of where in the dependency tree the E job is located, so it is impossible for the executor to distinguish the jobs, unless reading status.json.

Also note that BuildSet in model.py index jobs by their names, so the about construction has unpredicted result. The function getBuild(job_name) is extensively used and will always return the latest added Build, the older Builds are silently being discarded and results are unchecked.

Please change the logic to run each job only once, which will avoid the problems with BuildSet. It also allows for specifying any generic dependency graph, i.e. all acyclic graphs.

A simple pattern to specify a full dependency graph is then:
gate:
  - A:
    - B
    - C
  - B:
    - D
    - E
  - C:
    - E

In this graph, the job E will not start until both B and C have finished.

If possible, a check in the layout validator to avoid circular dependencies would be nice, but the validator does not seem to resolve regular expressions on the job names as done in Layout.getJob(name).

Reviewing an example above,
gate:
    - stage1:
        - pep8
        - requirements
    - stage2:
        - python26
        - python27
        - devstack
will become
gate:
    - pep8:
        - python26
        - python27
        - devstack
    - requirements:
        - python26
        - python27
        - devstack
which will only run python26, python27 and devstack if both pep8 AND requirements succeed.