Mir performance testing as a merge requirement

[Francis Ginther]

The mir team would like to run the glmark2 benchmarks on a native mir server across Desktop, N4 and N10 devices at the MP stage, and block merges on a significant drop in FPS performance.

We already have a glmark2 version that runs against a native mir server. The parts that are missing includes:

Infrastructure to run that test (jenkins and whatnot).

Policy around exactly when to fail the test, when to warn about a drop in performance, and when to pass the test.

A way to convert the data from glmark2 into something the CI team can report on.

A place to show the graphs of performance over time.

So, addressing these in reverse order (why not):

4: We already have performance graphs for mir, so I suspect this part will be easy.

3: This needs a small amount of python code, I suggest we emit a subunit result stream here - we could use this as a testbed for subunit. I imagine that this script would be done by someone in the QA team? Is the CI team ready to read subunit result streams? I'm more than happy to walk someone through what needs to happen here (the code is trivially easy)...

2: I suggest we get the tests running for a few weeks, graph the results, and decide the policy based on that data. We're going to be blocking merge proposals from this, so we need to make sure we get it right. I expect that the policy would be decided upon by a mixture of the mir and QA teams. The current idea is that < 5% drop would result in a warning, and >5% would result in a fail. It would be nice if we could tweak this policy easily without having to go through the CI team... but that's a minor detail.

1: Chris Gagnon is already working on a jenkins job that runs the mir demos and integration tests, but I think this should be a separate job, for a couple of reasons:

First, we're likely to want to run these performance tests in several places (MP CI runs to begin with, perhaps on a daily basis on top of the distro images as well?).

Second, I think it's worth keeping the concerns of "have we broken the mir demos or integration tests by altering API without updating them" and the concern of "have we regressed performance in this release" separate.

[Josh Arenson (josharenson)]

I'm looking to work on this. Can you point me to where the Jenkins script that actually runs the benchmark is located?