I've noticed while doing certification reviews that the test cpu/frequency_governors tends to fail fairly regularly. It's not 100% but the percentage is high enough that it's essentially ignored during Ubuntu Friendly testing and certification testing. We need to either fix this test to not fail so easily or remove it from any required testing regime.
I'm more in favor of refactoring the fail criteria myself. I have a feeling that what causes the test to fail can change depending on several factors, such as current system load, the type and model processor, variations in how the kernel handles throttling, etc. A one-size-fits-all kind of test is obviously not working so well.
One solution would be to raise the thresholds to only fail in egregious cases where a governor doesn't work (maybe set the threshold to 30% instead of the current 10%).
Second, the output could be fixed to be a little more useful. For example, in the test that fails, the test outputs this:
Error: measured speedup vs expected speedup is 19.5% and is not within 10.0% margin.
What does that even mean? Does that mean that what we saw was actually 19.5% faster? or that what we expected was 19.5% faster than what we saw? and 19.5% of what? Just looking at the output, I really only know that the test failed during the userspace governor test.
Here's an example from the test where it failed:
pid 2518's current affinity list: 0-3
pid 2518's new affinity list: 0
System Capabilites:
-------------------------------------------------
System has 4 cpus
Supported CPU Frequencies: 2501 MHz 2500 MHz 2200 MHz 2000 MHz 1800 MHz 1600 MHz 1400 MHz 1200 MHz 1000 MHz 800 MHz
Supported Governors: conservative ondemand userspace powersave performance
Current governors:
cpu0: userspace
cpu1: userspace
cpu2: userspace
cpu3: userspace
Userspace Governor Test:
-------------------------------------------------
Setting governor to userspace
Setting CPU frequency to 800 MHz
Running CPU load test...
pid 2518's current affinity list: 0
pid 2518's new affinity list: 0
Minimum frequency load test time: 34.95
Setting CPU frequency to 2501 MHz
Running CPU load test...
Maximum frequency load test time: 8.61
Note: found ida flag, increasing expected speedup by 8.0%
CPU Frequency Speed Up: 3.40
Measured Speed Up: 4.06
Percentage Difference 19.5%
Error: measured speedup vs expected speedup is 19.5% and is not within 10.0% margin.
On Demand Governor Test:
-------------------------------------------------
Setting governor to ondemand
Waiting 5 seconds...
pid 2518's current affinity list: 0
pid 2518's new affinity list: 0
done.
Running CPU load test...
On Demand load test time: 8.61
Percentage Difference vs. maximum frequency: 0.0%
Waiting 5 seconds...
pid 2518's current affinity list: 0
pid 2518's new affinity list: 0 done.
Performance Governor Test:
-------------------------------------------------
Setting governor to performance
Running CPU load test...
Performance load test time: 8.61
Percentage Difference vs. maximum frequency: 0.0%
Conservative Governor Test:
-------------------------------------------------
Setting governor to conservative
Waiting 10 seconds...
pid 2518's current affinity list: 0
pid 2518's new affinity list: 0
done.
Running CPU load test...
Conservative load test time: 34.63
Percentage Difference vs. minimum frequency: 0.9%
Restoring original governor to userspace
Confirming and setting to medium.