network script incorrectly failing

[Jeff Lane ]

The intention of the multiple runs in a network test was twofold:
1: Run 4 iterations and the average of the four should be higher than 80% line rate.
2: If the average of 4 iterations is below the 80% threshold, then examine the individual iterations and make a judgement call based on the number of iterations that ARE above the threshold.

What seems to be happening here, however, is that the test fails if the last iteration is below the threshold:

https://certification.canonical.com/hardware/201802-26098/submission/159167/test/75562/result/12060315/

INFO:root:-------------------- Test Run Number 1 --------------------
INFO:root:Using 10 threads.
INFO:root:Avg Transfer speed: 97402.65222222223 Mb/s
INFO:root:97.40% of theoretical max 100000 Mb/s
INFO:root:Average CPU utilization: 24.0%
INFO:root:
INFO:root:-------------------- Test Run Number 2 --------------------
INFO:root:Using 10 threads.
INFO:root:Avg Transfer speed: 80684.00158888889 Mb/s
INFO:root:80.68% of theoretical max 100000 Mb/s
INFO:root:Average CPU utilization: 24.2%
INFO:root:
INFO:root:-------------------- Test Run Number 3 --------------------
INFO:root:Using 10 threads.
INFO:root:Avg Transfer speed: 96833.79111111112 Mb/s
INFO:root:96.83% of theoretical max 100000 Mb/s
INFO:root:Average CPU utilization: 23.3%
INFO:root:
INFO:root:-------------------- Test Run Number 4 --------------------
INFO:root:Using 10 threads.
INFO:root:Avg Transfer speed: 77912.07777777777 Mb/s
INFO:root:77.91% of theoretical max 100000 Mb/s
INFO:root:Average CPU utilization: 20.8%
WARNING:root:Poor network performance detected against 192.168.104.101
WARNING:root: Transfer speed: 77912.07777777777 Mb/s
WARNING:root: 77.91% of theoretical max 100000 Mb/s

So this NIC was marked failed because the 4th run was below 80%, however, the overall average throughput of all four runs was actually nearly 90Gb/s at 88208.130375 Mb/s

The script should be corrected to honor the original intent outlined above so that a test run with these numbers produces a warning that at least one iteration was below the threshold, but still pass. This is because due to network activity, memory usage, system load, and a whole host of other variables on both the sender and receiver, it's possible to see widely varying results between iteratinos, as you can see in the above we go from 97Mb/s to 80Mb/s to 96Mb/s to 78Mb/s.

[Maksim Beliaev (beliaev-maksim)]

Bug was migrated to GitHub: https://github.com/canonical/checkbox/issues/154.
Bug is no more monitored here.