Filters in LAVA dashboard doesn't support the comparison of measurements

[vishal]

Filters in LAVA dashboard doesn't support the comparison of measurements. If you look at the filter below:
https://validation.linaro.org/lava-server/dashboard/filters/~vishalbhoj/origen_benchmark

It gives us comparison on number of benchmarks run but doesn't provide us feature to comare the measurements associated with them.

[Dave Pigott (dpigott)]

Hi Stevan - Can you update with any progress?

[Dave Pigott (dpigott)]

https://cards.linaro.org/browse/LAVA-634

[Stevan Radaković (stevanr)]

Hey Dave, this went under my radar for previous cycle. I think this must be discussed first because measurements can be contained within every test case in one test run, I'm not sure this can be done on this same page as compare builds. But if the Image report editor card covers what Vishal wants then we're on the right path.

[Antonio Terceiro (terceiro)]

Hi Stevan - any updates on this?

[Stevan Radaković (stevanr)]

Maybe we should put this on need info since Vishal hasn't yet told us if new Image reports cover the request he has for this.

[vishal (vishalbhoj)]

If you look at the filter:
https://validation.linaro.org/dashboard/filters/~vishalbhoj/origen_benchmark

It shows the number of subtests for the main test for example if you look at entries under monkeyrunner(benchmarks-script/antutu/run.py), it says 9/9 but once you click one of the entry it shows up the measurements for each subtests like this:
https://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/876ee62fd0346b0f74ad8d365a0dd94af005910f/48b62671-3c07-47b9-9a1a-8351f0e077f4/

The idea was to have a way to compare these measurements across the builds which is still not possible.

[Stevan Radaković (stevanr)]

Each cell in the table(test run) can have from 1 to X test results with different measurements.
Do you suggest that we put all those measurements from one test run in one cell?
I'm not sure what kind of view we would have to implement but the current will not do.

Our question to you was whether the new Image Report Editor functionality (staging link: https://staging.validation.linaro.org/dashboard/image-charts/ , about to hit production, card which describes implementation in detail is https://cards.linaro.org/browse/LAVA-634) would be enough to cover what you need?

[Mike Holmes (mike-holmes)]

LNG would like to be able to compare these results
http://validation.linaro.org/dashboard/streams/public/team/linaro/ci-linux-lng-preempt-rt/bundles/c0f23d04092f2d483e0bfbda380cd79b67eaa826/bed9a5ed-bbe3-464f-bdf4-4c09f988b64d/

Looking by hand you can see some fail every day, and some fail
periodically, some fail more frequently for RT kernels.
So want to correlate between multiple bundles (RT and non RT kernel) over
time so that we can highlight issues that are RT kernel specific.

Mike

On 4 October 2013 11:22, vishal <email address hidden> wrote:

> If you look at the filter:
>
> https://validation.linaro.org/dashboard/filters/~vishalbhoj/origen_benchmark
>
> It shows the number of subtests for the main test for example if you look
> at entries under monkeyrunner(benchmarks-script/antutu/run.py), it says 9/9
> but once you click one of the entry it shows up the measurements for each
> subtests like this:
>
> https://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/876ee62fd0346b0f74ad8d365a0dd94af005910f/48b62671-3c07-47b9-9a1a-8351f0e077f4/
>
> The idea was to have a way to compare these measurements across the
> builds which is still not possible.
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dashboard.
> https://bugs.launchpad.net/bugs/1121629
>
> Title:
> Filters in LAVA dashboard doesn't support the comparison of
> measurements
>
> Status in LAVA Dashboard:
> Incomplete
>
> Bug description:
> Filters in LAVA dashboard doesn't support the comparison of
> measurements. If you look at the filter below:
>
> https://validation.linaro.org/lava-server/dashboard/filters/~vishalbhoj/origen_benchmark
>
> It gives us comparison on number of benchmarks run but doesn't provide
> us feature to comare the measurements associated with them.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dashboard/+bug/1121629/+subscriptions
>

[Stevan Radaković (stevanr)]

Well that's exactly what new Image report will bring, ability to take different filters and unify them in one single report.
I'm not sure that's what's Vishal is talking about since he mentioned measurement comparability.

[vishal (vishalbhoj)]

 I meant comparing the result for subtests would mean its restricted to pass/fail for the subtests. What I was referring is measurements for these subtests. Measurements are used to record the benchmark results. if filters supported to compare these measurements then we can see how performance/benchmarks have improved over the long run.

[Mike Holmes (mike-holmes)]

I am also assuming measurements are covered. To see if the throughput /
latency of identical tests RT vs non RT have any significant differences.

On 4 October 2013 12:38, vishal <email address hidden> wrote:

> I meant comparing the result for subtests would mean its restricted to
> pass/fail for the subtests. What I was referring is measurements for
> these subtests. Measurements are used to record the benchmark results.
> if filters supported to compare these measurements then we can see how
> performance/benchmarks have improved over the long run.
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dashboard.
> https://bugs.launchpad.net/bugs/1121629
>
> Title:
> Filters in LAVA dashboard doesn't support the comparison of
> measurements
>
> Status in LAVA Dashboard:
> Incomplete
>
> Bug description:
> Filters in LAVA dashboard doesn't support the comparison of
> measurements. If you look at the filter below:
>
> https://validation.linaro.org/lava-server/dashboard/filters/~vishalbhoj/origen_benchmark
>
> It gives us comparison on number of benchmarks run but doesn't provide
> us feature to comare the measurements associated with them.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dashboard/+bug/1121629/+subscriptions
>

[Stevan Radaković (stevanr)]

As I mentioned above, we will have all of that covered in https://cards.linaro.org/browse/LAVA-634
It's finished and about to hit production today, I can organize a small walkthrough of the new features with you guys if you wish.

[Mike Holmes (mike-holmes)]

I would love a walk though, I plan to use this a lot.

Mike

On 7 October 2013 05:39, Stevan Radaković <email address hidden>wrote:

> As I mentioned above, we will have all of that covered in
> https://cards.linaro.org/browse/LAVA-634
> It's finished and about to hit production today, I can organize a small
> walkthrough of the new features with you guys if you wish.
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dashboard.
> https://bugs.launchpad.net/bugs/1121629
>
> Title:
> Filters in LAVA dashboard doesn't support the comparison of
> measurements
>
> Status in LAVA Dashboard:
> Incomplete
>
> Bug description:
> Filters in LAVA dashboard doesn't support the comparison of
> measurements. If you look at the filter below:
>
> https://validation.linaro.org/lava-server/dashboard/filters/~vishalbhoj/origen_benchmark
>
> It gives us comparison on number of benchmarks run but doesn't provide
> us feature to comare the measurements associated with them.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dashboard/+bug/1121629/+subscriptions
>

[Milosz Wasilewski (mwasilew)]

On 7 October 2013 13:45, Mike Holmes <email address hidden> wrote:
> I would love a walk though, I plan to use this a lot.

count me in :) I also have several reports which need this feature.

[vishal (vishalbhoj)]

Same here. Let me know when you plan to have it.

[Antonio Terceiro (terceiro)]

Is this fixed now that "Image reports 2.0" is up?

[vishal (vishalbhoj)]

Yes this feature is available in image reports 2.0 but there are other set of issues with image reports 2.0 that still needs to be addressed. Should be handled in a new bug.

[Stevan Radaković (stevanr)]

OK thanks i'll mark it as released.