Incorrect tests show when selecting filters for image reports 2.0

[Stevan Radaković]

1. Created a new image report 2.0 named test23.
2. Added a new chart and Selected this filter:
http://validation.linaro.org/dashboard/filters/~vishalbhoj/hisilicon-lsk
3. The Select tests column doesn't show the right tests part of that builds/job.

[Stevan Radaković (stevanr)]

Hi Vishal,

I'm not sure that this is correct.. I see all those tests listed in the select box in the bundles belonging to this filter. And that's exactly what we do, we display all the tests used in all the bundles of the selected filter.
Here are couple of examples from this filter:

http://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/6a3afe96bf74eefc5d248195782f1bf4d650e98b/
http://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/5989c5fa7087da601fab2d91d022ed1de8dc209b/
http://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/d5673279baad3e5bc7d58696dd976016c9b9fd9b/

The only problem I see here is that the list is not wide enough for user to see the whole test name if it's longer and that the test names are not sorted in any way.

[vishal (vishalbhoj)]

Hi Stevan,

The example taken above may not be very appropriate since the above bundle has generally run all the tests. Can you please pick the following filter:
http://validation.linaro.org/dashboard/filters/~fabo/juice-base-aosp

We see benchmark related test cases in the list but we don't see CTS tests that were part of the bundle.

[Stevan Radaković (stevanr)]

If chart you're trying is of the 'measurement' type, you will see the list of test cases names related to that filter.
If the type is 'pass/fail'(default) the list of test names will be displayed.
I've just tried the filter you provided with the 'pass/fail' chart type and I can see the CTS tests (screenshot attached).

[vishal (vishalbhoj)]

Thanks. Things are much lear now. The confusion arised with tests which report results as pass/fail and have measurements. I was trying to create a chart with measurement type but those tests were not showing up. They show up when I create a chart as pass/fail . It will help only to compare the whether the tests passed or failed but doesn't help to compare the measurement .

Here is an example filter:
http://validation.linaro.org/dashboard/image-charts/lsk-test

It reports the pass/fail result but unable to compare the results. Is this the expected behavior ?

Also, Are the filtering of tests is still based on all the test case present in the stream i.e "http://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/" or is it through the filter ?

I still see lot of tests in the Image chart filter which were never run on that build but show up there.

[Stevan Radaković (stevanr)]

> Here is an example filter:
> http://validation.linaro.org/dashboard/image-charts/lsk-test
> It reports the pass/fail result but unable to compare the results. Is this the expected behavior ?

Well yes, one chart can either be pass/fail or measurement... But you can add multiple charts in one report, so you can have one which tells you if any of the tests failed and one more which shows you the measurements.

> Also, Are the filtering of tests is still based on all the test case present in the stream i.e
> "http://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/" or is it through the filter ?

> I still see lot of tests in the Image chart filter which were never run on that build but show up there.

We take all the bundles from all the filters and then take all the tests (test cases, in case of measurement chart) that were EVER run in those bundles and show them for you to select which one you would like to see on your chart. It could be that the tests you see where only present in one test run in older date so that's why you get the impression that it's not shown in the bundle.

[vishal (vishalbhoj)]

Hi Stevan,

I have tried adding a new image chart filter with measurements but I cannot find the test that I reported above i.e hostshell-workload(-c config_iks3.csv -g ssh://<email address hidden>/srv/linaro-private.git.linaro.org/workload-automation.git )

but I can't find the same in the test list on the image chart filter when I create a measurement type chart. The results for the test has both pass/fail and measurements:
http://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/576cf20c8acabe8373f61028685b747f1a3b199d/c50a1c37-184f-4850-a648-86e5a7cd9f8e/

[Stevan Radaković (stevanr)]

Hey Vishal,

As mentioned above, if you are creating measurements report, you will be able to chose only from the test cases individually and not from the tests.. Now I guess you want to see measurements of all the test cases in your particular test, but unfortunately UI forces you to select them individually. If you find the test cases one-by-one and select them that would have the same result.
I.e. you'll have to select (Serial, Run, A7Cores, A15Cores...). Note that only test cases with actual measurements will be shown here.

Now I see that there some of those test cases are missing from the select field, let me check that up.
Meanwhile I'll see what can I do to make the UI at least a bit easier to use.

[Alan Bennett (akbennett)]

Reviewing the comments, things have gotten better and there may still be some issues. I believe the status is "in-progress" and the priority is no longer 'critical, drop everything and fix', but more in line with a high priority.

[Antonio Terceiro (terceiro)]

Hi Stevan, what's the status of this?

[Stevan Radaković (stevanr)]

Hey Antonio,
It's still under way, should have fix soon.

[Stevan Radaković (stevanr)]

Patch: https://review.linaro.org/744