Skipped test case in "pwrmgmt" test suite should be considered as Passed.

[Botao]

On TI Panda board with Linux Linaro ubuntu Quantal image:

http://snapshots.linaro.org/quantal/pre-built/panda/55/panda-quantal_developer_20130125-55.img.gz

"cpufreq_05.4" test case is marked as skipped, all other test cases passed:

http://validation.linaro.org/lava-server/dashboard/streams/private/team/linaro/pre-built-leb-panda-4430/bundles/feb153ddea85ed897af4e795337a06480cbe5e69/1ddf89e5-5b0d-4ae2-b43d-011e1a58e77c/

In this condition, we can consider the power management test passed. Skipped test case should be marked by the power management team according to their specific purpose, like different hardware, test point, etc.

[Andy Doan (doanac)]

The output says:
cpufreq_05.4: combine governors not supported... skip

why would we mark this as passed when its not being tested? I'm not
familiar with this test suite, but that seems like its doing exactly
what it should be.

[Botao (botao-sun)]

I'm not familiar with the power management test details either. They were developed by power management work group and contain many test cases according to their requirement in test result output.

They have a completed test case collection, but not all our boards support all of them. Ideally, they should specify the test suite for every hardware, but the current way is to skip those test cases if a hardware doesn't support them.

[Fathi Boudra (fboudra)]

On 1 February 2013 06:20, Andy Doan <email address hidden> wrote:
> The output says:
> cpufreq_05.4: combine governors not supported... skip
>
> why would we mark this as passed when its not being tested? I'm not
> familiar with this test suite, but that seems like its doing exactly
> what it should be.

it makes more sense to mark passed than failed.

skip means ignore the test (current reason is that combine governor
isn't supported).
the test has been skipped, it's the expected behavior = passed.

what could be the rationale to mark it failed? :)

[Andy Doan (doanac)]

On 02/01, Fathi Boudra wrote:
> On 1 February 2013 06:20, Andy Doan <email address hidden> wrote:
> > The output says:
> > cpufreq_05.4: combine governors not supported... skip
> >
> > why would we mark this as passed when its not being tested? I'm not
> > familiar with this test suite, but that seems like its doing exactly
> > what it should be.
>
> it makes more sense to mark passed than failed.
>
> skip means ignore the test (current reason is that combine governor
> isn't supported).
> the test has been skipped, it's the expected behavior = passed.
>
> what could be the rationale to mark it failed? :)

Who's suggesting to mark it failed? I'm saying we leave it skipped.
Skipped is a valid test result, why not use it.

[Fathi Boudra (fboudra)]

On 1 February 2013 18:28, Andy Doan <email address hidden> wrote:
> On 02/01, Fathi Boudra wrote:
>> On 1 February 2013 06:20, Andy Doan <email address hidden> wrote:
>> > The output says:
>> > cpufreq_05.4: combine governors not supported... skip
>> >
>> > why would we mark this as passed when its not being tested? I'm not
>> > familiar with this test suite, but that seems like its doing exactly
>> > what it should be.
>>
>> it makes more sense to mark passed than failed.
>>
>> skip means ignore the test (current reason is that combine governor
>> isn't supported).
>> the test has been skipped, it's the expected behavior = passed.
>>
>> what could be the rationale to mark it failed? :)
>
> Who's suggesting to mark it failed? I'm saying we leave it skipped.
> Skipped is a valid test result, why not use it.

misread. I thought it was marked failed.
If it's marked skipped, indeed I agree and bug is invalid.

[Michael Hudson-Doyle (mwhudson)]

Fathi Boudra <email address hidden> writes:

> misread. I thought it was marked failed.
> If it's marked skipped, indeed I agree and bug is invalid.

I think the problem might be that a skipped test turns the corresponding
cell on the image report red. We should probably fix that.

Cheers,
mwh

[Amit Kucheria (amitk)]

On 13 Feb 03, Michael Hudson-Doyle wrote:
>
> I think the problem might be that a skipped test turns the corresponding
> cell on the image report red. We should probably fix that.

What are the valid return results of a LAVA test? PASS, FAIL and SKIP? Where is
this documented?

Do you need us to change something?

Regards,
Amit

[Botao (botao-sun)]

@Michael, Yes, we'd better to fix the cell from red to green if there is something only skipped in test suite.

[Andy Doan (doanac)]

On 02/04, Amit Kucheria wrote:
> What are the valid return results of a LAVA test? PASS, FAIL and SKIP? Where is
> this documented?

Its sort of buried here:

 http://lava-test.readthedocs.org/en/latest/usage.html

they are pass, fail, skip, unknown

> Do you need us to change something?

nope

[Antonio Terceiro (terceiro)]

Hi Senthil,

I don't think that that changing the icon is a valid fix for this. What Botao is requesting is to change the cell from red to green in the image reports page, e.g. http://validation.linaro.org/lava-server/dashboard/image-reports/linux-linaro-tracking-panda-es

There you will see that in the pwrmgmt row, the columns in which there was 1 skipped test and all other tests passed, the cell is red (and should be green, because no test failed).

I would say please revert the icon change, and fix the image report screen instead. I am setting this bug back to "In Progress".

[Antonio Terceiro (terceiro)]

Actually, I sew now that this was already fixed by Andy in rev 394, but it seems he forgot to mark the bug as fixed. So, you just need to revert the icon change.

[Senthil Kumaran S (stylesen)]

Hi Antonio,

On Wednesday 20 March 2013 07:52 PM, Antonio Terceiro wrote:
> Actually, I sew now that this was already fixed by Andy in rev 394, but
> it seems he forgot to mark the bug as fixed. So, you just need to revert
> the icon change.

I just reverted my change.

Thank You.
--
Senthil Kumaran S
http://www.stylesen.org/
http://www.sasenthilkumaran.com/