can not find 9 x TFAIL in the ltp stdout log

[Sam Lin]

About Itp test of LAVA,
I couldn't find 9 x TFAIL in the ltp stdout log, just one

fork13: relocation error: fork13: symbol exit, version GLIBC_2.4 not defined in file libc.so.6 with link time reference
fork13 1 TFAIL : Unexpected exit status 7f, iteration 11890.

[Test environment]
- Lava Test Shell: git://git.linaro.org/qa/test-definitions.git
(commit :11e04aab4494b76e9bc451038636782440341338)
- kernel version: 3.10.0-rc5+
- rootfs: linaro-quantal-nano-20130422-342

[Milosz Wasilewski (mwasilew)]

Could you post the lint to LAVA result bundle?

[Sam Lin (sam-lin)]

Sorry, you can not access to our LAVA server because we built it on our internal network.
I also take new photos about LAVA and add attachment.
Do you require other information or log?

By the way, could you help me add "<email address hidden>" address to CC list in this bug?
Thanks.

[Our LAVA Environment]
- LAVA Server version 0.20
- Extension Dashboard version 0.29
- Extension Projects version deprecated
- Extension Scheduler version 0.28

[Milosz Wasilewski (mwasilew)]

OK,
Now I can see what the problem is :)
I guess the only way I can help is to reproduce the problem on some LAVA server I have access to. In order to do it I will need:
 - job definition (.json file)
 - software binaries (hwpack should be enough as I can download the rootfs you mentioned earlier)

I'm also not able to locate the commit 11e04aab4494b76e9bc451038636782440341338 in the git://git.linaro.org/qa/test-definitions.git

On the other hand I checked the current HEAD and the ltp.yaml considers TFAIL and TBROK as failed cases. So the remaining 8 failures seem to be TBROK
cat stdout.log | grep TBROK | wc -l

Please confirm if that solves the mystery so I can close the bug.

[Sam Lin (sam-lin)]

CC Andy, Doris
I attached our job (job_74.json)

Hi Andy,
Could you help to confirm the comment feedback from Milosz Wasilewski?
(BugID: #1192437: https://bugs.launchpad.net/linaro-qa/+bug/1192437)

Thanks.

Best Regards,
Sam

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Milosz Wasilewski
Sent: Wednesday, June 19, 2013 6:45 PM
To: Sam.Lin
Subject: [Bug 1192437] Re: can not find 9 x TFAIL in the ltp stdout log

OK,
Now I can see what the problem is :)
I guess the only way I can help is to reproduce the problem on some LAVA server I have access to. In order to do it I will need:
 - job definition (.json file)
 - software binaries (hwpack should be enough as I can download the rootfs you mentioned earlier)

I'm also not able to locate the commit
11e04aab4494b76e9bc451038636782440341338 in the git://git.linaro.org/qa
/test-definitions.git

On the other hand I checked the current HEAD and the ltp.yaml considers TFAIL and TBROK as failed cases. So the remaining 8 failures seem to be TBROK
cat stdout.log | grep TBROK | wc -l

Please confirm if that solves the mystery so I can close the bug.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1192437

Title:
  can not find 9 x TFAIL in the ltp stdout log

Status in Linaro QA:
  Incomplete

Bug description:
  About Itp test of LAVA,
  I couldn't find 9 x TFAIL in the ltp stdout log, just one

  fork13: relocation error: fork13: symbol exit, version GLIBC_2.4 not defined in file libc.so.6 with link time reference
  fork13 1 TFAIL : Unexpected exit status 7f, iteration 11890.

  [Test environment]
  - Lava Test Shell: git://git.linaro.org/qa/test-definitions.git
  (commit :11e04aab4494b76e9bc451038636782440341338)
  - kernel version: 3.10.0-rc5+
  - rootfs: linaro-quantal-nano-20130422-342

To manage notifications about this bug go to:
https://bugs.launchpad.net/linaro-qa/+bug/1192437/+subscriptions

[warmcat (andy-warmcat)]

@Milosz ... not so fast mate ^^

1) Why do these tests output something other than "TFAIL" when a test has failed? In the first case we're interested in pass / fail, "TBROK" is an elaboration we don't need.

2) What is the meaning of the original puzzle these tests are generating:

"fork13: relocation error: fork13: symbol exit, version GLIBC_2.4 not defined in file libc.so.6 with link time reference
fork13 1 TFAIL : Unexpected exit status 7f, iteration 11890."

this is telling us something libc, or it's telling us something about our kernel? iteration 11890 means it was working and blew chunks when stressed by repetition, or what?

3) Likewise, can you illuminate the meaning of:

 3a) getrusage03 8 TBROK : unexpected signal 15 received (pid = 13840).
getrusage03 9 TBROK : Remaining cases broken

 3b) getrusage03_child 0 TINFO : child allocate 500MB
getrusage03_child 1 TBROK : unexpected signal 15 received (pid = 13842).
getrusage03_child 2 TBROK : Remaining cases broken

 3c) setrlimit01 3 TPASS : RLIMIT_NPROC functionality is correct
setrlimit01 4 TBROK : caught unexpected signal: 15
setrlimit01 5 TBROK : Remaining cases broken

These are currently expected behaviours of the test suite, or something only the Fujitsu LT kernel is generating?

[Milosz Wasilewski (mwasilew)]

@andy

1) I'm not author of LTP, so I don't know the reasons for failure. Sam was asking about the missing failures (in LAVA) and these were found. LAVA result has to be one of: pass/fail/skip/unknown. If you're not happy with mapping TBROK to fail, you can change it in the yaml file.

2) I checked other board results in LAVA. fork13 passes:
fork13 1 TPASS : 1000000 pids forked, all passed
This job was running other rootfs, so I can't tell if the problem was in the libc or kernel. I will submit a job with linaro-quantal-nano-20130422-342 to confirm that.

3a) I'm getting identical results.
3b) I'm getting identical results.
3c) I'm getting different results here. No test cases report TBROK
setrlimit01 2 TPASS : RLIMIT_FSIZE test PASSED
setrlimit01 3 TPASS : RLIMIT_NPROC functionality is correct
setrlimit01 4 TPASS : RLIMIT_CORE functionality is correct

BUT!
The results I'm referring to come from different LTP version than your results: LTP Version: 20120614.
I'm running the test now with the rootfs you use and with the LTP version you use and post comment here once the test is complete.

[warmcat (andy-warmcat)]

Hi Milosz -

Sam sent me the log for ltp originally and I grepped it for "TFAIL", "TBROK" doesn't have any meaning for me.

Presumably someone in Linaro wrote the yaml for LTP, we should suggest to them to get rid of TBROK and replace with TFAIL. If you don't know who that is I'll open another bug.

Thanks for looking into fork13 and the other problems we're curious what your results are.

[Naresh Kamboju (naresh-kamboju)]

Here is the LTP test definitions,

https://git.linaro.org/gitweb?p=qa/test-definitions.git;a=blob_plain;f=ubuntu/ltp.yaml

If you are interested please submit a patch and send to "<email address hidden>" or attach to this bug.

you can find more test definitions in the git
https://git.linaro.org/gitweb?p=qa/test-definitions.git;a=summary

[Milosz Wasilewski (mwasilew)]

@andy

Both TBROK and TFAIL are there in the yaml file:
    fixupdict:
        TBROK: fail
        TCONF: skip
        TFAIL: fail
        TINFO: skip
        TPASS: pass
        TWARN: unknown

If TBROK doesn't necessarily mean 'failed' maybe it should be mapped to 'unknown'?

I tried to run ltp several times in LAVA last night. All runs failed because of various reasons. In one case I observed compiler segfault (Naresh is checking that). In other cases there were network failures and LAVA didn't manage to install dependencies. So for the moment I don't have any other LTP results to compare. I will try again today.

[Naresh Kamboju (naresh-kamboju)]

Regarding the headlines of the bug.
We are considering TBROK and failure in lava results ( and showing it in red color) along with TFAIL. so here TFAIL and TBROK are considered as failure. OTOH, we have started preparing a patch on lava to display all LTP notification/Legends as is it. one would see TBROK as TBROK soon.

The LTP results are different when i compare with LTP running on Vexpress TC2 board with your report. As you know the LTP results would vary from kernel version and target.

We have latest LTP results for your information,
http://validation.linaro.org/dashboard/streams/private/team/linaro/ci-LT-ARM-integration-ci-vexpress/bundles/636d6d87b01b1f722401414a03aa4e707b55d167/

1. fork13 was reported as failure in your case but it is passed in our testing.
fork13 1 TPASS : 1000000 pids forked, all passed

2. following the TBROK from our LTP results.

getrusage03 8 TBROK : unexpected signal 15 received (pid = 32432).
getrusage03 9 TBROK : Remaining cases broken

ioctl03 1 TBROK : opening /dev/net/tun failed: errno=ENOENT(2): No such file or directory
ioctl03 2 TBROK : Remaining cases broken

stdout.log file has been attached to this bug for your reference.

If you need more info please lets us know.
-Naresh

[Milosz Wasilewski (mwasilew)]

Sam, Andy,
Is there still anything unclear about the LTP test results? If not I will close this bug.

[Sam Lin (sam-lin)]

Hi Milosz,
I think we can closed it. Thanks.

Best regards,
Sam

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Milosz Wasilewski
Sent: Monday, February 03, 2014 7:49 PM
To: Sam.Lin
Subject: [Bug 1192437] Re: can not find 9 x TFAIL in the ltp stdout log

Sam, Andy,
Is there still anything unclear about the LTP test results? If not I will close this bug.

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1192437

Title:
  can not find 9 x TFAIL in the ltp stdout log

Status in Linaro QA:
  Incomplete

Bug description:
  About Itp test of LAVA,
  I couldn't find 9 x TFAIL in the ltp stdout log, just one

  fork13: relocation error: fork13: symbol exit, version GLIBC_2.4 not defined in file libc.so.6 with link time reference
  fork13 1 TFAIL : Unexpected exit status 7f, iteration 11890.

  [Test environment]
  - Lava Test Shell: git://git.linaro.org/qa/test-definitions.git
  (commit :11e04aab4494b76e9bc451038636782440341338)
  - kernel version: 3.10.0-rc5+
  - rootfs: linaro-quantal-nano-20130422-342

To manage notifications about this bug go to:
https://bugs.launchpad.net/linaro-qa/+bug/1192437/+subscriptions

[Milosz Wasilewski (mwasilew)]

Thanks, closing.