rendercheck test will pass without rendercheck installed

[Jeffrey Chang]

checkbox 0.15+bzr1867+201301070448~precise1
checkbox-certification 0.15+bzr562+201212191001~precise1
checkbox-ihv 0.3~ppa4.12.04
checkbox-ihv-graphics 0.3~ppa4.12.04

I ran rendercheck test without rendercheck installed,
The result is PASS with note
"Rendercheck completed. But there are errors. Please check /home/u/.checkbox/rendercheck-detail for details".
But the log file was not even exist. So no further info for tester to check.

[Jeff Lane (bladernr)]

The job for rendercheck should only run if the x11-apps package is installed.

Can you verify that rendercheck is indeed NOT installed on your system?

Also, can you check and see if rendercheck is no longer part of x11-apps but perhaps moved to a different package.

Finally, do you have submitted results showing this issue (the results will also contain a list of installed packages which is what I'm interested in here).

FYI:
plugin: shell
name: rendercheck/tests
requires:
 package.name == 'x11-apps'

[Daniel Manrique (roadmr)]

Hi!

See bug https://bugs.launchpad.net/checkbox-ihv/+bug/1103341, apparently the version of x11-apps on Precise does not include rendercheck. So the package being present does not guarantee rendercheck will be there.

The job is defined as follows:

rendercheck_test -d -o $CHECKBOX_DATA/rendercheck-results && echo
"Rendercheck tests completed successfully" || echo "Rendercheck completed, but
there are errors. Please see the log $CHECKBOX_DATA/rendercheck-results for
details"

So the message you're seeing indicates "rendercheck_test" exited with failure. I'd like to ask you to run "/usr/share/checkbox/scripts/rendercheck_test -d -o /tmp/test", and let us know what you see on-screen, maybe it's outputting something to stderr and this is being "eaten" by the command, that's why you don't see anything.

I suspect that, since rendercheck is not present, we're running into something similar to this: https://bugs.launchpad.net/checkbox/+bug/1102812, since when running rendercheck_test by hand, I had the same error as seen in that bug. If so, one thing to fix for this would be adding the "include errno" line to the script. However, the error is still logged to stderr, so we will probably need to update the command definition to output stderr to the log file as well.

Anyway, remains as incomplete pending Jeffrey's running of rendercheck_test.

[Jeff Lane (bladernr)]

Ahhh, I recall now... with Precise, you need to have the following PPA installed:
http://ppa.launchpad.net/canonical-hwe-team/piglit/ubuntu/

when we added this test for the original Bender sprint, it was done with the understanding that for Quantal this would work fine, but for Precise you'd need the piglit PPA to get rendercheck unless/until it was backported into Precise.

I would suggest getting the team to work on backporting rendercheck into x11-apps for precise.

[Jeff Lane (bladernr)]

wonder, would it be good enough to do something as simple as this:

which rendercheck && rendercheck_test <OPTS> && echo "Succeess message" || echo "Failure message"

Though that's a kludge workaround and I really don't like it. I would rather that HWE gets rendercheck backported into x11-apps

[Daniel Manrique (roadmr)]

rendercheck_test properly tests for the binary and reports its absence if that's the case. Problem is, it gets reported to stderr, which is why Jeffrey was unable to find a message.

rendercheck_test <OPTS> 2>&1 && echo "success" || echo "failure" would at least capture stderr in the log file, easing debugging of this kind of issue. Beware, this bears testing, to quote Knuth "I have only proved it correct, not tried it".

I agree that the final solution is to ensure rendercheck is present. For checkbox-ihv purposes, we can feed a custom PPA for it to fetch packages, and we'd have to ensure that proper dependencies are in place so that installing checkbox-ihv-graphics pulls an updated version of x11-apps from either the PPA or archive.

[Jeff Lane (bladernr)]

So really, the root problem lies here:

try:
            # Let's dump the output into file as it can be very large
            # and we don't want to store it in memory
            process = Popen(full_command, stdout=temp_file,
                            universal_newlines=True)
        except OSError as exc:
            if exc.errno == errno.ENOENT:
                logging.error('Error: please make sure that rendercheck '
                              'is installed.')
                exit(1)
            else:
                raise

Problem is, if I read that correctly, assuming Jeffrey's case is correct and rendercheck is not present, it "should" throw ENOENT (doesn't exist) and log a message and exit with a 1.

So logging.error, IIRC logs to wherever the logger is suppose to log, file or stdout, logging.error isn't internally redirected to stderr.

Second, it should exit with a 1, which should trigger the fail message, not a pass based on this command in the job description:

command: rendercheck_test -d -o $CHECKBOX_DATA/rendercheck-results && echo "Rendercheck tests completed successfully" || echo "Rendercheck completed, but there are errors. Please see the log $CHECKBOX_DATA/rendercheck-results for details"

which gives me this when I move the rendercheck tool:

bladernr@klaatu:~/development/checkbox$ which rendercheck
/usr/bin/rendercheck
bladernr@klaatu:~/development/checkbox$ sudo mv /usr/bin/rendercheck /usr/bin/not-the-rendercheck-youre-looking-for
[sudo] password for bladernr:
bladernr@klaatu:~/development/checkbox$ which rendercheck
bladernr@klaatu:~/development/checkbox$ ./scripts/rendercheck_test -d -o /tmp/rendercheck.log
Traceback (most recent call last):
  File "./scripts/rendercheck_test", line 134, in get_suites_list
    stderr=PIPE, universal_newlines=True)
  File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.2/subprocess.py", line 1361, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'rendercheck'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./scripts/rendercheck_test", line 255, in <module>
    main()
  File "./scripts/rendercheck_test", line 165, in main
    all_tests = RenderCheck().get_suites_list()
  File "./scripts/rendercheck_test", line 136, in get_suites_list
    if exc.errno == errno.ENOENT:
NameError: global name 'errno' is not defined

[Jeff Lane (bladernr)]

Which is very quickly fixed by importing errno which was not caught originally. Patch incoming.

bladernr@klaatu:~/development/checkbox$ ./scripts/rendercheck_test -d -o /tmp/rendercheck.log
ERROR:root:Error: please make sure that rendercheck is installed.

[Jeff Lane (bladernr)]

One last thing to fix, I think because the command as listed still will return an incorrect code.

bladernr@klaatu:~/development/1103343-rendercheck-test-fix$ ( true && echo "True" ) || ( echo "False" && false )
True
bladernr@klaatu:~/development/1103343-rendercheck-test-fix$ echo $?
0
bladernr@klaatu:~/development/1103343-rendercheck-test-fix$ ( false && echo "True" ) || ( echo "False" && false )
False
bladernr@klaatu:~/development/1103343-rendercheck-test-fix$ echo $?
1

so this should work:

( command && echo "Success" ) || ( echo "Failure" && false )

[Jeff Lane (bladernr)]

Yep.
bladernr@klaatu:~/development/1103343-rendercheck-test-fix$ ( rendercheck_test -d -o $CHECKBOX_DATA/rendercheck-results && echo "Rendercheck tests completed successfully" ) || ( echo "Error running rendercheck. Please see the log $CHECKBOX_DATA/rendercheck-results for details" && false )
rendercheck_test: command not found
Error running rendercheck. Please see the log /rendercheck-results for details
bladernr@klaatu:~/development/1103343-rendercheck-test-fix$ echo $?
1