checkbox-qt issues running on nonstandard desktops

[Akkana Peck]

I'm not sure if ubuntu-friendly cares about collecting information from systems that aren't running the standard Unity/Gnome desktop, but in case you do, I encountered some minor problems trying to run it on my relatively lightweight Openbox setup:

- Missing package dependency on python-dateutils (that's a real bug, so I reported it separately).

- The automatic SD card test didn't work, just sat there forever. I guess it was waiting for some kind of automount service to do something with the card. It didn't time out after 10 seconds as it said it would, either. After a minute or two of waiting to see if it would do something, I gave up and clicked Yes (since Ubuntu does handle the SD card reader hardware, hooray).

- Same for the automatic USB stick test -- nothing happened after I inserted the stick, so after a while I clicked Yes.

- The wireless test was greek to me since it's all based on wording that presumably appears on the standard desktop. I wasn't sure exactly what was being tested. I clicked Yes since wireless more or less works (though it's flaky and seldom connects on the first try, and I have to retry several times no matter what software package I use to make the connection).

- When I reviewed the results before uploading, I discovered that a lot of tests had failed because of missing packages like alsa-base. I would have been happy to install these packages if I'd known they were needed, and re-run the tests, but there's no way to know about the failures until the whole suite of tests is finished. Maybe an optional status window I could look at so I could see when things were failing due to something I could fix?

- At the end, I guess (hope) it sent the information correctly, but there wasn't any feedback, so I'm not sure and have no idea how to find out. It would be very helpful if it printed "Uploading results ..." and "Results uploaded successfully!" or something like that. In general, there wasn't much status info about when it was busy vs. when I needed to do something. (And the status bar, dark blue on dark grey, is almost invisible even when it's present.)

- After a few minutes I gave up waiting and clicked the windowmanager 'x' to close the window. At exit, it printed
/usr/bin/checkbox-qt: line 26: 6085 I/O possible python $CHECKBOX_SHARE/run "$@" $CHECKBOX_SHARE/configs/$(basename $0).ini
[2] + exit 157 checkbox-qt

- General UI comment: I found it confusing trying to guess when I was supposed to click Yes or No versus when I was supposed to click Next to skip a test (e.g. firewire, which this laptop doesn't have), especially since Next is in a completely different place from Yes and No. It would be helpful if tests gave suggestions like "If you don't have firewire, please click Next."

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: checkbox-qt 0.13.7
ProcVersionSignature: Ubuntu 3.2.0-26.41-generic 3.2.19
Uname: Linux 3.2.0-26-generic i686
ApportVersion: 2.0.1-0ubuntu11
Architecture: i386
CheckboxSubmission: 59f5c4278a96c566305a5aafb84ff28a
CheckboxSystem: d00f84de8a555815fa1c4660280da308
Date: Wed Jul 25 13:17:53 2012
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Beta i386 (20110413)
ProcEnviron:
 TERM=xterm
 LC_COLLATE=C
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/usr/bin/zsh
SourcePackage: checkbox
UpgradeStatus: Upgraded to precise on 2012-05-07 (79 days ago)

[Daniel Manrique (roadmr)]

Hello,

Thanks for taking the time to make this detailed report.

The dependency on python-dateutil is fixed in the development version of checkbox, I think I marked that bug as Fix Released.

SD and USB tests depend on dbus, and I noticed some related errors in your log. Do you know if your system runs dbus? this is something we'd have to investigate.

We've received many requests for some way to show status and indication of why a test failed or wasn't executed, this is something we're working on for a future version.

We're also working on providing better results upload feedback and a more responsive UI in general, including addressing the confusion from the yes/no/next buttons; you're only supposed to use Next when you want to skip a test, but this is not as clear as we would have wanted it to be.

We do care about results from ubuntu flavors, however it's been difficult to accomodate the many different package combinations and ways to access hardware that these flavors provide. We've done some work to ensure that things work relatively well on Kubuntu but, as your case proves, that's not enough. However this is difficult to achieve if the set of packages checkbox needs to interact with is not consistent. And we can't make them all checkbox dependencies because that'd potentially mess up people's systems (e.g. Kubuntu users complaining because checkbox tried to bring in some Gtk packages). I think the best thing to do would be to at least ensure that checkbox informs the user when the system doesn't have an expected package, and fixing the parts that fail even in the presence of the expected package.

I'm marking incomplete pending your confirmation on the dbus issue, so we can look into that and get it fixed so the tests behave as described.

Thanks!

[Launchpad Janitor (janitor)]

[Expired for checkbox (Ubuntu) because there has been no activity for 60 days.]

[Akkana Peck (akkzilla)]

Sorry, I never got the mail from launchpad, so I didn't know you'd requested info, and unfortunately I don't have access to the machine I was using then. I'm pretty sure it had dbus, but probably not hal and other related services, and definitely no automount.

I just tried it on a different machine (running pangolin) and the SD card test worked, and a lot of the other problems were gone too. The biggest problems I had this time around was it wanting to test hardware I don't have (firewire, CF reader, an MMC card in the SD slot) and the "did a notification come up?" (I'm not running a desktop that does notifications, but I lied and said yes for things that I know were working).

[Akkana Peck (akkzilla)]

Oh, and that I couldn't view the results: Error: no "view" mailcap rules found for type "application/xml"
/usr/bin/xdg-open: 461: /usr/bin/xdg-open: x-www-browser: not found. I guess I need to make some symlinks or something. That message is never shown in the gui, btw, only on stdout ... and if it assumes I can read stdout, couldn't it print the results to stdout after it prints the error message? Or perhaps offer a URL so I can view the results online after uploading.

[Akkana Peck (akkzilla)]

One last comment then I promise I'll stop. :-)
Somehow it loaded the final report into my running firefox despite /etc/alternatives/x-www-browser pointing to a nonexistent place.

On this system (which I think is an upgrade ->ocelot->pangolin of the same system I was using for the original natty report, though running on different hardware now), here's what ps shows for dbus:
102 837 0.0 0.0 3468 1248 ? Ss 11:43 0:08 dbus-daemon --system --fork --activation=upstart
akkana 2653 0.0 0.0 3920 496 tty1 S 12:57 0:00 dbus-launch --autolaunch cda680566f66b6ed2b1ca16f000003a3 --binary-syntax --close-stderr
akkana 2654 0.0 0.0 3380 1068 ? Ss 12:57 0:00 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session

It looks like it failed all the sound tests because alsa-base isn't installed. Sound via alsa works fine on this system, though -- I use alsa-utils, sox python-alsa and players like vlc and mplayer, and flash inside firefox.

Looks like the SD card test failed and the stack trace boils down to
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UDisks was not provided by any .service files
so I guess the problem is not dbus but udisks. Same failure for the usb/storage-automated test.

It also got a runtime divide by 0 error in the memory tests: Runtime error (func=(main), adr=17): Divide by zero /usr/share/checkbox/scripts/memory_compare: line 23: -3: substring expression < 0

[Daniel Manrique (roadmr)]

Thanks for your comments! They are appreciated, I'm just very surprised that you have so many failures that nobody else has.

Some of the tests whose hardware you don't have are shown because said hardware is not detectable (either that or we haven't gotten around to autodetecting it; I think it'd be possible with firewire). So checkbox shows you the test but if you don't have the hardware you are expected to just skip the test. This is normal behavior.

So your system lacks udisks, it also lacks alsa-base, and x-www-browser doesn't point to a usable browser. This is not normal for a default Ubuntu installation. You also mention "I'm not running a desktop that does notifications", which further points to the fact that your environment is somewhat idiosyncratic.

Developing checkbox scripts that work under every imaginable user configuration and combination would be extremely time-consuming, so it's probably not something we would be able to do (patches welcome, though!). Perhaps a reasonable solution would be being clearer in test requirements so they don't run if you don't have, say, udisks or alsa-base (hey, not running beats running and crashing!).

I'll set this as Triaged since we know what to do (both this comment and #2 are a good indication of that: making sure that package requirements are as explicit as possible so that behavior doesn't seem too erratic on systems not running unity and/or gnome).

[Akkana Peck (akkzilla)]

Understood -- I know it's a lot more work to develop careful error correction. Though it might be fairly easy and worthwhile to throw a specific exception when packages aren't installed, and if that exception happens, mark those tests skipped rather than failed.

In the UI, it would be helpful to the user to know it's okay to click Skip. When I started, it showed me a list of tests, and I tried to turn off tests I knew were irrelevant (like firewire), but it gave me a warning that if I turned off any tests at all, my results wouldn't be uploaded. So I was very reluctant to click Skip, and even now am not sure whether Skipping a few tests invalidated my results.

[Daniel Manrique (roadmr)]

Ubuntu 14.04 and newer ship the plainbox-based testing stack, so I set this as won't fix for Ubuntu/checkbox. Setting as triaged and moving from the old upstream checkbox project to the new plainbox-provider-checkbox project where the jobs live, since most of the work here is being very explicit with test and package dependencies.

[Zygmunt Krynicki (zyga)]

I'm marking this as WONT FIX but please reopen if this is actively hampering anyone's work. My goal is to limit the number of open bugs to get a better idea as to what is really important.

Remember that you can always escalate bugs by contacting us in #checkbox on freenode (or #cert-infra in the internal IRC) or by responding in bugs directly.

NOTE: I welcome patches that just correct dependencies. Having one mega-bug with nobody on our side (people hacking on checkbox for Canonical) actively fixing this (ENOTIME) doesn't help the situation.