checkbox does not recover well from a suspend crash

[David D Lowe]

Checkbox causes my machine to crash (solved only by a cold reboot). When I start Checkbox again, it asks me if I want to rerun the last test, or continue with the next. I selected continue, only to checkbox restart before the suspend tests, and then crash once the suspend tests were run again.

(It's worth noting, that suspend is indeed faulty, and causes my computer to crash, but this bug is about Checkbox, not about the bug with the suspend functionality.)

What I expected to happen:

- Before running the test, I expected some sort of prompt, that would allow me to skip the test. Suspend/hibernate tests have a high likelyhood of failing, so I think this should be required.
- Checkbox should save how far I got along in the tests more accurately.
- I'm not sure if this is the case or not, but the results should be uploaded before I reach the final tests. I am now dead set against running Checkbox, because I don't want to have to reboot my computer again. Ubuntu should be collecting stats on this, on how many people do not finish the tests and why.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: checkbox-gui 0.17.6-0ubuntu6
ProcVersionSignature: Ubuntu 3.13.0-24.47-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.1
Architecture: amd64
CurrentDesktop: Unity
Date: Wed May 14 14:15:27 2014
InstallationDate: Installed on 2014-05-14 (0 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: checkbox
UpgradeStatus: No upgrade log present (probably fresh install)

[Zygmunt Krynicki (zyga)]

Hey.

Thanks for reporting this problem. Please look at my answers below:

- Before running the test, I expected some sort of prompt, that would allow me to skip the test. Suspend/hibernate tests have a high likelyhood of failing, so I think this should be required.

That's an interesting idea. It is not implemented right now. We have a similar bug that explores this idea already https://bugs.launchpad.net/plainbox/+bug/1232029

- Checkbox should save how far I got along in the tests more accurately.

We do that. You should have seen a prompt if you want to 'resume', 'continue' or 'restart' (those are quite confusing) did you see that after restarting checkbox?

- I'm not sure if this is the case or not, but the results should be uploaded before I reach the final tests. I am now dead set against running Checkbox, because I don't want to have to reboot my computer again. Ubuntu should be collecting stats on this, on how many people do not finish the tests and why.

I don't think we need to upload results before every dangerous test. We try very hard to save everything on the filesystem before running each job so that if it crashes you should still be safe.

Given the second question I'm setting this to incomplete until you reply if you saw the resume screen or not

[David D Lowe (flimm)]

Thanks for your prompt response!

> That's an interesting idea. It is not implemented right now. We have a similar bug that explores this idea already https://bugs.launchpad.net/plainbox/+bug/1232029

That bug report looks good. Is plainbox the backend for checkbox? I'm unfamiliar with it.

> We do that. You should have seen a prompt if you want to 'resume', 'continue' or 'restart' (those are quite confusing) did you see that after restarting checkbox?

Yes, I did see that box. I clicked "continue", as I deliberately wanted to skip the suspend tests. It started a microphone test, if I recall correctly, and then ran the suspend tests again, causing my computer to crash. I'll upload a log later tonight if I get other things off my plate first. The bug isn't that the dialog failed to appear, the bug is that it failed to skip a test that had previously caused my computer to crash, even though I chose "continue" instead of "rerun last test".

> I don't think we need to upload results before every dangerous test. We try very hard to save everything on the filesystem before running each job so that if it crashes you should still be safe.

Yes, but how do you know how many users never even try to run Checkbox again? Just because the data is saved on disk doesn't mean it will get uploaded. How do you know how many users give up before completing the process? My intuition says that a significant proportion do. There's a vast amount of UX materiel out there that examines form completion rates on the web. On the web, this stuff is measured, and you would be surprised how even tiny things put people off from completing a form.

My proposal is simply to start measuring this by uploading every result as you go along, rather than assuming that the majority of users finish the process.

Given that users are more likely to finish the process if nothing crashes, the fact that data isn't being uploaded in increments means that the received data is skewed towards test passes.

Thinking about, this really is two bugs in one report, so let me know if I should file each one separately.

[Zygmunt Krynicki (zyga)]

On Wed, May 14, 2014 at 6:14 PM, David D Lowe <email address hidden>wrote:

> Thanks for your prompt response!
>
> > That's an interesting idea. It is not implemented right now. We have a
> similar bug that explores this idea already
> https://bugs.launchpad.net/plainbox/+bug/1232029
>
> That bug report looks good. Is plainbox the backend for checkbox? I'm
> unfamiliar with it.
>

Yes

PlainBox is also a standalone tool but it is more low-level and aimed at
generic testing rather than certification. It is also a development tool
for test authors. Have a look at
http://checkbox.readthedocs.org/en/latest/stack.html for details.

>
> > We do that. You should have seen a prompt if you want to 'resume',
> 'continue' or 'restart' (those are quite confusing) did you see that
> after restarting checkbox?
>
> Yes, I did see that box. I clicked "continue", as I deliberately wanted
> to skip the suspend tests. It started a microphone test, if I recall
> correctly, and then ran the suspend tests again, causing my computer to
>

IIRC there are many instances of suspend tests. Do you remember which
"whitelist" (test plan) did you select? If it really re-ran the suspend
test that is a new bug (though we heard about that case happening from time
to time but didn't manage to reproduce it). The gui is pretty buggy so if
you can try the same set of test with plainbox that would help to nail the
problem better (you want plainbox run -w /path/to/whitelist, make sure to
run --help to decide what to do with results up front)

> crash. I'll upload a log later tonight if I get other things off my
> plate first. The bug isn't that the dialog failed to appear, the bug is
> that it failed to skip a test that had previously caused my computer to
> crash, even though I chose "continue" instead of "rerun last test".
>

I understand.

>
> > I don't think we need to upload results before every dangerous test.
> We try very hard to save everything on the filesystem before running
> each job so that if it crashes you should still be safe.
>
> Yes, but how do you know how many users never even try to run Checkbox
> again? Just because the data is saved on disk doesn't mean it will get
> uploaded. How do you know how many users give up before completing the
> process? My intuition says that a significant proportion do. There's a
> vast amount of UX materiel out there that examines form completion rates
> on the web. On the web, this stuff is measured, and you would be
> surprised how even tiny things put people off from completing a form.
>

We don't measure that and I cannot estimate how many users will do that.
Uploading results (where?) before every test is a total overkill though.
The current checkbox user interface (checkbox-gui) has lots of design
issues but this is not a solution IMHO.

>
> My proposal is simply to start measuring this by uploading every result
> as you go along, rather than assuming that the majority of users finish
> the process.
>

This is not possible at this time. We don't have anyone that could be
dedicated to do such research, let alone maintain the infrastructure and
coding changes.

Still. If you want to contribute (maybe not by hacking...

[Launchpad Janitor (janitor)]

[Expired for checkbox (Ubuntu) because there has been no activity for 60 days.]