Submissions fail due to bad JSON in tarball, seen w/ 14.04 and 18.04

[Rod Smith]

Submissions of test results from runs under Ubuntu 14.04 are failing for me. I've run into this in two cases:

Case 1:
-------

I'm trying to submit two certification-run tarballs using "checkbox-cli submit". The first attempt succeeds:

$ checkbox-cli submit 4BkxQnEwfavu2efosrjnkQ ~/Downloads/submission_2018-03-08T10.20.16.848663.tar.xz Successfully sent, submission status at https://certification.canonical.com/submissions/status/e8a0b44f-0427-4bf6-97f0-16348eed3391

The second attempt fails with a rather unhelpful error message:

$ checkbox-cli submit 4BkxQnEwfavu2efosrjnkQ ~/Downloads/submission_2018-03-12T21.52.51.149103.tar.xz
400 Client Error: BAD REQUEST for url: https://certification.canonical.com/api/v1/submission/4BkxQnEwfavu2efosrjnkQ/

I'm attaching the problem tarball to this bug report. Note that this tarball was sent to me by a partner. It appears to be valid, but I don't know the details of how it was generated, except that it's for a test run under Ubuntu 14.04.

$ apt-cache policy checkbox-ng
checkbox-ng:
  Installed: 1.0.0~ppa~ubuntu16.04.1
  Candidate: 1.0.0~ppa~ubuntu16.04.1
  Version table:
 *** 1.0.0~ppa~ubuntu16.04.1 500
        500 http://ppa.launchpad.net/hardware-certification/public/ubuntu xenial/main amd64 Packages
        500 http://ppa.launchpad.net/hardware-certification/public/ubuntu xenial/main i386 Packages
        100 /var/lib/dpkg/status
     0.23-2 500
        500 http://nessus.rodsbooks.com/ubuntu xenial/main amd64 Packages
        500 http://nessus.rodsbooks.com/ubuntu xenial/main i386 Packages

Case 2:
-------

To further investigate the above, I deployed a node with Ubuntu 14.04 and ran a few small tests on it using canonical-certification-server. The tests completed, but at the end, I was unable to submit them:

Do you want to submit 'upload to certification' report?
  y => yes
  n => no
[yn]: y
WARNING:checkbox-ng.launcher.subcommands:Problem occured when submitting upload to certification report: 400 Client Error: BAD REQUEST
Do you want to retry?
  y => yes
  n => no
[yn]: y
WARNING:checkbox-ng.launcher.subcommands:Problem occured when submitting upload to certification report: 400 Client Error: BAD REQUEST
Do you want to retry?
  y => yes
  n => no
[yn]: n

I tried running "checkbox-cli submit" on the submission_*.tar.xz file, but it failed in the same way as described in Case 1. This was true both when I tried to submit from the SUT and from my laptop, after transferring the submission_*.tar.xz file to it.

[Jeff Lane  (bladernr)]

Setting this to Critical and confirming it as this is being reported from multiple sources and it is currently blocking the ability to submit results for active certification work.

I'm also getting pinged on this from a partner who is trying to submit certification results as well. Both they, and I get the 400 errors from C3 when trying to submit the 14.04 tarball, however, the 16.04 results were posted fine.

I have also heard that there were issues submitting 18.04 results too, but I haven't been able to confirm that one yet.

[Sheila Miguez (codersquid)]

Hi, I've reproduced this in staging, and the 400 is coming back because of bad json. In the submission.json, the dkms info is invalid: "dkms_info": ,

[Sheila Miguez (codersquid)]

It would help to print the contents of the message that comes back from c3. It looks like this: {"code": "invalid_json", "message": "Submission contains an invalid json file", "result": false, "submission_id": "JrsevR9y3WLimeF2GNmXjV"}

[Jeff Lane  (bladernr)]

I thought we fixed that ages ago? blank dkms_info in the json has been the cause of issues before.

[Sheila Miguez (codersquid)]

I really thought so too. I wonder if the partner is using an outdated version?

[Rod Smith (rodsmith)]

As per the original bug report (case 2), I reproduced this with a fresh deployment of Ubuntu 14.04; and if I read Jeff's comments, he's done the same. Thus, I don't think this is the result of any weirdness on just one or two systems' configurations. That said, it could well be that Ubuntu 14.04 is setting up some old or broken package; it's just not a site-specific thing, or caused by somebody deviating from our instructions and setting things up weirdly.

[Sylvain Pineau (sylvain-pineau)]

I've checked 0.42 tarballs and https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1695912 is targeted for 0.43. I'll release only this provider to stable to fix the problem. Just have to check that I can push it w/o pulling other dependencies too. I'll keep the bug updated asap.

[Jeff Lane  (bladernr)]

Here's a tarball from 18.04 that also has this issue:

ubuntu@s1lp10-jefflane:~$ canonical-certification-submit a00D000000feLCJIA2 submission_2018-03-20T13.40.52.534933.tar.xz
400 Client Error: BAD REQUEST for url: https://certification.canonical.com/api/v1/submission/a00D000000feLCJIA2/

[Jeff Lane  (bladernr)]

Hi Sylvain,

Any update on pushing 0.43 yet?

[Jeff Lane  (bladernr)]

sylvain, just FYI, I presume the fix in https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1695912 from your comment #8 above is currently in the daily builds. I've just tested a quick run with the daily build and this error is still present.

ubuntu@xwing:~$ canonical-certification-submit a00D000000gYQEbIAO .local/share/checkbox-ng/submission_2018-03-27T21.42.04.695557.tar.xz
400 Client Error: BAD REQUEST
ubuntu@xwing:~$ apt-cache policy plainbox-provider-checkbox
plainbox-provider-checkbox:
  Installed: 0.42.0+git201803210840+pkg77~ubuntu14.04.1
  Candidate: 0.42.0+git201803210840+pkg77~ubuntu14.04.1
  Version table:
 *** 0.42.0+git201803210840+pkg77~ubuntu14.04.1 0
        500 http://ppa.launchpad.net/checkbox-dev/ppa/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.42.0~ppa~ubuntu14.04.1 0
        500 http://ppa.launchpad.net/hardware-certification/public/ubuntu/ trusty/main amd64 Packages
     0.4-1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
ubuntu@xwing:~$

[Sylvain Pineau (sylvain-pineau)]

@Jeff, I'm really interested to know why on 18.04 with dev ppa version it fails the same way as on 14.04. If you can easily reproduce the issue, could you please run:

$ checkbox-cli run com.canonical.certification::dkms_info_json