testrunner failing with "bash: run_test.py: command not found"

[Ursula Junque]

Test runner step is failing after you can see a "bash: run_test.py: command not found" on /var/log/upstart/ci_airline_tr_rabbit_worker.log (https://pastebin.canonical.com/112148/). You can see two tickets failed the same way.

{{{
INFO:image_test:The test bed is ready
INFO:image_test:Testing libpng3
bash: run_test.py: command not found
INFO:image_test:Saving subunit results for libpng3
INFO:image_test:Saving adt-run log for libpng3
INFO:image_test:Saving adt-run summary for libpng3
INFO:image_test:Test completed for ticket 8
}}}

Also, testbed cloud-init fails: (testbed-cloud-init.log: https://pastebin.canonical.com/112162/)

{{{
Cloud-init v. 0.7.3 running 'modules:final' at Sat, 21 Jun 2014 02:17:50 +0000. Up 604.52 seconds.
-su: /home/ubuntu/uci-vms_uploads: No such file or directory
2014-06-21 02:17:50,416 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [127]
}}}

According to vila both errors can be related, and also related to saucy as the tests run using a precise image and they don't fail with that.
{{{
      vila | Ursinha: I'm surprised, the test I added with that bug should fail if run_test.py not being there...
      vila | Ursinha: hmm, except that test is incomplete :-/ So indeed I may have miss something with my semi-manual testing, let me re-check
   Ursinha | vila: so that might mean that run_test is in the wrong path, for example, and would never work?
      vila | Ursinha: more probably it's not there at all for reasons yet unknown
      vila | Ursinha: an additional bug is that I don't catch that cloud-init failure
   Ursinha | vila: I wonder if the HOME change caused side effects?
      vila | Ursinha: I'd be more inclined to suspect some saucy related issue, the side-effect of the HOME change is *really* limited
      vila | Ursinha: the tests use a precise image because there is no saucy image available :-/
      vila | Ursinha: and the incomplete test shouldn't matter here as there is a complete one with a... precise image that covers running the package test
}}}

[Vincent Ladeuil (vila)]

The testbed silently fails to install run_test.py and test_runner.tstrun.tests.test_worker.TestWorkerHandleRequest.test_handle_request has been silently masking the bug by checking the existence of files and not their content (the files are created with an error message inside them in this case).

And indeed the content of the files helped in the diagnosis :-}

[Vincent Ladeuil (vila)]

test_runner.tstrun.tests.test_testbed.TestTestbed.test_create_usable_testbed checks that the run_test.py file is properly installed so this seems to be saucy-specific.

[Ursula Junque (ursinha)]

I think the fix caused another problem, I got the failure below in testbed.setup() twice (deployed r612):

ERROR:image_test:The testbed setup for 1 failed
Traceback (most recent call last):
  File "./test_runner/tstrun/run_worker.py", line 138, in handle_request
    test_bed.setup()
  File "/srv/ci-airline-tr-rabbit-worker/test_runner/tstrun/testbed.py", line 190, in setup
    self.ensure_ssh_key_is_available()
  File "/srv/ci-airline-tr-rabbit-worker/test_runner/tstrun/testbed.py", line 157, in ensure_ssh_key_is_available
    os.makedirs(os.path.dirname(self.test_bed_key_path))
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/root/.ssh'