testbed ensure_ssh_key_is_available() fails with File exists: '/root/.ssh'

[Vincent Ladeuil]

Reported by Ursula after the fix for bug #133323 landed:

I think the fix (for bug #133323) caused another problem, I got the failure below in testbed.setup() twice (deployed r612):

ERROR:image_test:The testbed setup for 1 failed
Traceback (most recent call last):
  File "./test_runner/tstrun/run_worker.py", line 138, in handle_request
    test_bed.setup()
  File "/srv/ci-airline-tr-rabbit-worker/test_runner/tstrun/testbed.py", line 190, in setup
    self.ensure_ssh_key_is_available()
  File "/srv/ci-airline-tr-rabbit-worker/test_runner/tstrun/testbed.py", line 157, in ensure_ssh_key_is_available
    os.makedirs(os.path.dirname(self.test_bed_key_path))
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/root/.ssh'

[Vincent Ladeuil (vila)]

The fix for bug #133323 doesn't change the working dir, something is going on.

The fix for bug #1331989 was about the worker running without HOME being set and now we have '~root' coming into play, pretty weird.

[Vincent Ladeuil (vila)]

Reproduced by adding a breakpoint in test_process_ticket() as mentioned in the comment and running:

$ ./run-tests tests.test_test_runner.SmokeTestTestRunner.test_worker_running

and then:

$ juju ssh ci-airline-tr-rabbit-worker/0

To investigate on the worker itself.

Note that inspecting the swift artifacts produced is also confirming the issue:

- image_test.output.log: the traceback mentioned in the report,
- libpng.log:

 libpng.log couldn't be copied from testbed 15.125.96.199:
error: ssh: connect to host 15.125.96.199 port 22: No route to host

I.e. the ssh setup is broken.

[Vincent Ladeuil (vila)]

Addin a log.info() call in handle_request gives:

INFO:image_test:Running in /srv/ci-airline-tr-rabbit-worker with HOME: /root

So something changed since bug #1331989 we now have a HOME...

[Vincent Ladeuil (vila)]

Got it, the regression was introduced in revno 602 which makes the upstart job run via su which has the side-effect of setting HOME by running as root instead of setting uid and gid.