bootstrap fails because Permission denied (publickey)

[Curtis Hovey]

juju-core r2116 fails to bootstrap on canonistack and aws. The tests passed yesterday (r2113). We were not able to capture logs for the failed bootstrap. The issue may be in r2116 which introduced synchronous bootstrap.

2013-12-03 15:39:17 INFO juju.provider.openstack provider.go:733 started instance "1a217478-f6f9-44ca-beb6-a87834ffdc7b"
 - 1a217478-f6f9-44ca-beb6-a87834ffdc7b
Waiting for DNS name..............
 - 10.55.32.197
Attempting to connect to 10.55.32.197:22.....
2013-12-03 15:40:01 INFO juju.cloudinit.sshinit configure.go:24 Provisioning machine agent on ubuntu@10.55.32.197
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '10.55.32.197' (ECDSA) to the list of known hosts.
Permission denied (publickey).
Stopping instance...
2013-12-03 15:40:02 ERROR juju supercommand.go:282 exit status 255
Command '('juju', 'bootstrap', '-e', 'test-release-canonistack', '--constraints', 'mem=2G', '--show-log')' returned non-zero exit status 1
+ EXIT_STATUS=1

I am attaching the full jenkins logs for canonistack and aws.

WORK ARROUND
ensure ~/.ssh/id_rsa and its pub exist.
Symlink the keys to that location, or define rules in .ssh/config to us a non-standard name.

[Curtis Hovey (sinzui)]

This is the jenkins aws log

[Curtis Hovey (sinzui)]

Note at http://162.213.35.54:8080/job/hp-upgrade-and-deploy/ shows that hp successfully upgraded for the first time since Novemer 4. It appears that Hp got fixed in r2116 at the cost of aws and canonistack, or that there is an undocumented behavioural difference that testing is not aware of.

[Curtis Hovey (sinzui)]

Using the deb from CI I ran the aws case several times in various combinations. I do see intermittent failures when boostraping r2116 with an identical tool. When comparing the attached log with CI the error is different. All logs show failures bootstrapping r2116 with r2116. My own logs get a few steps further then dies in the apt phase -- git or mongodb failed to install.

[Andrew Wilkins (axwalk)]

FWIW, I tested ec2 before landing and it worked fine. I'll do it again on trunk.

[Andrew Wilkins (axwalk)]

I'm not able to reproduce this at all. sinzui, is it possible for me to get access to the test environment so I can see if that makes a difference?

[Andrew Wilkins (axwalk)]

Also, a copy of the deb. Shouldn't make a difference, but just so I'm testing the exact same thing.

[Curtis Hovey (sinzui)]

Attached is the deb. The test set tools-url. I will push the tools to a sublocation that CI does not over write. I will post info about CI when we bring it back up.

[Andrew Wilkins (axwalk)]

Never mind, I can see the problem. jam brought up a good point, that we're now using SSH as part of the bootstrap process where we never did before. I can see in your build output that you're passing specific identity to "juju scp", which suggests to me that you either don't have a default SSH key on the CI machine, or it's otherwise not usable.

So, your immediate options are either to symlink /var/lib/jenkins/juju-ci/staging-juju-rsa to ~/.ssh/id_rsa (if possible?), or to modify ~/.ssh/config to specify which key to use for the HP/canonistack/etc. cloud IP ranges. Note that in either case, you'll need to also set authorized-keys or authorized-keys-path in your environment config.

Longer term, we may need to add configuration to Juju to tell it to use a particular key. Alternatively, is there a reason why we shouldn't just generate a new SSH key for each environment at bootstrap time?

[Curtis Hovey (sinzui)]

With the exception of the local provider tests all envs specify authorizes-keys-path, (and tools-url, tools-metadata-url). We can look into the symlink of ssh config path.

As an aside, yesterday I built and tested the windows juju for the first time and I discovered that I had to use "id_rsa" when setting up the test.

[Curtis Hovey (sinzui)]

I have down grade the bug to high since we have a work around for an unreleased bug.

[Andrew Wilkins (axwalk)]

IMO, this should be closed. We have solutions:
 - don't specify authorized-keys, and the client keys and default identities will be attempted (pending fix for lp:1275657)
 - specify authorized-keys and ensure either ~/.ssh/config has an appropriate entry, symlink/copy your private key in ~/.juju/ssh, or just ensure it's loaded into the ssh-agent

We should definitely document these things though.

[Curtis Hovey (sinzui)]

I am happy to redefine this issue as a documentation problem. We can change the configs to match the documented practices.