manual provider bootstrap fails with error about sftp scheme

[Andrew Wilkins]

Reported by Bjorn Tillenius:

"""
bootstrapping the environment fails for me, with this error message:

2013-10-05 15:33:46 INFO juju.environs.tools simplestreams.go:235
Fetching tools to generate hash:
sftp://juju-bootstrap//var/lib/juju/storage/tools/releases/juju-1.15.1-precise-amd64.tgz
2013-10-05 15:33:46 ERROR juju supercommand.go:282 Get
sftp://juju-bootstrap//var/lib/juju/storage/tools/releases/juju-1.15.1-precise-amd64.tgz:
unsupported protocol scheme "sftp"
"""

Not sure why this is happening in Bjorn's env, but not mine. The null provider's bootstrap storage should probably wrap the sshstorage in an httpstorage, to get real URLs for things that expect them.

[Björn Tillenius (bjornt)]

Here's the output of juju bootstrap --debug

[Andrew Wilkins (axwalk)]

@bjornt: thanks, I've reproduced the issue with your debug log.

This log message stuck out: "2013-10-05 17:53:23 INFO juju.environs.sync sync.go:112 found 8 tools in target; 0 tools to be copied"

The issue is that one attempt to bootstrap copied the tools, but didn't generate the tools metadata on the target machine. Then another attempt to bootstrap must've been made, which found the tools, found no metadata, and attempted to create the metadata from the tools on the target machine and failed due to the sftp scheme.

The solution is simple: as mentioned in the description, we should wrap the sshstorage in an httpstorage.

[Andrew Wilkins (axwalk)]

After some discussion with rogpeppe, we concluded that introducing httpstorage here is not the right solution. Instead, writing tools metadata should not use the URL.

I'm proposing a new CL which changes the tools metadata writing to use the storage interface's Get to fetch tools as necessary. There's also a bug fixed to not forget about previously computed sha256/size, and general refactoring of environs/tools/simplestrreams.go and users.

[Jorge Castro (jorge)]

For those of you running into this that missed it on the mailing list if you remove `/var/lib/juju` from the node by hand bootstrap can complete.

[Kapil Thangavelu (hazmat)]

will the solution here work with bootstrap --upload-tools?

[Kapil Thangavelu (hazmat)]

was looking at jorge's comment to remove /var/lib/juju and noticed it already had the relevant tools..

/var/lib/juju/storage/tools/
/var/lib/juju/storage/tools/releases
/var/lib/juju/storage/tools/releases/juju-1.17.0.1-precise-amd64.tgz
/var/lib/juju/storage/tools/releases/juju-1.17.0.1-saucy-amd64.tgz
/var/lib/juju/storage/tools/streams
/var/lib/juju/storage/tools/streams/v1
/var/lib/juju/storage/tools/streams/v1/index.json
/var/lib/juju/storage/tools/streams/v1/com.ubuntu.juju:released:tools.json

[Kapil Thangavelu (hazmat)]

so i'm trying to bootstrap on raring, but i'm on saucy, and only precise and saucy tools are built.. so i never get the right tools, so it could be considered another bug entirely.. see attached re manual bootstrap from $ juju bootstrap -v --debug --upload-tools

[Andrew Wilkins (axwalk)]

@hazmat: https://bugs.launchpad.net/juju-core/+bug/1236691

[Andrew Wilkins (axwalk)]

Confirmed that I can bootstrap, remove everything but the tools/metadata in storage, and bootstrap again.