use a utils/http client for all HTTP(S) calls across the codebase

[Dimiter Naydenov]

Right now we're using http.DefaultClient in some places and other parts of the code assume that (e.g. simplestreams registers a file:// handler for testing purposes, but it's inside the production code serving any file:// URL from root / - look at init() in simplestreams.go). If ssl-hostname-verification is set to false though, we use utils.GetNonValidatingHTTPClient, which creates a TLS transport and returns a client with it. This opens up an issue with manual bootstrapping an environment with ssl-hostname-verification: false:

2014-03-14 08:04:54 ERROR juju.cmd supercommand.go:296 cannot load state from URL "file:///var/lib/juju/storage/provider-state" (read from "/tmp/provider-state-url"): Get file:///var/lib/juju/storage/provider-state: unsupported protocol scheme "file"

Bootstrap fails, because environs/state LoadStateFromURL() with the non-validating HTTP client does not have the file:// protocol registered. It doesn't fail with ssl-hostname-verification: true (the default) because of simplestreams testing file:// proto registration on http.DefaultTransport.

We need to change all places in the code where we use an http client to create one from utils/http and provide a way to register a file handler (or others), as needed by the providers, on both TLS and HTTP clients. Only the manual provider uses file:// URLs for now, but due to the simplestreams testing-code-slipped-into-production it wasn't detected before.

It's not a regression, because it occurs only with the manual provider, the only one using file:// URLs, and it was like that since 1.16.

[Andrew Wilkins (axwalk)]

Why is a file:// URL being used at all? That doesn't seem right.

[Andrew Wilkins (axwalk)]

My mistake; it should be using file://. LoadStateFromURL is only used by the bootstrap agent, and storage is located on the bootstrap agent machine's filesystem.

[Ian Booth (wallyworld)]

The simplestreams registration file:// is not for testing - it is for production code. If it were just for testing it would be in a test package somewhere :-) A simplestreams data source can be a local directory and since simplestreams fetches data using http.Get(), the file:// protocol needs to be registered.

[Andrew Wilkins (axwalk)]

If we change jujud bootstrap-state to not read the provider-state file, but take information over the command line, the manual provider issue will be irrelevant. We can do this now with synchronous bootstrap, since we start the instance and then construct the command line after we know its instance ID/hardware characteristics.

I'm working on a proposal for this now.