bootstrap requires unrestricted HTTP egress

[Nick Moffitt]

After installing juju-core on a precise server in a private MAAS environment in a highly sensitive network, bootstrap failed in the following manner:

    2013-08-23 13:36:57 INFO juju tools.go:25 environs: reading tools with major version 1
    2013-08-23 13:36:57 INFO juju tools.go:29 environs: falling back to public bucket
    2013-08-23 13:36:57 INFO juju.environs.sync sync.go:69 listing available tools listing available tools
    2013-08-23 13:38:01 ERROR juju supercommand.go:274 command failed: Get https://juju-dist.s3.amazonaws.com/: dial tcp 72.21.195.15:443: connection timed out
    error: Get https://juju-dist.s3.amazonaws.com/: dial tcp 72.21.195.15:443: connection timed out

This is correct behaviour, due to the egress filtering required by this network.

I had to look around to find out that I needed to use the --upload-tools parameter to get it to use my local copy of the binary. Since I was using the packaged version of juju-core, my intention to use that specific version of the juju tools should have been obvious.

Could the default (at least for the packaged version) be changed not to download untested versions off the Internet?

[Matthew Wedgwood (mew)]

I second Nick's request, and also point out that the provider might be an appropriate indicator of whether the tools should be collected from AWS. Is the behavior the same on Azure or HP Cloud?

My vote would be behind --upload-tools being the default, with a convenience option to use canonical-hosted buckets.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2013-08-23 18:33, Matthew Wedgwood wrote:
> I second Nick's request, and also point out that the provider
> might be an appropriate indicator of whether the tools should be
> collected from AWS. Is the behavior the same on Azure or HP Cloud?
>
> My vote would be behind --upload-tools being the default, with a
> convenience option to use canonical-hosted buckets.
>

The tools on your local system are generally not valid for all
permutations of target machines. (i386, amd64, precise, quantal,
raring, saucy).

It is intended that you would get a copy of the tools into your local
cloud from some other location. (see 'juju sync-tools' and the
- --source parameter if you want to copy them manually from a local disk
copy.)

I understand not wanting "juju bootstrap" to automatically grab them
for you. "--upload-tools" will work in a subset of cases (where the
platform you are running on matches exactly the platforms of all
machines you will be running).

Without some sort of tools copying, your environment wouldn't have
been able to bootstrap anyway, so I'm not sure failing at trying to
copy tools is worse than failing at not having tools available.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIZlokACgkQJdeBCYSNAAM+CgCgjHZIogvkLnN58HWx8IsMDVMv
CSMAoJ0cPZeJO8uK3ZgzeJ2Qi1/voJOQ
=5zXz
-----END PGP SIGNATURE-----

[Nick Moffitt (nick-moffitt)]

Wow, really? This seems like a pretty serious regression from the Python juju. At least then I could trivially tell it to use the same PPA I was, or to use the version in the Ubuntu archive.

Perhaps a tool to grab all the architectures and releases of the tools in a directory suitable for sync-tools would be helpful. Is there such a thing?

[Jacek Nykis (jacekn)]

I just hit this bug myself. The error message is very misleading:

$ juju bootstrap
error: failed to list contents of container: juju-dist
caused by: Unauthorised URL http://xxxx:8080/v1/AUTH_0000/juju-dist?delimiter=&marker=&prefix=tools%2Fjuju-
caused by: request (http://xxxx:8080/v1/AUTH_0000/juju-dist?delimiter=&marker=&prefix=tools%2Fjuju-) returned unexpected status: 401; error info: 401 Unauthorized
This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.

I spent quite some time checking access details and testing while the solution was to run "juju bootstrap --upload-tools". If downloading data from amazon is permanent solution then this error message could be improved. Juju could also suggest way of fixing this problem.

Ideal solution would be for juju to realise it does not have access to amazon and try "juju bootstrap --upload-tools" to make this transparend to end user.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2013-08-29 15:37, Jacek Nykis wrote:
> I just hit this bug myself. The error message is very misleading:
>
> $ juju bootstrap error: failed to list contents of container:
> juju-dist caused by: Unauthorised URL
> http://xxxx:8080/v1/AUTH_0000/juju-dist?delimiter=&marker=&prefix=tools%2Fjuju-
>
>
>
caused by: request
(http://xxxx:8080/v1/AUTH_0000/juju-dist?delimiter=&marker=&prefix=tools%2Fjuju-)
returned unexpected status: 401; error info: 401 Unauthorized
> This server could not verify that you are authorized to access the
> document you requested. Either you supplied the wrong credentials
> (e.g., bad password), or your browser does not understand how to
> supply the credentials required.
>
> I spent quite some time checking access details and testing while
> the solution was to run "juju bootstrap --upload-tools". If
> downloading data from amazon is permanent solution then this error
> message could be improved. Juju could also suggest way of fixing
> this problem.
>
> Ideal solution would be for juju to realise it does not have
> access to amazon and try "juju bootstrap --upload-tools" to make
> this transparend to end user.
>

- --upload-tools is actually a bit of a hack for developers that want to
test with a set of tools that has never been actually released. It
isn't intended to actually work in a real deployment, and has a bunch
of edge bits that may fail as you try to deploy multiple series, etc.

We can fail and try to offer a better place to sync-tools from, though.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIfNo4ACgkQJdeBCYSNAAPNggCfWrx81zmO4gP+XXW/G4Zgh4Xn
OEEAnjiwstleUuh4Tk1kU3rna8UTgIG2
=AdP9
-----END PGP SIGNATURE-----

[Nick Moffitt (nick-moffitt)]

The ubuntu archive would be ideal. This stuff should all be in main!

[Curtis Hovey (sinzui)]

There will be a mirror command very soon that will someone to gather all the public tools. Starting with juju 1.15, the tools-url can be set to a local directory or a apache set to mirror the tools.

[Mark Ramm (mark-ramm)]

What is the status on the mirror command?

[Curtis Hovey (sinzui)]

I lowered the priority of this bug. We might choose to close it. Juju has proxy and apt-proxy support as of 1.18.x Private clouds can generate their own streams for images and tools to bootstrap.

[Mark Ramm (mark-ramm)]

Let's close it as fixed for now. Further work on this subject can be
taken up in new bugs with specific action items as needed.

On Mon, May 12, 2014 at 2:16 PM, Curtis Hovey <email address hidden> wrote:

> I lowered the priority of this bug. We might choose to close it. Juju
> has proxy and apt-proxy support as of 1.18.x Private clouds can generate
> their own streams for images and tools to bootstrap.
>
> ** Changed in: juju-core
> Importance: High => Medium
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1215948
>
> Title:
> bootstrap requires unrestricted HTTP egress
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1215948/+subscriptions
>