network error causes tools download to fail

[David Britton]

Similar to https://bugs.launchpad.net/juju-core/+bug/1252974 which is fixed released. I'm still getting this on 1.20.2. Here is the latest one:

Get:1 http://us.archive.ubuntu.com//ubuntu/ trusty/universe rsyslog-gnutls amd64 7.4.4-1ubuntu2 [13.9 kB]
Fetched 13.9 kB in 0s (0 B/s)
Selecting previously unselected package rsyslog-gnutls.
(Reading database ... 56773 files and directories currently installed.)
Preparing to unpack .../rsyslog-gnutls_7.4.4-1ubuntu2_amd64.deb ...
Unpacking rsyslog-gnutls (7.4.4-1ubuntu2) ...
Setting up rsyslog-gnutls (7.4.4-1ubuntu2) ...
curl: (7) Failed to connect to juju-dist.s3.amazonaws.com port 443: Connection timed out
tools from https://juju-dist.s3.amazonaws.com/rc-testing/tools/releases/juju-1.20.2-trusty-amd64.tgz downloaded: HTTP 000; time 126.352s; size 0 bytes; speed 0.000 bytes/s ERROR bootstrap failed: subprocess encountered error code 1
Stopping instance...
Bootstrap failed, destroying environment
ERROR subprocess encountered error code 1

Can we use the '--retry 10' option on curl? Seems a shame to just do this once when the next bootstrap worked.

[Andreas Hasenack (ahasenack)]

It looks like juju-dist.s3.amazonaws.com translates to several IPs, and it's hard to find a list of all. In fact, such a list doesn't exist. The firewall has to play catch up and filter new ones as we find them.

In this particular case, we believe that the IP juju-dist.s3.amazonaws.com resolved too wasn't allowed in the egress filtering rules of our firewall. We will add it, but this will happen again.

My suggestions are:
- log which IP was used to try the connection
- if attempting a retry, make sure to do a DNS resolution one more time before