environs/ec2: certain operations should be retried if possible

[Dave Cheney]

lucky(~/src/launchpad.net/juju-core) % juju bootstrap --upload-tools --debug -e 'us-west-1' && juju deploy -e 'us-west-1' mongodb && juju expose -e 'us-west-1' mongodb
2012/08/27 18:01:15 JUJU environs/ec2: opening environment "us-west-1"
2012/08/27 18:01:15 JUJU environs/ec2: bootstrapping environment "us-west-1"
2012/08/27 18:01:27 JUJU environs: putting tools tools/juju-0.0.1-precise-amd64.tgz
2012/08/27 18:02:38 JUJU environs/ec2: starting machine 0 in "us-west-1" running tools version "0.0.1-precise-amd64" from "https://s3-us-west-1.amazonaws.com/juju-us-west-1-dfc/tools/juju-0.0.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Expires=1377590558&Signature=CxRZilHdnKJ1%2F%2Fn2m5sPIwYnCwo%3D"
2012/08/27 18:04:05 JUJU:DEBUG juju bootstrap command failed: cannot start bootstrap instance: cannot set up groups: Get https://ec2.us-west-1.amazonaws.com?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Action=DescribeSecurityGroups&GroupName.1=juju-us-west-1&Signature=YKIcSN9m90YBXv%2F439ofBQnCwfHn6tDOBXvrRQFW4uY%3D&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2012-08-27T08%3A02%3A41Z&Version=2011-12-15: read tcp 204.246.162.140:443: connection reset by peer
error: cannot start bootstrap instance: cannot set up groups: Get https://ec2.us-west-1.amazonaws.com?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Action=DescribeSecurityGroups&GroupName.1=juju-us-west-1&Signature=YKIcSN9m90YBXv%2F439ofBQnCwfHn6tDOBXvrRQFW4uY%3D&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2012-08-27T08%3A02%3A41Z&Version=2011-12-15: read tcp 204.246.162.140:443: connection reset by peer

At this point I had to remove the bucket manually and try again.

[Marco Ceppi (marcoceppi)]

This happens frequently, and can be resolved by just trying "over and over" again

marco@home:~$ juju bootstrap -e ondina2
error: Get https://s3.amazonaws.com/juju-asfasdfsadfasfdgojuju/?delimiter=&prefix=tools%2Fjuju-&marker=: read tcp 207.171.163.205:443: connection reset by peer
marco@home:~$ juju bootstrap -e ondina2
error: Get https://s3.amazonaws.com/juju-asfasdfsadfasfdgojuju/?delimiter=&prefix=tools%2Fjuju-&marker=: read tcp 205.251.242.196:443: connection reset by peer
marco@home:~$ juju bootstrap -e ondina2
error: Get https://s3.amazonaws.com/juju-asfasdfsadfasfdgojuju/?prefix=tools%2Fjuju-&marker=&delimiter=: read tcp 205.251.242.196:443: connection reset by peer
marco@home:~$ juju bootstrap -e ondina2
marco@home:~$

Possibly add a set amount of times to retry? (Say 5) before erroring out with a slightly more helpful error message ("Unable to communicate with S3 Bucket." possibly with more verbose instructions even?)

[John A Meinel (jameinel)]

Can you run with '-v' when this happens? Most of the code I've seen does retry, if you are finding a case that doesn't, it could be fixed.

[William Reade (fwereade)]

Initial description has a JUJU:DEBUG line, so it's even more detailed than "-v" can give us. AFAICT we don't retry in this case, and we probably should indeed.