Comment 12 for bug 1888409
Revision history for this message
| Ben Hoyt (benhoyt) wrote Re: running parallel bootstraps in aws gives request limit execeeded | #12 |
© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
35a32c5
(Get the code!)
Hi Adam -- just an FYI that we've made some progress on this: I've been able to repro locally using your bash script with "parallel" (though I have 8 cores so it bootstrapped 8 in parallel). I saw a bunch of RequestLimitExc
We suspect this is happening because there are two AWS libraries being used in Juju: the legacy one for most EC2 calls, but the new AWS SDK for bootstrapping. A lot of AWS API calls are made during bootstrapping (and multiplied by the # of parallel bootstraps) to determine instance types / costs -- on the order of 300 requests in a few seconds when bootstrapping 8 controllers.
Ian added request retries to the legacy library. The AWS SDK has retries, but I believe they're not turned on by default, so all the "instance type" requests that happen on startup don't have retrying / exponential backoff enabled.
I'm going to spend some more time on this tomorrow to confirm the above and (hopefully) get in a fix that enables retrying for calls made from the AWS SDK library too.