juju deploy -n 15 gets rate limited in EC2
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| juju-core |
Fix Released
|
High
|
Michael Foord | ||
Bug Description
In trying to do some scale testing, I've been trying to deploy a bunch of machines at once.
We currently get a bunch of failures do to rate limiting which causes the machines to never be started. There are 2 bugs here (I think)
1) We are doing something new that is issuing more requests than we used to. One thing that I see is:
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:52 WARNING juju.worker.
2014-02-07 08:43:53 WARNING juju.worker.
I'm guessing the instance updater is overly aggressive about trying to figure out about new machines, getting Request Limit Exceeded, but continuing to hammer the EC2 API periodically.
2) When we go to start a machine, if we get Request Limit Exceeded, we should just try to start that machine again later.
Given that when a machine fails to provision, it fails forever, I think this is reasonably serious. Otherwise any "deploy -n X" is likely to have *some* failures.
Related branches
- Juju Engineering: Pending requested
-
Diff: 444 lines (+330/-19)8 files modifiedcontainer/lxc/lxc.go (+1/-1)
dependencies.tsv (+2/-1)
instance/address.go (+8/-0)
instance/address_test.go (+9/-0)
worker/instancepoller/aggregate.go (+120/-0)
worker/instancepoller/aggregate_test.go (+187/-0)
worker/instancepoller/worker.go (+2/-17)
worker/instancepoller/worker_test.go (+1/-0)
| Changed in juju-core: | |
| status: | Triaged → In Progress |
| assignee: | nobody → Michael Foord (mfoord) |
| milestone: | none → 1.18.0 |
| Changed in juju-core: | |
| milestone: | 1.20.0 → 1.17.6 |
| status: | In Progress → Fix Committed |
| Changed in juju-core: | |
| status: | Fix Committed → Fix Released |
