juju deploy -n 15 gets rate limited in EC2

Bug #1277397 reported by John A Meinel on 2014-02-07
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju-core
High
Michael Foord

Bug Description

In trying to do some scale testing, I've been trying to deploy a bunch of machines at once.

We currently get a bunch of failures do to rate limiting which causes the machines to never be started. There are 2 bugs here (I think)

1) We are doing something new that is issuing more requests than we used to. One thing that I see is:

2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:52 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)
2014-02-07 08:43:53 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": Request limit exceeded. (RequestLimitExceeded)

I'm guessing the instance updater is overly aggressive about trying to figure out about new machines, getting Request Limit Exceeded, but continuing to hammer the EC2 API periodically.

2) When we go to start a machine, if we get Request Limit Exceeded, we should just try to start that machine again later.

Given that when a machine fails to provision, it fails forever, I think this is reasonably serious. Otherwise any "deploy -n X" is likely to have *some* failures.

Related branches

John A Meinel (jameinel) on 2014-03-07
Changed in juju-core:
status: Triaged → In Progress
assignee: nobody → Michael Foord (mfoord)
milestone: none → 1.18.0
John A Meinel (jameinel) on 2014-03-18
Changed in juju-core:
milestone: 1.20.0 → 1.17.6
status: In Progress → Fix Committed
Curtis Hovey (sinzui) on 2014-03-20
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers