Juju does not handle 429 "too many requests" from Azure

[Aaron Bentley]

As seen here:
http://reports.vapour.ws/releases/issue/571a75d7749a5618d867765e
and
http://reports.vapour.ws/releases/issue/56af6c8a749a561ea206843b

Juju should not fail when Azure tells it "too many requests" -- it should slow down.

This issue is critical be it consistently fails revision tests. It is contributing to resource exhaustion because Juju cannot clean up -- contributing to other failures.

[Aaron Bentley (abentley)]

We are now seeing this frequently, so I have upped the priority.

[Andrew Wilkins (axwalk)]

I think we should be able to modify the response handlers to back off when we receive 429. This will involve modifying the clients created in "azureEnviron.SetConfig".

[Andrew Wilkins (axwalk)]

I have to pause on this for now, so unassigning myself in case someone else wants to pick it up.

I think we want to decorate the autorest.Client's Sender such that it retries (for a specified duration) with backoff when a 429 occurs. We may want or need to update azure-sdk-for-go first, as it appears that the API for autorest has changed a bit.

[Andrew Wilkins (axwalk)]

The Azure SDK for Go has changed quite a bit, I'll need to rejigger some things on our end. Also there may be a problem caused by them using vendoring.

[Andrew Wilkins (axwalk)]

Vendor issue was my fault, disregard.

[Andrew Wilkins (axwalk)]

I'm going to bump as far as 3b480eaaf6b4236d43a3c06cba969da6f53c8b66, which I've already done the work for (https://github.com/juju/juju/pull/3807). Moving to master is too much of a pain right now. I've filed an Azure SDK for Go issue to ask what's up: https://github.com/Azure/azure-sdk-for-go/issues/326.

[Andrew Wilkins (axwalk)]

So I was previously thinking we'd inject in to the autorest senders, but I don't think that's going to work in general. PUT/POST will send a body, and the autorest stuff doesn't use rewindable buffers.

So I think I'll just wrap all API calls with juju/retry.Call.