Cannot bring up hosted model machines in azure

[Curtis Hovey]

As seen in this example
    http://reports.vapour.ws/releases/4249/job/azure-arm-deploy/attempt/810#highlight

CI can bootstrap in azure, but none of the machines in the hosted model come up. We can see that the rate limit issue is gone and we have cleaned up all the resources in our subscription. Ever azure test can bootstrap and tear down. None can deploy services.

[Andrew Wilkins (axwalk)]

I don't know what the deal is yet, but in the logs for the workload machines I'm seeing auth failures. Looking at the logs on the controller, the workload machine appears to be trying to authenticate with another machine's nonce.

[Andrew Wilkins (axwalk)]

Nope, that's not it. The nonces are formatted in a misleading way, but the correct one is being used.

[Andrew Wilkins (axwalk)]

OK, I think I understand what's going on.

Due to the recent Azure SDK change, when you create a VM the SDK now waits for the "provisioningState" to go to "Succeeded". This is causing us to run into a latent race condition in the provisioner. The provisioner does not record the machine's nonce until it thinks it has been provisioned; so if the machine attempts an API login before the provisioner thinks the machine is ready, the machine gets an auth error back, and uninstalls itself.

It may be that we can hack the Azure code to stop waiting for the provisioningState in this case, but I think we would be better off fixing the race condition.

[Andrew Wilkins (axwalk)]

Fixing the race is probably going to be a fairly large amount of work, so I'm working on a bandaid solution while I come up with a proposal.