authentication token expiry with openstack provider

[James Page]

I hit this in a long running environment hosted on an openstack deployment.

After a certain period of time (not sure how long) all provisioning agent calls fail with 401 errors:

2012-08-09 10:55:25,555: juju.openstack@DEBUG: openstack: 401 '401 Unauthorized\n\nThis server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.\n\n Authentication required '
2012-08-09 10:55:25,556: juju.agents.provision@ERROR: Cannot get machine list
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/juju/agents/provision.py", line 175, in process_machines
    provider_machines = yield self.provider.get_machines()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1043, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/juju/providers/openstack/provider.py", line 103, in get_machines
    instances = yield self.nova.list_servers()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/juju/providers/openstack/client.py", line 268, in _json
    response.code, body))
ProviderInteractionError: Unexpected 401: '401 Unauthorized\n\nThis server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.\n\n Authentication required '

restarting the provisioning agent on the bootstrap node resolved this issue.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jim Baker (jimbaker)]

According to http://docs.openstack.org/api/quick-start/content/index.html#Getting-Credentials-a00665, code using the OpenStack REST API should request a new token upon receiving a 401 status code, then retry.

In the OpenStack provider, for every client request, the `_OpenStackClient.is_authenticated` method is checked to see if a token needs to be first requested. However, as currently implemented, this method will never reset to being False. So solving this bug simply requires ensuring that any 401 status code does in fact do that; something as simple as setting `self.token = None` in the `_empty` and `_json` result processing methods should suffice. (We can also return a better exception too, so as to improve the logging in the provisioning agent, just to indicate this expiration is to be occasionally expected.)

With that change, the provisioning agent will retry as it always does when it encounters provider errors, thus fufilling the requirements of the above link, and thereby fixing this bug.