authentication token expiry with openstack provider

Bug #1034837 reported by James Page on 2012-08-09
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
pyjuju
Critical
Jim Baker
0.5
Critical
Clint Byrum
juju (Ubuntu)
Critical
Unassigned
Quantal
Critical
Unassigned

Bug Description

I hit this in a long running environment hosted on an openstack deployment.

After a certain period of time (not sure how long) all provisioning agent calls fail with 401 errors:

2012-08-09 10:55:25,555: juju.openstack@DEBUG: openstack: 401 '401 Unauthorized\n\nThis server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.\n\n Authentication required '
2012-08-09 10:55:25,556: juju.agents.provision@ERROR: Cannot get machine list
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/juju/agents/provision.py", line 175, in process_machines
    provider_machines = yield self.provider.get_machines()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1043, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/juju/providers/openstack/provider.py", line 103, in get_machines
    instances = yield self.nova.list_servers()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/juju/providers/openstack/client.py", line 268, in _json
    response.code, body))
ProviderInteractionError: Unexpected 401: '401 Unauthorized\n\nThis server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.\n\n Authentication required '

restarting the provisioning agent on the bootstrap node resolved this issue.

Related branches

Changed in juju:
status: New → Confirmed
importance: Undecided → High
Changed in juju:
importance: High → Critical
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

affects: ubuntu → juju (Ubuntu)
Changed in juju (Ubuntu):
status: New → Confirmed
Changed in juju (Ubuntu):
status: New → Confirmed
Changed in juju (Ubuntu Quantal):
importance: Undecided → High
Changed in juju (Ubuntu Quantal):
importance: High → Critical
Changed in juju:
milestone: none → 0.6
Changed in juju (Ubuntu Quantal):
milestone: none → ubuntu-12.10
Jim Baker (jimbaker) on 2012-09-22
Changed in juju:
assignee: nobody → Jim Baker (jimbaker)
Jim Baker (jimbaker) wrote :

According to http://docs.openstack.org/api/quick-start/content/index.html#Getting-Credentials-a00665, code using the OpenStack REST API should request a new token upon receiving a 401 status code, then retry.

In the OpenStack provider, for every client request, the `_OpenStackClient.is_authenticated` method is checked to see if a token needs to be first requested. However, as currently implemented, this method will never reset to being False. So solving this bug simply requires ensuring that any 401 status code does in fact do that; something as simple as setting `self.token = None` in the `_empty` and `_json` result processing methods should suffice. (We can also return a better exception too, so as to improve the logging in the provisioning agent, just to indicate this expiration is to be occasionally expected.)

With that change, the provisioning agent will retry as it always does when it encounters provider errors, thus fufilling the requirements of the above link, and thereby fixing this bug.

Jim Baker (jimbaker) on 2012-09-26
Changed in juju:
status: Confirmed → Won't Fix
status: Won't Fix → In Progress
Jim Baker (jimbaker) on 2012-09-27
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
Changed in juju (Ubuntu Quantal):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers