Temporary network outage results in connection refused and invalid token

[Vish Ishaya]

The auth_token middleware does not retry if it gets an http error. This means a temporary network outage or keystone restart will cause the token to be listed as invalid. The middleware should retry a few times before failing.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/23729

[Vish Ishaya (vishvananda)]

keystoneclient fix:
https://review.openstack.org/23728

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/23728
Committed: http://github.com/openstack/python-keystoneclient/commit/9bb29b9dff4d220eaabf86cd286e651c5ae15b98
Submitter: Jenkins
Branch: master

commit 9bb29b9dff4d220eaabf86cd286e651c5ae15b98
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Mar 6 12:10:25 2013 -0800

    Retry http_request and json_request failure.

    Temporary network outages or keystone server restarts can lead
    to a connection refused in auth_token middleware. Rather than
    failing immediately, this patch attempts to retry a few times.

    Fixes bug 1150299

    Change-Id: I2ecf0d7745290976efcb3e3cd6511817a53d3e0a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/23729
Committed: http://github.com/openstack/keystone/commit/9e0a97d426165842617e55fe74341d06526f9d1e
Submitter: Jenkins
Branch: stable/folsom

commit 9e0a97d426165842617e55fe74341d06526f9d1e
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Mar 19 16:07:07 2013 -0700

    Retry http_request and json_request failure.

    Temporary network outages or keystone server restarts can lead
    to a connection refused in auth_token middleware. Rather than
    failing immediately, this patch attempts to retry a few times.

    Fixes bug 1150299

    (pulled from python-keystoneclient commit
    9bb29b9dff4d220eaabf86cd286e651c5ae15b98)

    Change-Id: I2ecf0d7745290976efcb3e3cd6511817a53d3e0a

[Adam Young (ayoung)]

I think this might be the wrong fix. Instead of retrying a fixed number of times, the auth_token middleware should not record failures in memcached if they are due to a network failure.