Hi,
If delete and re-create keystone data the 'auth_token' middleware stops working and thus neither Nova nor Glance (I suppose the same case for Swift, not tested) will work again until you restart it. This is because of the 'auth_token' middleware only refresh its token after a 401 error, however if you delete and re-create the service tenant either a 500 (current master) or 404 (after applying https://review.openstack.org/#/c/6875/) occurs. How to reproduce it on fresh Devstack:
Refresh data:
$ mysql -e 'drop database keystone; create database keystone';
$ keystone-manage db_sync
$ files/keystone_data.sh
Check nova still able to authenticate itself:
$ curl -X POST -d '{"auth": {"tenantName": "service", "passwordCredentials": {"username": "nova", "password": "secrete"}}}' -H "Content-type: application/json" http://127.0.0.1:35357/v2.0/tokens
{"access": {"token": {"expires": "2012-06-13T18:39:56Z", "id": "3a35b2ea4d0346b4b97dc83334c0731e", "tenant": {"enabled": true, "id": "7aaf52ba306f4bb895cbb457387ba3cb", "name": "service", "description": null}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.27.100:8774/v2/7aaf52ba306f4bb895cbb457387ba3cb", "region": "RegionOne", "publicURL": "http://192.168.27.100:8774/v2/7aaf52ba306f4bb895cbb457387ba3cb", "internalURL": "http://192.168.27.100:8774/v2/7aaf52ba306f4bb895cbb457387ba3cb"}], "endpoints_links": [], "type": "compute", "name": "Compute Service"}, {"endpoints": [{"adminURL": "http://192.168.27.100:3333", "region": "RegionOne", "publicURL": "http://192.168.27.100:3333", "internalURL": "http://192.168.27.100:3333"}], "endpoints_links": [], "type": "s3", "name": "S3 Service"}, {"endpoints": [{"adminURL": "http://192.168.27.100:9292", "region": "RegionOne", "publicURL": "http://192.168.27.100:9292", "internalURL": "http://192.168.27.100:9292"}], "endpoints_links": [], "type": "image", "name": "Image Service"}, {"endpoints": [{"adminURL": "http://192.168.27.100:8776/v1/7aaf52ba306f4bb895cbb457387ba3cb", "region": "RegionOne", "publicURL": "http://192.168.27.100:8776/v1/7aaf52ba306f4bb895cbb457387ba3cb", "internalURL": "http://192.168.27.100:8776/v1/7aaf52ba306f4bb895cbb457387ba3cb"}], "endpoints_links": [], "type": "volume", "name": "Volume Service"}, {"endpoints": [{"adminURL": "http://192.168.27.100:8773/services/Admin", "region": "RegionOne", "publicURL": "http://192.168.27.100:8773/services/Cloud", "internalURL": "http://192.168.27.100:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "EC2 Service"}, {"endpoints": [{"adminURL": "http://192.168.27.100:35357/v2.0", "region": "RegionOne", "publicURL": "http://192.168.27.100:5000/v2.0", "internalURL": "http://192.168.27.100:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "Identity Service"}], "user": {"username": "nova", "roles_links": [], "id": "cab9a6a20e4d40b5a2ad109eee6b80a7", "roles": [{"id": "70dc17905fc44c44ab913f7700f27dcd", "name": "admin"}], "name": "nova"}}}
Tyring a request on nova:
$ source openrc
$ nova list
ERROR: n/a (HTTP 401)
Looking at tcpflow, the request fails because the validate token is failing, since the token used for validation contains invalid information and it's never refreshed. I've also attached the outputs with --debug flag and tcpflow.
This could be easily solved just returning a 401 error whenever a validate token call is done or invalidating the validation token, however I don't really like any of that solutions, since a 404 error is the right error for a resource that doesn't exist and discarding a token after a "real" 404 might cause 'auth_token' getting new tokens much more frequently than really needs. Any thought??