After grizzly upgrade, EC2 API requests fail:Could not find: credential

Bug #1158563 reported by Adam Gandelman on 2013-03-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
keystone (Ubuntu)
Undecided
Unassigned

Bug Description

Upgrading a fairly standard Folsom setup to Grizzly using our packages. After the updated config files are put in place and database has been migrated, nova's EC2 API fails to authenticate requests with keystone. The OSAPI end point works just fine.

On the client:

$ euca-describe-instances
Unauthorized: Failure communicating with keystone

On the keystone server, with debug enabled:

2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ********************
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x2d64250 200 OK>}
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] REQUEST_METHOD = POST
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] PATH_INFO = /ec2tokens
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] CONTENT_LENGTH = 436
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] eventlet.posthooks = []
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] RAW_PATH_INFO = /v2.0/ec2tokens
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] REMOTE_ADDR = 192.168.20.50
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x2d56050>
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.url_scheme = http
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] webob._body_file = (<LimitedLengthFile(<eventlet.wsgi.Input object at 0x2d56050>, maxlen=436)>, <eventlet.wsgi.Input object at 0x2d56050>)
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SERVER_PORT = 5000
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.input = <_io.BytesIO object at 0x2d5c1d0>
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': None, 'is_admin': False}
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] HTTP_HOST = test-09.os.magners.qa.lexington:5000
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.multithread = True
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] openstack.params = {u'ec2Credentials': {u'access': u'8e283c4e394247fbbabe5474cdbda9e4', u'host': u'test-02.os.magners.qa.lexington:8773', u'verb': u'POST', u'params': {u'SignatureVersion': u'2', u'AWSAccessKeyId': u'8e283c4e394247fbbabe5474cdbda9e4', u'Timestamp': u'2013-03-21T22:53:36Z', u'SignatureMethod': u'HmacSHA256', u'Version': u'2010-08-31', u'Action': u'DescribeInstances'}, u'signature': u'gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=', u'path': u'/services/Cloud/'}}
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SERVER_NAME = 192.168.20.57
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.run_once = False
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x7f0bbbf08270>
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] webob.is_body_seekable = True
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] CONTENT_TYPE = application/json
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi]
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] {"ec2Credentials": {"access": "8e283c4e394247fbbabe5474cdbda9e4", "host": "test-02.os.magners.qa.lexington:8773", "verb": "POST", "params": {"SignatureVersion": "2", "AWSAccessKeyId": "8e283c4e394247fbbabe5474cdbda9e4", "Timestamp": "2013-03-21T22:53:36Z", "SignatureMethod": "HmacSHA256", "Version": "2010-08-31", "Action": "DescribeInstances"}, "signature": "gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=", "path": "/services/Cloud/"}}
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi]
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] arg_dict: {}
2013-03-21 18:53:36 WARNING [keystone.common.wsgi] Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] Content-Type = application/json
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] Content-Length = 120
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi]
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] {"error": {"message": "Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4", "code": 404, "title": "Not Found"}}
2013-03-21 18:53:36 INFO [access] 192.168.20.50 - - [21/Mar/2013:22:53:36 +0000] "POST http://test-09.os.magners.qa.lexington:5000/v2.0/ec2tokens HTTP/1.0" 404 120
2013-03-21 18:53:36 DEBUG [eventlet.wsgi.server] 192.168.20.50 - - [21/Mar/2013 18:53:36] "POST /v2.0/ec2tokens HTTP/1.1" 404 256 0.010670

The nova-api-ec2.log:

2013-03-21 18:54:28.516 13600 ERROR nova.api.ec2 [-] Unauthorized: Failure communicating with keystone
2013-03-21 18:54:28.516 13600 INFO nova.api.ec2 [-] 0.14199s 192.168.20.1 POST /services/Cloud/ None:None 400 [Boto/2.3.0 (linux2)] application/x-www-form-urlencoded text/xml
2013-03-21 18:54:28.517 13600 INFO nova.ec2.wsgi.server [-] 192.168.20.1 "POST /services/Cloud/ HTTP/1.1" status: 400 len: 327 time: 0.0149109

tags: added: upgrade
Michael Still (mikal) on 2013-04-02
tags: added: ec2
Adam Gandelman (gandelman-a) wrote :

This is total configuration / packaging error. After the package upgrade, the EC2 was set to use the KVS backend, when it was originally using SQL.

Changed in keystone (Ubuntu):
status: New → Confirmed
Michael Still (mikal) wrote :

@Adam -- can we therefore remove the upstream tasks from this bug?

Adam Gandelman (gandelman-a) wrote :

Michael, I'd love nothing more, but LP seems to be timing out anytime I try touching those tasks. :\

no longer affects: keystone
no longer affects: nova
no longer affects: nova (Ubuntu)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package keystone - 1:2013.1-0ubuntu1

---------------
keystone (1:2013.1-0ubuntu1) raring; urgency=low

  [ Adam Gandelman ]
  * debian/patches/sql_connection.patch: Ensure SQL by default for all
    backends. (LP: #1158563)
  * debian/rules: Reinstate use of test_overrides.conf to target upstream
    defaults when running unit tests.

  [ Chuck Short ]
  * New upstream release.
 -- Chuck Short <email address hidden> Fri, 05 Apr 2013 22:32:17 -0500

Changed in keystone (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers