tempest failing with boto==2.35.0

[Attila Fazekas]

logstash: message: 'Signature not provided' and message: 'AuthFailure'

Gate permanently failing since the boto 2.35.0 release.

[Attila Fazekas (afazekas)]

Looks like boto switched to hmac-v4 auth https://github.com/boto/boto/commit/32abe3e99961069912b5004e1507d40e785300ae .

[Attila Fazekas (afazekas)]

Change for changing the global requirements https://review.openstack.org/#/c/146049/.

Adding nova for implementing HMAC-V4 support.
Nova expects a 'Signature' named header, which is now part of the 'Authorization'.

The issue can be reproduced with euca2ools:

# source /opt/stack/new/devstack/accrc/demo/demo
# euca-describe-keypairs --debug
2015-01-09 13:14:32,349 euca2ools [DEBUG]:Using access key provided by client.
2015-01-09 13:14:32,350 euca2ools [DEBUG]:Using secret key provided by client.
2015-01-09 13:14:32,351 euca2ools [DEBUG]:Method: POST
2015-01-09 13:14:32,351 euca2ools [DEBUG]:Path: /services/Cloud/
2015-01-09 13:14:32,352 euca2ools [DEBUG]:Data:
2015-01-09 13:14:32,352 euca2ools [DEBUG]:Headers: {}
2015-01-09 13:14:32,352 euca2ools [DEBUG]:Host: 172.16.40.26
2015-01-09 13:14:32,353 euca2ools [DEBUG]:Port: 8773
2015-01-09 13:14:32,353 euca2ools [DEBUG]:Params: {'Action': 'DescribeKeyPairs', 'Version': '2010-08-31'}
2015-01-09 13:14:32,354 euca2ools [DEBUG]:establishing HTTP connection: kwargs={'port': 8773, 'timeout': 70}
2015-01-09 13:14:32,354 euca2ools [DEBUG]:Token: None
2015-01-09 13:14:32,355 euca2ools [DEBUG]:CanonicalRequest:
POST
/services/Cloud/

host:172.16.40.26:8773
x-amz-date:20150109T131432Z

host;x-amz-date
93691be75657638bb0188c9dd56303b89bb2818598871011d73eee11e14e0cec
2015-01-09 13:14:32,356 euca2ools [DEBUG]:StringToSign:
AWS4-HMAC-SHA256
20150109T131432Z
20150109/16/172/aws4_request
f8748433ff623a4e9cbd616ef63ebe6e506b36f1fd341a41983c955e59b82de7
2015-01-09 13:14:32,357 euca2ools [DEBUG]:Signature:
2dfa2098a8b893cec02f42b0e2abbe7ae5c6077ca1e5d8e1426cad5621e93e24
2015-01-09 13:14:32,357 euca2ools [DEBUG]:Final headers: {'Content-Length': '42', 'User-Agent': 'Boto/2.35.0 Python/2.7.5 Linux/3.17.7-200.fc20.x86_64', 'Host': '172.16.40.26:8773', 'X-Amz-Date': '20150109T131432Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Authorization': 'AWS4-HMAC-SHA256 Credential=6d8332aeeeb94e11bb23d4fc09c0a0f3/20150109/16/172/aws4_request,SignedHeaders=host;x-amz-date,Signature=2dfa2098a8b893cec02f42b0e2abbe7ae5c6077ca1e5d8e1426cad5621e93e24'}
send: 'POST /services/Cloud/ HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 42\r\nUser-Agent: Boto/2.35.0 Python/2.7.5 Linux/3.17.7-200.fc20.x86_64\r\nHost: 172.16.40.26:8773\r\nX-Amz-Date: 20150109T131432Z\r\nContent-Type: application/x-www-form-urlencoded; charset=UTF-8\r\nAuthorization: AWS4-HMAC-SHA256 Credential=6d8332aeeeb94e11bb23d4fc09c0a0f3/20150109/16/172/aws4_request,SignedHeaders=host;x-amz-date,Signature=2dfa2098a8b893cec02f42b0e2abbe7ae5c6077ca1e5d8e1426cad5621e93e24\r\n\r\nAction=DescribeKeyPairs&Version=2010-08-31'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: text/xml
header: Content-Length: 203
header: Date: Fri, 09 Jan 2015 13:14:32 GMT
2015-01-09 13:14:32,365 euca2ools [DEBUG]:Response headers: [('date', 'Fri, 09 Jan 2015 13:14:32 GMT'), ('content-length', '203'), ('content-type', 'text/xml')]
2015-01-09 13:14:32,366 euca2ools [DEBUG]:<?xml version="1.0"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>Signature not provided</Message></Error></Errors><RequestID>req-5e70be08-7c34-4cf7-84f3-e907a7c4765c</Re...

[Steven Hardy (shardy)]

Note it should be possible for nova to support v4 signatures via a similar method to heat:

https://github.com/openstack/heat/blob/master/heat/api/aws/ec2token.py#L87

We detect the format and extract the signature from the header where required, before validating with keystone (which already knows how to validate v4 signatures)

https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/ec2/utils.py#L42

[Sean Dague (sdague)]

From a triage perspective this is clearly a Nova bug. And the real question is whether this is going to get fixed or not. If not, I think we need to consider whether it makes any sense to have the EC2 code in tree any more.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/146124

[Steven Hardy (shardy)]

Unassigned myself for now as my patch is more an example than a complete fix atm.

[Steven Hardy (shardy)]

Ok, patch now works, possibly needs some tests unless testing by unpinning boto is viewed as sufficient

[Matt Riedemann (mriedem)]

Need to update the e-r query for the nova unit test failures with ec2:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwic2VsZi5lYzIuY3JlYXRlX2tleV9wYWlyKGtleW5hbWUpXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIgQU5EIGJ1aWxkX25hbWU6XCJnYXRlLW5vdmEtcHl0aG9uMjdcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyMDgzMTYyOTIyOX0=

[Kieran Spear (kspear)]

Note that for Icehouse the Keystone side requires this keystoneclient commit to work properly:

https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=cf5e45dd5b1ae9b98698a05e7d39989b6bfd4747

It was first released in python-keystoneclient 0.11.0.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/146124
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f7b1af9e13df728d086047f6763bd98cb2cad1b2
Submitter: Jenkins
Branch: master

commit f7b1af9e13df728d086047f6763bd98cb2cad1b2
Author: Steven Hardy <email address hidden>
Date: Fri Jan 9 16:05:51 2015 +0000

    Make ec2 auth support v4 signature format

    Extract the signature and access key via whatever method is needed
    for the version of the request (e.g headers for v4), and add the
    headers and hashed body, which is required for keystone to calculate
    the correct v4 signature when validating the request.

    Change-Id: I161eccc4ea48a21a80d689f6a328ca95cace2e6e
    Closes-Bug: #1408987

[Matt Riedemann (mriedem)]

Uncapping requirements on master: https://review.openstack.org/#/c/146592/1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/146609

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/146609
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=56e3dd3331a6dbece1f12ce8a01edc9b2f6b5dde
Submitter: Jenkins
Branch: stable/juno

commit 56e3dd3331a6dbece1f12ce8a01edc9b2f6b5dde
Author: Steven Hardy <email address hidden>
Date: Fri Jan 9 16:05:51 2015 +0000

    Make ec2 auth support v4 signature format

    Extract the signature and access key via whatever method is needed
    for the version of the request (e.g headers for v4), and add the
    headers and hashed body, which is required for keystone to calculate
    the correct v4 signature when validating the request.

    Conflicts:
            nova/api/ec2/__init__.py

    Change-Id: I161eccc4ea48a21a80d689f6a328ca95cace2e6e
    Closes-Bug: #1408987
    (cherry picked from commit f7b1af9e13df728d086047f6763bd98cb2cad1b2)

[Matt Riedemann (mriedem)]

This is still broken on master, when I tried to uncap the boto version on master I get new auth failures:

http://logs.openstack.org/92/146592/1/check/check-tempest-dsvm-full/7c375f8/console.html#_2015-01-12_19_11_36_102

2015-01-12 19:11:36.102 | tempest.thirdparty.boto.test_ec2_security_groups.EC2SecurityGroupTest.test_create_authorize_security_group
2015-01-12 19:11:36.102 | ----------------------------------------------------------------------------------------------------------
2015-01-12 19:11:36.102 |
2015-01-12 19:11:36.102 | Captured traceback:
2015-01-12 19:11:36.102 | ~~~~~~~~~~~~~~~~~~~
2015-01-12 19:11:36.103 | Traceback (most recent call last):
2015-01-12 19:11:36.103 | _StringException: Empty attachments:
2015-01-12 19:11:36.103 | stderr
2015-01-12 19:11:36.103 | stdout
2015-01-12 19:11:36.103 |
2015-01-12 19:11:36.103 | pythonlogging:'': {{{
2015-01-12 19:11:36.103 | 2015-01-12 19:07:12,279 27381 DEBUG [keystoneclient.auth.identity.v2] Making authentication request to http://127.0.0.1:5000/v2.0/tokens
2015-01-12 19:11:36.103 | 2015-01-12 19:07:13,359 27381 ERROR [boto] 401 Unauthorized
2015-01-12 19:11:36.103 | 2015-01-12 19:07:13,359 27381 ERROR [boto] <?xml version="1.0"?>
2015-01-12 19:11:36.103 | <Response><Errors><Error><Code>AuthFailure</Code><Message>Unauthorized</Message></Error></Errors><RequestID>req-81391f74-7caf-42a6-a3b8-ccd2c7d1cbdf</RequestID></Response>
2015-01-12 19:11:36.104 | }}}
2015-01-12 19:11:36.104 |
2015-01-12 19:11:36.104 | Traceback (most recent call last):
2015-01-12 19:11:36.104 | File "tempest/thirdparty/boto/test_ec2_security_groups.py", line 32, in test_create_authorize_security_group
2015-01-12 19:11:36.104 | group_description)
2015-01-12 19:11:36.104 | File "tempest/services/botoclients.py", line 84, in func
2015-01-12 19:11:36.104 | return getattr(conn, name)(*args, **kwargs)
2015-01-12 19:11:36.104 | File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 3003, in create_security_group
2015-01-12 19:11:36.104 | SecurityGroup, verb='POST')
2015-01-12 19:11:36.105 | File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1207, in get_object
2015-01-12 19:11:36.105 | raise self.ResponseError(response.status, response.reason, body)
2015-01-12 19:11:36.105 | EC2ResponseError: EC2ResponseError: 401 Unauthorized
2015-01-12 19:11:36.105 | <?xml version="1.0"?>
2015-01-12 19:11:36.105 | <Response><Errors><Error><Code>AuthFailure</Code><Message>Unauthorized</Message></Error></Errors><RequestID>req-81391f74-7caf-42a6-a3b8-ccd2c7d1cbdf</RequestID></Response>

It's something to do with security groups this time.

http://logs.openstack.org/92/146592/1/check/check-tempest-dsvm-full/7c375f8/logs/screen-n-api.txt.gz#_2015-01-12_19_07_13_357

2015-01-12 19:07:13.357 24624 DEBUG nova.api.ec2.faults [-] EC2 error response: AuthFailure: Unauthorized ec2_error_response /opt/stack/new/nova/nova/api/ec2/faults.py:29

[Matt Riedemann (mriedem)]

New bug 1410622 reported.

[Matt Riedemann (mriedem)]

boto should be capped on stable/juno with this now: https://review.openstack.org/#/c/146344/

[Matt Riedemann (mriedem)]

The boto version is already capped on stable/icehouse now so there won't be a problem on icehouse:

https://github.com/openstack/requirements/commit/f6e01ece0666521580f674a3b1ddc30a3021ec85

[Matt Riedemann (mriedem)]

Also capped on stable/juno now: https://review.openstack.org/#/c/146344/

So the stable branches should be safe for now.

[nikeshmahalka (nikesh-mahalka)]

below tempest test case is failing on lvm in kilo devstack

======================================================================
FAIL: tearDownClass
(tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest)
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/test.py", line 301, in tearDownClass
    teardown()
  File "/opt/stack/tempest/tempest/thirdparty/boto/test.py", line 272,
in resource_cleanup
    raise exceptions.TearDownException(num=fail_count)
TearDownException: 1 cleanUp operation failed

How to fix this?