Heat CFN signals do not pass authorization

[Miguel Grinberg]

Note that this bug applies to the Kilo release. Master does not appear to have this problem. I did not test liberty yet.

Heat templates that rely on CFN signals timeout because the API calls that execute these signals return 403 errors. Heat signals, on the other side, do work.

The problem was reported to me by Alex Cantu. I have verified it on his multinode lab and have also reproduced on my own single-node system hosted on a public cloud server. I suspect liberty/master avoided the problem after Jesse and I reworked the Heat configuration to use Keystone v3 the last day before the L release.

Example template, which can be executed in an AIO after running the tempest playbook:

    heat_template_version: 2013-05-23

    resources:
      wait_condition:
        type: AWS::CloudFormation::WaitCondition
        properties:
          Handle: { get_resource: wait_handle }
          Count: 1
          Timeout: 600

      wait_handle:
        type: AWS::CloudFormation::WaitConditionHandle

      my_instance:
        type: OS::Nova::Server
        properties:
          image: cirros
          flavor: m1.tiny
          networks:
            - network: "private"
          user_data_format: RAW
          user_data:
            str_replace:
              template: |
                #!/bin/sh
                echo "wc_notify"
                curl -H "Content-Type:" -X PUT wc_notify --data-binary '{"status": "SUCCESS"}'
              params:
                wc_notify: { get_resource: wait_handle }

This template should end very quickly, as it starts a cirros instance that just sends a signal back to heat. But instead, it timeouts. The user data script dumps the signal URL to the console log, if you then try to send the signal manually you will get a 403. The original 403 can also be seen in the heat-api-cfn.log file. Here is the log snippet:

2015-11-12 05:13:34.491 1862 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2015-11-12 05:13:34.492 1862 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2015-11-12 05:13:34.493 1862 INFO heat.api.aws.ec2token [-] Authenticating with http://172.29.236.100:5000/v3/ec2tokens
2015-11-12 05:13:34.533 1862 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2015-11-12 05:13:34.534 1862 INFO eventlet.wsgi.server [-] 10.0.3.181,172.29.236.100 - - [12/Nov/2015 05:13:34] "PUT /v1/waitcondition/arn%3Aopenstack%3Aheat%3A%3A683acadf4d04489f8e991b44014e6fc1%3Astacks%2Fwc1%2Faa4083b6-ce6c-411f-9df9-d059abacf40c%2Fresources%2Fwait_handle?Timestamp=2015-11-12T05%3A12%3A27Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=65657d1021e24e49ba4fb6f217ca4a22&SignatureVersion=2&Signature=aCG%2FO04MNLzSlf5gIBGw1hMcC7bQzB3pZXVKzXLLNSo%3D HTTP/1.1" 403 301 0.043961

For reference, the curl command to trigger the signal is: curl -H "Content-Type:" -X PUT "<cfn-signal-url".

[yanyao (yanwei-yao)]

I also met the same error(in kilo branch). keystone.log indicated it's 'fernet' related.

After I changed variable 'keystone_token_provider' to 'keystone.token.providers.uuid.Provider'. The issue resolved.

[Dolph Mathews (dolph)]

yanyao: What did the keystone logs say to indicate a relationship to Fernet?

[Dolph Mathews (dolph)]

Backtrace in keystone's logs (thanks to Miguel!):

2015-11-12 05:13:34.503 4664 INFO keystone.common.wsgi [-] POST http://172.29.236.100:5000/v3/ec2tokens
2015-11-12 05:13:34.528 4664 ERROR keystone.common.wsgi [-] 'NoneType' object has no attribute 'get'
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi Traceback (most recent call last):
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 238, in __call__
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi result = method(context, **params)
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/keystone/contrib/ec2/controllers.py", line 397, in authenticate
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi metadata_ref=metadata_ref)
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/keystone/token/provider.py", line 344, in issue_v3_token
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi parent_audit_id)
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/keystone/token/providers/fernet/core.py", line 184, in issue_v3_token
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi if auth_context.get('bind'):
2015-11-12 05:13:34.528 4664 TRACE keystone.common.wsgi AttributeError: 'NoneType' object has no attribute 'get'

[Lance Bragstad (lbragstad)]

I've posted a backport to stable/kilo that passes tests locally and might address the issue [0].

[0] https://review.openstack.org/#/c/244853/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244853
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=f66badae63b1544c3bcc83e0c1afc8e9cdc71bab
Submitter: Jenkins
Branch: stable/kilo

commit f66badae63b1544c3bcc83e0c1afc8e9cdc71bab
Author: Lance Bragstad <email address hidden>
Date: Mon Jun 29 18:12:37 2015 +0000

    Consolidate the fernet provider issue_v3_token()

    When the Fernet token provider was implemented, it extended the
    provider.common.py:BaseProvider class. It also overrode most all common methods
    the BaseProvider implemented. Other token providers in Keystone (like the UUID
    an PKI providers) just implemente a _get_token_id method because token ids may
    be different across providers.

    This commit removes the issue_v3_token() method from the fernet.Provider. This
    ensure that the Fernet provider uses the same issue_v3_token() that all the
    other token providers use.

    Subsequent patches will do the same for issue_v2_token, validate_v3_token, and
    validate_v2_token.

    Conflicts:
     keystone/token/providers/fernet/core.py

    Closes-Bug: #1515485
    Change-Id: I03f56c9c84389a6d6cdb3a6863fcbfca486af337
    (cherry picked from commit 91a0b29809cb71c1b2df1642d6c34a3f60a801ed)

[Jesse Pretorius (jesse-pretorius)]

OpenStack-Ansible included this fix in the general SHA update: https://review.openstack.org/245968