_build_deployments_metadata fails with context provided to describe_stack_resource

Bug #1290621 reported by Steve Baker
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Steve Baker

Bug Description

After applying the fix for bug 1288523, os-collect-config polling now causes the following error

2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/engine/service.py", line 858, in describe_stack_resource
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp return api.format_stack_resource(stack[resource_name])
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/engine/api.py", line 118, in format_stack_resource
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp api.RES_METADATA: resource.metadata,
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/engine/resources/server.py", line 431, in metadata
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp return self._build_deployments_metadata()
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/engine/resources/server.py", line 368, in _build_deployments_metadata
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp self.heat(), self.resource_id)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/engine/resource.py", line 385, in heat
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp return self.stack.clients.heat()
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/engine/clients.py", line 322, in heat
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp endpoint_type=endpoint_type)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/engine/clients.py", line 96, in url_for
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp return self.keystone().url_for(**kwargs)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/common/heat_keystoneclient.py", line 517, in url_for
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp return self.client_v3.service_catalog.url_for(**kwargs)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/common/heat_keystoneclient.py", line 108, in client_v3
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp self._client_v3 = self._v3_client_init()
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/heat/heat/common/heat_keystoneclient.py", line 168, in _v3_client_init
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp client_v3.authenticate()
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/python-keystoneclient/keystoneclient/httpclient.py", line 383, in authenticate
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp resp = self.get_raw_token_from_identity_service(**kwargs)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/python-keystoneclient/keystoneclient/v3/client.py", line 167, in get_raw_token_from_identity_service
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp return a.get_auth_ref(self.session)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/python-keystoneclient/keystoneclient/auth/identity/v3.py", line 106, in get_auth_ref
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp authenticated=False)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/python-keystoneclient/keystoneclient/session.py", line 268, in post
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp return self.request(url, 'POST', **kwargs)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp File "/home/steveb/dev/localstack/python-keystoneclient/keystoneclient/session.py", line 209, in request
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp raise exceptions.from_response(resp, method, url)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp Unauthorized: Could not find project, e62854ab473544d7bc1710f48e1d8071-a. (HTTP 401)
2014-03-11 13:38:23.845 TRACE heat.openstack.common.rpc.amqp

This is the same stack as for bug 1288523, so e62854ab473544d7bc1710f48e1d8071 is the user's tenant, 523de76d5b9342288143eac4b3e88058 is the stack_user_project_id

summary: - asfd
+ _build_deployments_metadata fails with context provided to
+ describe_stack_resource
Changed in heat:
status: New → Triaged
importance: Undecided → High
milestone: none → icehouse-rc1
Revision history for this message
Steven Hardy (shardy) wrote :

> Unauthorized: Could not find project, e62854ab473544d7bc1710f48e1d8071-a. (HTTP 401)

Can you confirm, is this the stack owner's project, or the stack-domain-user project ID?

It looks like it may be trying to get a token scoped to the owners project, which should not be allowed since the point of domain isolated users is they can talk to heat without having the permissions of the stack owner.

Can you provide a sample template so I can reproduce please?

Changed in heat:
assignee: nobody → Steven Hardy (shardy)
Revision history for this message
Steven Hardy (shardy) wrote :

Steve, I've carried out some basic tests on a fresh devstack install, and AFAICS the instance-users side of this is working as expected.

I created a stack:

heat_template_version: 2013-05-23

resources:
  user:
    type: AWS::IAM::User

  access:
    type: AWS::IAM::AccessKey
    properties:
      UserName: { get_resource : user }

  instance:
    type: OS::Nova::Server
    properties:
      image: fedora-20.x86_64
      flavor: m1.small
      key_name: stack_key

outputs:
  user:
    description: user
    value: { get_resource : user }
  access:
    description: access
    value: { get_resource : access}
  secret:
    description: secret
    value: { get_attr: [access, SecretAccessKey] }

Then used the access/secret to set credentials in /etc/boto.cfg and polled the metadata for the instance via heat-boto:
# /usr/bin/heat-boto resource ua1 instance
SHDEBUG scriptname=heat-boto
StackId : arn:openstack:heat::828f06e27eb94b1e920f047fd24dbfa2:stacks/ua1/eb0770d7-28d7-475a-9a57-8865f1d1f530
ResourceStatus : CREATE_COMPLETE
Description :
ResourceType : OS::Nova::Server
ResourceStatusReason : state changed
LastUpdatedTimestamp : 2014-03-11T22:29:49Z
StackName : ua1
PhysicalResourceId : 7a3199d2-f62a-49d7-80a2-33629dc519d2
LogicalResourceId : instance
Metadata : {}

So I think this proves that the credentials are working, and you are hitting an issue (in one of the new software config resources?) where the metadata-poll path is trying to create a client connection, which by design the domain-isolated instance-user credentials cannot do.

Previously this would have worked because we had no isolation between the instance-user credentials and the stack owner (because they were in the same project), but now any compromised instance which e.g tried to get a token to launch or modify nova instances would be denied - as, I think, we are seeing here.

If you can provide a software-config template example I'm happy to dig a bit deeper, but it seems like we need to ensure the client connections are attempted in appropriately lazy fashion that this doesn't happen when polling for metadata from the DB?

Changed in heat:
status: Triaged → Incomplete
Changed in heat:
status: Incomplete → In Progress
assignee: Steven Hardy (shardy) → Steve Baker (steve-stevebaker)
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I'm guessing you'll trigger this bug by setting
      user_data_format: SOFTWARE_CONFIG
on your instance resource. (I have a fix coming btw)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/79824

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/80867

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/80867
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=42a9e22ae570e1266b7ff64e2698a27b55bcad84
Submitter: Jenkins
Branch: master

commit 42a9e22ae570e1266b7ff64e2698a27b55bcad84
Author: Steve Baker <email address hidden>
Date: Mon Mar 17 10:07:22 2014 +1300

    Never specify project_name in keystone v3 auth

    All of our middlewares populate the context with both the tenant name and the
    tenant_id. The tenant_id, is unique across domains which means you can do lookups
    without having to know what domain the project is in.

    When project_id and project_name are both specified, it seems that project_name
    takes precedence, which will fail when the credentials are from the stack domain since
    we cannot lookup by project names when the domain is not specified.

    It looks like specifying project_name is never the right thing to do, and we should
    assume that the context always has a tenant_id.

    Change-Id: Ifea21e13be482a6347438d6baaca3111caad4a97
    Closes-Bug: #1290621

Changed in heat:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.