Comment 2 for bug 1290621

Steve, I've carried out some basic tests on a fresh devstack install, and AFAICS the instance-users side of this is working as expected.

I created a stack:

heat_template_version: 2013-05-23

resources:
  user:
    type: AWS::IAM::User

  access:
    type: AWS::IAM::AccessKey
    properties:
      UserName: { get_resource : user }

  instance:
    type: OS::Nova::Server
    properties:
      image: fedora-20.x86_64
      flavor: m1.small
      key_name: stack_key

outputs:
  user:
    description: user
    value: { get_resource : user }
  access:
    description: access
    value: { get_resource : access}
  secret:
    description: secret
    value: { get_attr: [access, SecretAccessKey] }

Then used the access/secret to set credentials in /etc/boto.cfg and polled the metadata for the instance via heat-boto:
# /usr/bin/heat-boto resource ua1 instance
SHDEBUG scriptname=heat-boto
StackId : arn:openstack:heat::828f06e27eb94b1e920f047fd24dbfa2:stacks/ua1/eb0770d7-28d7-475a-9a57-8865f1d1f530
ResourceStatus : CREATE_COMPLETE
Description :
ResourceType : OS::Nova::Server
ResourceStatusReason : state changed
LastUpdatedTimestamp : 2014-03-11T22:29:49Z
StackName : ua1
PhysicalResourceId : 7a3199d2-f62a-49d7-80a2-33629dc519d2
LogicalResourceId : instance
Metadata : {}

So I think this proves that the credentials are working, and you are hitting an issue (in one of the new software config resources?) where the metadata-poll path is trying to create a client connection, which by design the domain-isolated instance-user credentials cannot do.

Previously this would have worked because we had no isolation between the instance-user credentials and the stack owner (because they were in the same project), but now any compromised instance which e.g tried to get a token to launch or modify nova instances would be denied - as, I think, we are seeing here.

If you can provide a software-config template example I'm happy to dig a bit deeper, but it seems like we need to ensure the client connections are attempted in appropriately lazy fashion that this doesn't happen when polling for metadata from the DB?