admin role can't delete other tenants stacks

[Sam Morrison]

With the admin role you can't show or delete or do anything with other tenants stacks.

I see multiple places where this is prevented in the API level and in the DB level.

[Rico Lin (rico-lin)]

IIUC, The RBAC designed to prevents any non authorized actions. Admin role should not get access right from another project which it not fully authorized with. It's true that current admin role by default still been consider as overall administrator cross all projects, but it's not the best authorization design, and will be improved some day.
IMO, if you try to show or delete stacks from one tenant, then just give yourself the rights from that tenant and do what ever you can.

[Sam Morrison (sorrison)]

Hi Rico,

That isn't really a viable solution. As an operator of a large cloud I need a way of deleting and showing other tenants stacks.

Heat is extremely unfriendly to operators and needs to be addressed.

[Kairat Kushaev (kkushaev)]

Ok, will analyse additionally what can we do here.

[Steve Baker (steve-stevebaker)]

Our default policy.json doesn't make assumptions about the privilege of the admin role. Operators are expected to customise their /etc/heat/policy.json to meet their needs.

Can you start with a customised policy.json with appropriate rule:context_is_admin then raise specific bugs for operations that you can't perform?

Also you should get some advice from the rackspace heat team (like Randall Burt) who need to perform similar actions.

[Sam Morrison (sorrison)]

Stack show is one example.

In policy.json I add:

"resource:show": "rule:context_is_admin",

As an admin I am able to do

heat stack-list -g

This lists all stacks

however if I do a

heat stack-show <UUID>

I get:

Stack not found: <UUID>

[Sam Morrison (sorrison)]

Wonder if this bug can get any traction, it's a bit of a operational headache. We have lots of users with un deletable stacks and only way for us to delete them is nasty SQL.

[Pavlo Shchelokovskyy (pshchelo)]

Exactly what Rico said - as you are an admin, you have control over users in projects. Add admin user to the project affected and than admin user have full access to resources of that project.

[Sam Morrison (sorrison)]

I'm sorry that is not an acceptable solution. For one it's not consistent with other openstack projects. Also adding users to a project just to do a delete or a show isn't feasible in a large cloud. Our support team don't have access to manage users and projects and we don't want to allow this. I can give more reasons if you need.

[Steven Hardy (shardy)]

Ok, so we had some discussion on this topic at summit (Sam was that you in the session? Apologies, I had to leave before the end so didn't catch you to discuss), and also has just been mentioned on the ML:

http://lists.openstack.org/pipermail/openstack-dev/2015-November/078846.html

In summary, this is essentially by design all requests to heat have always been scoped to a project, but we can look more closely at providing a solution here.

The main problem with comparing to other projects is several projects bake in assumptions about scope of admin-ness, ref the long-standing bug #968696. We have tried hard to explicitly avoid doing stuff like using a global "is admin" flag because then we get added to the list of projects on that hard-to-fix bug and become part of that long standing problem.

There's really no need for an admin to create any additional users, all they need to do is get a token scoped to the correct project, then they should have full control of all resources in that project, so while I appreciate creating users isn't acceptable, I always assumed an admin would be OK with switching tokens to gain access to a specific project.

If that's a bad assumption, then I guess we have to look at how we enable an (optional disabled by default) policy.json definition which describes global admin-ness, but attempt to do it in a way which doesn't make bug #968696 even harder to solve.

The other aspect of this is the risk involved with any such global flag - we currently have per-project scoping wired in at the DB layer inside heat, and any global flag that defeats it does increase the risk that a small programming error will cause a very large vulnerability that affects all projects. Clearly tests can mitigate this, but it's another reason why we've so far resisted the "global admin" thing, and hoped that having API requests scoped to exactly one project (e.g what we get after token validation from the keystone auth_token middleware) would be sufficient.

[Steven Hardy (shardy)]

Triaged as this issue is vaid, priority undecided pending further discussion

[Steve Baker (steve-stevebaker)]

On 10/11/15 05:46, Steven Hardy wrote:
>
> There's really no need for an admin to create any additional users, all
> they need to do is get a token scoped to the correct project, then they
> should have full control of all resources in that project, so while I
> appreciate creating users isn't acceptable, I always assumed an admin
> would be OK with switching tokens to gain access to a specific project.
>
The impression I got from Sam during the heat operators session was that
the target user for this is front-line support staff, so the solution
needs to not involve complicated steps.

The first thing we need to know is whether support staff are doing these
stack operations through horizon or on the cli. For the latter I assume
we can come up with some form of wrapper script to do the token switch.

[Sam Morrison (sorrison)]

Usually from the CLI.

I think the main thing is that the policy.json file doesn't work as expected and is not the same as other projects. If I set in the policy file that some role can see any stack that should work. I think all policy should be in the policy level and be stripped out of the DB level. Nova went through the same issue where that had policy checks at the policy layer and the DB layer.

Let operators decide what they want to do and give them the power to do this via the policy.json file. If they want to set a global admin let them to that.

[Steven Hardy (shardy)]

Ok, so in terms of the CLI wrapper:

-bash-4.3$ . ~/devstack/openrc admin admin
-bash-4.3$ env | grep OS_
OS_REGION_NAME=RegionOne
OS_IDENTITY_API_VERSION=2.0
OS_PASSWORD=foobar
OS_AUTH_URL=http://192.168.1.17:5000/v2.0
OS_USERNAME=admin
OS_TENANT_NAME=admin
OS_VOLUME_API_VERSION=2
OS_NO_CACHE=1

Isn't it just a case of doing OS_TENANT_NAME=the_tenant heat stack-delete <id of stack> ?

If that works then it's trivial to wrap a tiny script which grabs the project from the stack-list -g and switches to the correct scope before making any further request?

But I acknowledge the desire to have full control over the policy file, and I want us to be responsive to operator requirements, I'm just worried about the bug #968696 impact of any such change.

I'll take this bug and look into possible fixes, and start a ML thread to seek guidance from the keystone folks wrt the best-practice way to implement this in terms of default policy rules.

[Steven Hardy (shardy)]

ML thread started:

http://lists.openstack.org/pipermail/openstack-dev/2015-November/079006.html

[Bruno Bompastor (bruno-bompastor)]

I agree with the points raised by Sam. For example at CERN we have more than 2300 tenants and it is not a solution to scope the heat operations for every support case that we have.

Also using os-tenant-name implies that the admin user needs to have access to the user tenant.

We are using the cli (like in nova) to do these operations and some solution around the policy.json would be appreciated.
I saw from the ML thread that the keystone folks have a workaround for his...

[Steven Hardy (shardy)]

Ok, so looping back to this (apologies for the delay), here's the response from ayoung on the ML:

"""
We are going to add a value to the Keystone token validation response
that will indicate that the proejct is an admin project. Use that.
Don't develop something for Mitaka that does not use that.

The spec is here

https://review.openstack.org/#/c/242232/

And, as you can see, Henry and I are working on getting it tight enough
to stand the test of time.

There is a sample implementation underway here:

https://review.openstack.org/240719
"""

It would appear that the implementation has now landed, so we can go ahead and figure out how to wire that in to heat.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/316627

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/320236

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/316627
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ac86702172ddf01f5bdc3f3cd99d2e32ad9b7024
Submitter: Jenkins
Branch: master

commit ac86702172ddf01f5bdc3f3cd99d2e32ad9b7024
Author: rabi <email address hidden>
Date: Mon May 16 10:31:10 2016 +0530

    Authorize super admin actions on all projects

    This allows admin super user (user with admin role in admin_project)
    to do stack operations across all projects.

    Change-Id: Ifbf56fde02b89248ee788e6a212ef9d11e665dc0
    Partial-Bug: #1466694

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/320236
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3b0f8ad21f83f294b9fe7a04e20aadd487a441ec
Submitter: Jenkins
Branch: master

commit 3b0f8ad21f83f294b9fe7a04e20aadd487a441ec
Author: rabi <email address hidden>
Date: Tue May 24 10:18:33 2016 +0530

    Add integration tests for admin actions

    Add integration tests for super admin actions.

    Change-Id: I277f23800274729cf7128fdcb4521e4b221233cf
    Partial-Bug: #1466694

[Rabi Mishra (rabi)]

Waiting for oslo patches to land before submitting the reverted patch again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/333308
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=168a7797e5e7643141d81420ae248fbbbef6f9ef
Submitter: Jenkins
Branch: master

commit 168a7797e5e7643141d81420ae248fbbbef6f9ef
Author: Rabi Mishra <email address hidden>
Date: Fri Jul 1 10:38:28 2016 +0530

    Use is_admin_project from context

    Now that oslo.context has been bumped to >=2.6.0,
    we can use `is_admin_project` from the context which
    is backward compatible.

    This also adds a new rule `project_admin` to make
    resource types accessible inline with current policy
    of other services like nova, that are yet to use the
    `is_admin_project` feature. Once those services start
    using the is_admin_project feature, we can remove this.

    Change-Id: I5be8176042f8839e86f77984222e7fac66dfaed6
    Related-Bug: #1466694