Different heat stack-list results on Newton

[Franciraldo Cavalcante]

   Until Heat/Mitaka, heat stack-list for an user with admin privileges would show only the stacks created for his project(tenant). On Newton, we see the stacks from all tenants. Also: there doesn't seem to be a way to filter the stacks by tenant for an admin user.

   This behavior change is not documented on the Heat/Newton release notes, and can be found by running either 'heat stack-list' or 'openstack stack list'.

   If this is a desired change of behavior, then we need a new flag to be able to specify the project. If it's not, and the existence of (-g, --global-tenant) flag leads me to think it's the case, then we need a fix for the default behavior.

[Thomas Herve (therve)]

It was probably introduced by https://review.openstack.org/#/c/344086/. I don't know the details of that change, but I think the reason is that the admin role is somewhat global anyway.

That said, I don't exactly reproduce the behavior you're describing. Here's an output of local testing:

$ source openrc admin demo
$ openstack stack list -f value -c 'Stack Name'
stack 1
$ source openrc admin admin
$ openstack stack list -f value -c 'Stack Name'
stack1
stack2
$ source openrc admin alt_demo
$ openstack stack list -f value -c 'Stack Name'
stack2

Switching projects seemed filter properly, but we see all stacks when on the admin project. Maybe it changed the behavior on the admin project?

[Franciraldo Cavalcante (c-franciraldo-cavalcante)]

Hi Herve,

Yes, the modification seems to be in the default behavior of admin.

Specifically, I use the admin user sentinel:
# openstack user role list sentinel
+----------------------------------+-------------------------+----------+----------+
| ID | Name | Project | User |
+----------------------------------+-------------------------+----------+----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | sentinel | sentinel |
| b8325cc94abf43cfab8eff0a149353a4 | ProjectOwner | sentinel | sentinel |
| 7086479e48a24838ba0814100c679131 | heat_project_admin | sentinel | sentinel |
| 71f12dfa07d145b48d013efa657a1ecf | admin | sentinel | sentinel |
| 092e503c59594f8186247cf86a26426c | swift_project_admin | sentinel | sentinel |
| 5526a201d100427dbb1133284aa56f40 | designate_project_admin | sentinel | sentinel |
+----------------------------------+-------------------------+----------+----------+

After the newton upgrade, the result of a stack list is:
# openstack stack list -f value -c 'Stack Name'
bfd02-sentinel-001
bfd02-sentinel-trove-005
bfd02-sentinel-trove-004
bfd02-sentinel-trove-003
bfd02-sentinel-006
bfd02-sentinel-005
bfd02-sentinel-004
bfd02-sentinel-002
bfd02-sentinel-003
cv_bfd02_servers-cvmgmt
cv_bfd02_servers-cvdata
cv_bfd02_ports
elk-kibanas
elk-datas
elk-masters
...

And it used to only show the stacks that are created on the sentinel project, the ones whose name include sentinel. None of the other stacks is part of sentinel project.

[Franciraldo Cavalcante (c-franciraldo-cavalcante)]

   I would expect that the default behavior would be even for an admin user, to only show the stacks in it's project. And you pass the --global-tenant flag if you want to see results across different tenants.

[Rabi Mishra (rabi)]

Yes, this has been changed in newton to fix bug 1466694. So in essence user having admin role is admin project can list/update/delete stacks from other projects and this is expected behaviour.

[Franciraldo Cavalcante (c-franciraldo-cavalcante)]

   If this is the new default behavior, then the 'heat stack-list' command should be modified to support specifying a project. Like the Neutron commands, that shows results across projects, if run as admin, but provide a flag to specify a project from cmd line (like --tenant-id TENANT_ID).

[Rabi Mishra (rabi)]

Not sure I understand the need here.

Stack operations are always project scoped in heat. The only difference is user with admin role in admin tenant is treated as a super user across the projects and can operate on stacks of all projects.

I think you can use any other user(without admin role) in admin project to get the stacks for that project like any other project.

Also, users with admin role in any other project(other than admin) can access stacks of that project only.

[Franciraldo Cavalcante (c-franciraldo-cavalcante)]

This is not what I see in the example I pasted above. The user is called sentinel and the project is also sentinel. sentinel user has admin role. When I list the stacks on Heat/Newton, these include stacks that are not in sentinel project.

[Rabi Mishra (rabi)]

Assuming that you're using keystone v3, can you please check what 'admin_project_name' and 'admin_project_domain_name' set to in your keystone.conf?

[Franciraldo Cavalcante (c-franciraldo-cavalcante)]

# egrep 'admin_project_name|admin_project_domain_name' keystone.conf
# Name of the domain that owns the `admin_project_name`. Defaults to None.
#admin_project_domain_name = <None>
#admin_project_name = <None>

[Rabi Mishra (rabi)]

Yeah, you have to set the admin_project_name. By default it falls back to True for all projects for backward compatibility.

[resource]
admin_project_name = admin
admin_project_domain_name = Default

You can check bug 968696 for further details on the admin scoping discussion.

[Franciraldo Cavalcante (c-franciraldo-cavalcante)]

Rabi,

   Thanks for your help. I've tested these parameters on our system, but still seeing same faulty behavior. We use same values (admin, Default). Could there be other configuration parameters on Keystone, or Heat, related to this?

[Rabi Mishra (rabi)]

Do you have a customized etc/heat/policy.json? Does your heat policy.json have this?
https://github.com/openstack/heat/blob/master/etc/heat/policy.json#L2

[Rabi Mishra (rabi)]

I've not heard back from the reporter, removing the milestone.

[Matt Fischer (mfisch)]

After making all the changes to the policy file...

I dug into this more today with Franza and I think the issue may in fact be in oslo.context. With all the settings configured as stated, I can validate a token using the "bob" project (which is not defined as the admin project) and can confirm that the is_admin_project is not present in the validated token reply.

However, once I'm in the heat container, I can look at the context we get from oslo.context and I see the following:

(Pdb) print req.context.is_admin
True
(Pdb) print req.context.is_admin_project
True

(Pdb) print req.context.to_dict().keys()['tenant']
bob

The full context is posted here: http://paste.openstack.org/show/590999/, you can see that there's no is "is_admin_project" in the token reply, yet oslo.context still sets is_admin_project to true.

Relevant keystone.conf settings we are using:

# Name of the domain that owns the `admin_project_name`. Defaults to None.
# (string value)
#admin_project_domain_name = <None>
admin_project_domain_name = Default

# Special project for performing administrative operations on remote services.
# Tokens scoped to this project will contain the key/value
# `is_admin_project=true`. Defaults to None. (string value)
#admin_project_name = <None>
admin_project_name = admin

[Matt Fischer (mfisch)]

possible dupe of https://bugs.launchpad.net/keystone/+bug/1577996

[Franciraldo Cavalcante (c-franciraldo-cavalcante)]

I've implemented the changes to keystone.conf and heat/policy.json file and upgraded Keystone to Newton. These settings resolve our problem. @Rabi we can close the issue. Thanks a lot for your help.

[Matt Fischer (mfisch)]

Would like to leave this here in case other folks run into this issue. If you run Keystone Mitaka you WILL hit this issue due to Bug #1577996. The fix was not backported. In that case is_admin_project is ALWAYS set in the context due to that bug. Keystone Newton + Heat Newton should work fine as long as the settings are set for admin_project.

[Launchpad Janitor (janitor)]

[Expired for heat because there has been no activity for 60 days.]