tenantId/default_project_id missing on Keystone service user in Mitaka

Bug #1604479 reported by Kam Nasim on 2016-07-19
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Critical
Kam Nasim
puppet-keystone
Undecided
Kam Nasim

Bug Description

On upgrading to Mitaka, we saw that the user ref in Keystone does not have a tenantId or default_project_id field. This breaks:
1) The Detailed view in Horizon in the Identity pane where ProjectID is shown as "None"
2) any services project based RBAC policies that we have in place.

Noticed a new local_user DB table for all the services users (no project/tenantId field in here):
keystone=# select * from local_user;
 id | user_id | domain_id | name
----+----------------------------------+-----------+------------
  1 | 3c1bd8c0f6324dcc938900d8eb801aa5 | default | admin
  2 | d1c4f7a244f74892b612b9b2ded6d602 | default | neutron
  3 | a481a1f43ec0463083b7a30d20493d38 | default | nova
  4 | 951068b3372f47ac827ade8f67cc19b4 | default | glance
  6 | 4b76763e375946998445b65b11c8db73 | default | ceilometer
  7 | 15c8e1e463cc4370ad369eaf8504b727 | default | cinder
  8 | 5c3ea23eb8e14070bc562951bb266073 | default | sysinv
  9 | 2b62ced877244e74ba90b546225740d0 | default | heat
 10 | 5a506282b45c4064b262f3f414f1f699 | default | kam
(9 rows)

Note that an admin role is assigned for these services users in the services project. It is just not present within the user reference or keystone user-get:

$ keystone user-role-list

+----------------------------------+-------+----------------------------------+----------------------------------+
| id | name | user_id | tenant_id |
+----------------------------------+-------+----------------------------------+----------------------------------+
| f9985117736b4684904b4eb55476f30a | admin | a481a1f43ec0463083b7a30d20493d38 | c211dda10c9a4b2db16f239dccf65acd |
+----------------------------------+-------+----------------------------------+----------------------------------+

$ keystone user-get

+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| email | nova@localhost |
| enabled | True |
| id | a481a1f43ec0463083b7a30d20493d38 |
| name | nova |
| username | nova |
+----------+----------------------------------+

Contrast this to Kilo/Liberty where tenantId is visible within user reference:

$ keystone user-get b7a3bcd588b5482ab9741efcf3f9bb33

+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| email | nova@localhost |
| enabled | True |
| id | b7a3bcd588b5482ab9741efcf3f9bb33 |
| name | nova |
| tenantId | 2e4a21e1a37840879321320107c74f86 | <<<<<<<<<<<<<<<<<<<<<<<<<<
| username | nova |
+----------+----------------------------------+

Revision history for this message
Henry Nash (henry-nash) wrote :

What do you see if you use the openstack cli (remember that the keystone cli is deprecated)?

Revision history for this message
Henry Nash (henry-nash) wrote :

(as an aside, the default_project_id is stored in the users table not the local_users table)

Revision history for this message
Kam Nasim (knasim-wrs) wrote :

same with openstack cli:

# openstack user show a481a1f43ec0463083b7a30d20493d38
+----------+----------------------------------+
| Field | Value |
+----------+----------------------------------+
| email | nova@localhost |
| enabled | True |
| id | a481a1f43ec0463083b7a30d20493d38 |
| name | nova |
| username | nova |
+----------+----------------------------------+

user DB table only lists the postgres user:

keystone=# \dt

                    List of relations
 Schema | Name | Type | Owner
--------+------------------------+-------+----------------
 public | access_token | table | admin-keystone
 public | assignment | table | admin-keystone
 public | config_register | table | admin-keystone
 public | consumer | table | admin-keystone
 public | credential | table | admin-keystone
 public | domain | table | admin-keystone
 public | endpoint | table | admin-keystone
 public | endpoint_group | table | admin-keystone
 public | federated_user | table | admin-keystone
 public | federation_protocol | table | admin-keystone
 public | group | table | admin-keystone
 public | id_mapping | table | admin-keystone
 public | identity_provider | table | admin-keystone
 public | idp_remote_ids | table | admin-keystone
 public | implied_role | table | admin-keystone
 public | local_user | table | admin-keystone
 public | mapping | table | admin-keystone
 public | migrate_version | table | admin-keystone
 public | password | table | admin-keystone
 public | policy | table | admin-keystone
 public | policy_association | table | admin-keystone
 public | project | table | admin-keystone
 public | project_endpoint | table | admin-keystone
 public | project_endpoint_group | table | admin-keystone
 public | region | table | admin-keystone
 public | request_token | table | admin-keystone
 public | revocation_event | table | admin-keystone
 public | role | table | admin-keystone
 public | sensitive_config | table | admin-keystone
 public | service | table | admin-keystone
 public | service_provider | table | admin-keystone
 public | token | table | admin-keystone
 public | trust | table | admin-keystone
 public | trust_role | table | admin-keystone
 public | user | table | admin-keystone
 public | user_group_membership | table | admin-keystone
 public | whitelisted_config | table | admin-keystone
(37 rows)

keystone=# select * from user;
 current_user
--------------
 postgres
(1 row)

Revision history for this message
Henry Nash (henry-nash) wrote :

is the user you are doing show for a postgres user or an LDAP/AD user?

Revision history for this message
Kam Nasim (knasim-wrs) wrote :

it is a services user stored in the SQL identity backend (not using LDAP for service users)

Revision history for this message
Henry Nash (henry-nash) wrote :

(Sorry for all the questions)...and is your client configured to talk v2 or v3 of the Identity API?

Revision history for this message
Henry Nash (henry-nash) wrote :

So for the V2 API, if there is no default project defined for a user, the tenantID attribute will not appear in the returned entity (which is a bit odd, I agree)

Did these users have default projects before the migration? If so, then one thing to look at (if possible) is a dump of the Users table, to see if the entries that match these do or do not have a default_proejct_id defined.

Revision history for this message
Kam Nasim (knasim-wrs) wrote :

Thanks Henry, your responses lended to me some important clues which is if the project is not provided during user creation then the default_project_id mapping will not be added. Subsequently even if a role is granted for this user on said project then also no default_project_id is created.

In our deployment we are using Packstack/Puppet to create the services users.

Mitaka's puppet-keystone (unlike Kilo/Liberty's puppet-keystone) does not provide a project when creating Keystone services users {ceilometer, nova, neutron etc}. It subsequently assigns an admin role for each service user to the services project and ideally that should have created a project Id mapping within the user reference but it doesn't.

You can also reproduce this issue through standard openstack user create/openstack role add cmds

Therefore the users objects in keystone end up with NO default_project_id/tenantId fields which has the following two disadvantages:
1) The Users detail view in Horizon's Identity pane shows ProjectID as "None"
2) RBAC policies pertaining to the services project do not work since it cannot correlate the services users to the services project

I have a fix for this in my deployment which is to update the user reference with project Id during role grants.

I'll raise a review for this shortly incase this is something the community is interested in.

Changed in keystone:
assignee: nobody → Ron De Rose (ronald-de-rose)
Revision history for this message
Kam Nasim (knasim-wrs) wrote :

Ron, I already have a fix for this.

Changed in keystone:
assignee: Ron De Rose (ronald-de-rose) → Kam Nasim (knasim-wrs)
Changed in keystone:
status: New → Confirmed
Changed in keystone:
status: Confirmed → New
Changed in keystone:
milestone: none → newton-3
importance: Undecided → Critical
Revision history for this message
Guang Yee (guang-yee) wrote :

I am a bit confused by this bug.

1. Are we saying existing users with default_project_id field set to some project prior to upgrading to Mitaka are now seeing default_project_id set to None after upgraded to Mitaka?
2. Is this impacting ALL users or just the service users? Comments in #8 seem to suggest this is only impacting the service users.
3. Why would assigning the 'admin' role to the services users for the service project automatically set the 'default_project_id' field? I don't remember we ever support that.
4. Horizon behavior is expected if user have no default project.
5. Not sure if I understand this statement, "RBAC policies pertaining to the services project do not work since it cannot correlate the services users to the services project". If the given token is scoped to a project, RBAC policies should work accordingly. Are we saying we have policies that act on the 'default_project_id'?

Revision history for this message
Kam Nasim (knasim-wrs) wrote :

 See inline with [Kam]:

1. Are we saying existing users with default_project_id field set to some project prior to upgrading to Mitaka are now seeing default_project_id set to None after upgraded to Mitaka?

[Kam]: Yes, as part of our general Openstack upgrade, we also updated the puppet modules. In Kilo's puppet-keystone, a project Id was provided when creating the services users. Whereas in puppet-keystone for Mitaka, the services users are created in two stages, 1) create user (without specifying project Id), 2) grant a member/admin role for these users to the services project.

However I would like to add that puppet just exposed the problem and is not specific to Puppet. This issue can be reproduced with openstack client.

2. Is this impacting ALL users or just the service users? Comments in #8 seem to suggest this is only impacting the service users.

[Kam]: Impacts all users that do not have projectId specified during creation and subsequently have a role granted to them on said project.

3. Why would assigning the 'admin' role to the services users for the service project automatically set the 'default_project_id' field? I don't remember we ever support that.

[Kam]: This has nothing to do with the 'admin' role in general and can be seen for any role/any user.

4. Horizon behavior is expected if user have no default project.

[Kam]: If a user does NOT have a default project(none specified during creation), shouldn't granting the first role on this user to a project set that as default project? Otherwise you end up with a scenario where Horizon indicates that user is projectless whereas it has a role on a project

5. Not sure if I understand this statement, "RBAC policies pertaining to the services project do not work since it cannot correlate the services users to the services project". If the given token is scoped to a project, RBAC policies should work accordingly. Are we saying we have policies that act on the 'default_project_id'?

[Kam]: Yes, this is specific to our deployment as we have modified the RBAC policies. I was mentioning it for posterity, it should not affect Upstream. Horizon not showing ProjectID, as well as logic that expects default_project_id to be within the user reference breaks because of this issue

Revision history for this message
Adam Young (ayoung) wrote :

On the the initial report: it looks suspiciously like postgres is executing a view when doing select * from user;

The user table does not have a current_user field. That looks like a postgres entity.

Revision history for this message
Henry Nash (henry-nash) wrote :

So a few additional comments:

1) In Keystone V2, if a user has a default project, then they are automatically assigned a role on it (the member role). This is not true for V3 (and this is by design). Having a default project with a user created via the V3 API does not give them any role assignments. You have to add these manually.

2) Sounds to me like the puppet models for deploying keystone you are using do not take this into account

3) For a user that does not have a default project, granting them a role on a project will absolutely not make this project their default project (and nor should it).

4) The one thing I am not clear on, is whether there is an issue for a client (e.g. Horizon) talking the keystone V2 API with the fact that if a user has no default project, then the tennatID attribute will not appear at all in the user entity returned (whereas using V3 you would get default_project_id=None in the entity)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/349059

Changed in keystone:
status: New → In Progress
Revision history for this message
Kam Nasim (knasim-wrs) wrote :

Henry, sent out the code review since it was already in my pipeline and so as to visualize the changes but if this conflicts with Keystone's design philosophy then feel free to indicate that on the code inspection.

If there is no fix in Keystone then we need to modify the puppet-keystone modules to account for no default project being assigned when creating roles(puppet-kilo/liberty used to do this)

Revision history for this message
Henry Nash (henry-nash) wrote :

Hi

I think the puppet change is the right thing, I don't think there will be much support for changing the keystone design here.

Changed in keystone:
status: In Progress → Invalid
Revision history for this message
Henry Nash (henry-nash) wrote :

I've marked this as invalid, but let me know if there are other aspects of this that we have not discussed.

Changed in keystone:
milestone: newton-3 → none
Revision history for this message
Kam Nasim (knasim-wrs) wrote :

Summary thus far:

The change to modify Keystone server to assign a default project Id to a user reference, if one does not exist, has been withdrawn since Identity V3 is functioning as designed.

Instead, we will address this as a puppet-keystone change to:
Specify project Id when creating keystone admin/services users so that:
       a) Horizon>Identity panel does not show an empty value for Project ID under the User detailed view
       b) RBAC policy(ies) that relies on target.user.default_project_id continues to function as intended

Changed in puppet-keystone:
assignee: nobody → Kam Nasim (knasim-wrs)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on keystone (master)

Change abandoned by Steve Martinelli (<email address hidden>) on branch: master
Review: https://review.openstack.org/349059
Reason: bug is invalid and no change in months

Revision history for this message
Anilkumar Thovi (thovi) wrote :

using V3 API we cannot access user email?

users = keystone.users.list()
for row in users:
        print row.email

Getting below error :
/*******************************************/
 File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 496, in __getattr__
    raise AttributeError(k)
AttributeError: email

/*******************************************/
becuase in the user details, email field does not exist:
<User domain_id=e07a8e155862444eb9e966121fd161ce, enabled=True, id=8cda896fe33e4ed58e39b97157d38280, links={u'self': u'http::5000/v3/users/8cda896fe33e4ed58e39b97157d38280'}, name=, password_expires_at=None>

if deafult project exist then only we can access email field?
<User default_project_id=8cb63b3a72894e18ad18b1c56b4dc3e9, domain_id=e07a8e155862444eb9e966121fd161ce,
<email address hidden>, enabled=True, id=9ba608bc868d4333be1d06a8db3e8cbc,
links={u'self': u'http://xyz:5000/v3/users/9ba608bc868d4333be1d06a8db3e8cbc'}, name=anil, password_expires_at=None>

So how to access user email field when default_project_id does not exist?

Revision history for this message
Lance Bragstad (lbragstad) wrote :

I set up a test locally to see if I could recreate what you're seeing [0]. I am able to list users regardless of the default project id or the email attribute being present. Are you using a specific version of python-keystoneclient. I attempted this with master of both keystone and python-keystoneclient.

[0] http://cdn.pasteraw.com/d79a18f4pnydnqboni2svg5zdolsmos

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers