Duplicated entries in users API

Bug #1848342 reported by Pedro Henrique Pereira Martins
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
Pedro Henrique Pereira Martins

Bug Description

System version
==============
Keystone: 14.0.1 (Rocky)

The error happens when I try to show a federated user via the user name as the key instead of using the ID

Reproducing the error:

======================

First of all, I created a user in my Identity Provider (Idp) that is configured my KeyStone instance.

We use OpenID Connect protocol to create the federated environment. In this test, the user created is

 named as 'use_case'. At this moment, the user is not in the OpenStack database, just in the Idp

 (ephemeral user type).

Authenticating with the ephemeral federated user:

    Let's see the database tables before the login process.

    Users already created in the Idp (before the login):

        select * from user;

        +----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+

        | id | extra | enabled | default_project_id | created_at | last_active_at | domain_id |

        +----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+

        | 1b0ed400ec974420a3574a8788acc795 | {} | 1 | NULL | 2019-09-16 20:06:39 | NULL | default |

        +----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+

        select * from local_user;

        +-----+----------------------------------+-----------+-------+-------------------+----------------+

        | id | user_id | domain_id | name | failed_auth_count | failed_auth_at |

        +-----+----------------------------------+-----------+-------+-------------------+----------------+

        | 1 | 1b0ed400ec974420a3574a8788acc795 | default | admin | 0 | NULL |

        +-----+----------------------------------+-----------+-------+-------------------+----------------+

        select * from federated_user;

        Empty set (0.00 sec)

    So far, we have only the local user admin

Then, we execute the login using the ephemeral user that we created in the IdP.

    Let's the database state after the first login:

        select * from user;

        +----------------------------------+---------------------------------+---------+--------------------+---------------------+----------------+----------------------------------+

        | id | extra | enabled | default_project_id | created_at | last_active_at | domain_id |

        +----------------------------------+---------------------------------+---------+--------------------+---------------------+----------------+----------------------------------+

        | 1b0ed400ec974420a3574a8788acc795 | {} | 1 | NULL | 2019-09-16 20:06:39 | NULL | default |

        | f0327d98278f4b1881aec9c051b027d3 | {"email": "<email address hidden>"} | 1 | NULL | 2019-10-15 18:08:17 | NULL | d28c2423e9b546deb37a034bb2134f4d |

        +----------------------------------+---------------------------------+---------+--------------------+---------------------+----------------+----------------------------------+

        select * from local_user;

        +----+----------------------------------+-----------+-------+-------------------+----------------+

        | id | user_id | domain_id | name | failed_auth_count | failed_auth_at |

        +----+----------------------------------+-----------+-------+-------------------+----------------+

        | 1 | 1b0ed400ec974420a3574a8788acc795 | default | admin | 0 | NULL |

        +----+----------------------------------+-----------+-------+-------------------+----------------+

        select * from federated_user;

        +----+----------------------------------+--------+-------------+-----------+--------------+

        | id | user_id | idp_id | protocol_id | unique_id | display_name |

        +----+----------------------------------+--------+-------------+-----------+--------------+

        | 1 | f0327d98278f4b1881aec9c051b027d3 | my_idp | openid | use_case | use_case |

        +----+----------------------------------+--------+-------------+-----------+--------------+

        Here we can see that the user did not exist in OpenStack database before the first login; then, KeyStone receives the user attributes (during the authentication process) from the Idp and register the user in the OpenStack database as a federated user.

        (user and federated_user tables).

        Here is the OpenStack client output (when executing the show command after the first login):

        openstack user show use_case

        +---------------------+----------------------------------+

        | Field | Value |

        +---------------------+----------------------------------+

        | domain_id | d28c2423e9b546deb37a034bb2134f4d |

        | email | <email address hidden> |

        | enabled | True |

        | id | f0327d98278f4b1881aec9c051b027d3 |

        | name | use_case |

        | options | {} |

        | password_expires_at | None |

        +---------------------+----------------------------------+

    Afterwards, we logout and then we login again in Horizon via the IdP. Please invalidate your session, OpenStack has a bug, and it is not invalidating IdP sessions when the logout is called in Horizon):

        select * from user;

        +----------------------------------+---------------------------------+---------+--------------------+---------------------+----------------+----------------------------------+

        | id | extra | enabled | default_project_id | created_at | last_active_at | domain_id |

        +----------------------------------+---------------------------------+---------+--------------------+---------------------+----------------+----------------------------------+

        | 1b0ed400ec974420a3574a8788acc795 | {} | 1 | NULL | 2019-09-16 20:06:39 | NULL | default |

        | f0327d98278f4b1881aec9c051b027d3 | {"email": "<email address hidden>"} | 1 | NULL | 2019-10-15 18:08:17 | NULL | d28c2423e9b546deb37a034bb2134f4d |

        +----------------------------------+---------------------------------+---------+--------------------+---------------------+----------------+----------------------------------+

        select * from local_user;

        +----+----------------------------------+----------------------------------+----------+-------------------+----------------+

        | id | user_id | domain_id | name | failed_auth_count | failed_auth_at |

        +----+----------------------------------+----------------------------------+----------+-------------------+----------------+

        | 1 | 1b0ed400ec974420a3574a8788acc795 | default | admin | 0 | NULL |

        | 4 | f0327d98278f4b1881aec9c051b027d3 | d28c2423e9b546deb37a034bb2134f4d | use_case | 0 | NULL |

        +----+----------------------------------+----------------------------------+----------+-------------------+----------------+

        select * from federated_user;

        +----+----------------------------------+--------+-------------+-----------+--------------+

        | id | user_id | idp_id | protocol_id | unique_id | display_name |

        +----+----------------------------------+--------+-------------+-----------+--------------+

        | 1 | f0327d98278f4b1881aec9c051b027d3 | my_idp | openid | use_case | use_case |

        +----+----------------------------------+--------+-------------+-----------+--------------+

        Here is the bug, the user already existed in the OpenStack database, and during the second login, Keystone tried to update data with the attributes that come from the IdP. During the update process, Keystone will create an entry in the local_user table.

        The code that is generating this behavior can be found in https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L1421

        After that, if I try to show the user via OpenStack client, I receive the following error

        openstack user show use_case

        More than one user exists with the name 'use_case'.

Here is an excerpt of our Attribute Mapping JSON:

    [

    {

        "local": [

        {

            "user": {

            "name": "{0}",

            "email": "{1}",

            "domain": {

                "name": "{2}"

            }

            },

            "domain": {

            "name": "{2}"

            },

            "projects": [

            {

                "name": "{3}",

                "roles": [

                {

                    "name": "member"

                },

                {

                    "name": "domadmin"

                },

                {

                    "name": "heat_stack_owner"

                },

                {

                    "name": "load-balancer_member"

                }

                ]

            }

            ]

        }

        ],

        "remote": [

        {

            "type": "OIDC-preferred_username"

        },

        {

            "type": "OIDC-email"

        },

        {

            "type": "OIDC-openstack-user-domain"

        },

        {

            "type": "OIDC-openstack-default-project"

        },

        {

            "type": "OIDC-openstack-user-role",

            "any_one_of": [

            "domain-admin"

            ]

        }

        ]

    },

    {

        "local": [

        {

            "user": {

            "name": "{0}",

            "email": "{1}",

            "type": "local",

            "domain": {

                "name": "{2}"

            }

            },

            "domain": {

            "name": "{2}"

            },

            "projects": [

            {

                "name": "{3}",

                "roles": [

                {

                    "name": "member"

                },

                {

                    "name": "heat_stack_owner"

                },

                {

                    "name": "load-balancer_member"

                }

                ]

            }

            ]

        }

        ],

        "remote": [

        {

            "type": "OIDC-preferred_username"

        },

        {

            "type": "OIDC-email"

        },

        {

            "type": "OIDC-openstack-user-domain"

        },

        {

            "type": "OIDC-openstack-default-project"

        },

        {

            "type": "OIDC-openstack-user-status",

            "any_one_of": [

            "local"

            ]

        },

        {

            "type": "OIDC-openstack-user-role",

            "any_one_of": [

            "member"

            ]

        }

        ]

    }

    ]

Here is a link to the PR that propose to fix that issue:

https://review.opendev.org/#/c/687990

Changed in keystone:
assignee: nobody → Pedro Henrique Pereira Martins (pedrohpmartins)
status: New → In Progress
Revision history for this message
Colleen Murphy (krinkle) wrote :

Thanks for the detailed bug report, I've managed to reproduce the issue.

The problem is not that the entries are duplicated in list output, that is just a symptom. The problem is that an ephemeral federated user is ending up being created in the local_user table, which should never happen. In this case, it's because your mapping has the "email" property in the user, and it's hitting this case:

https://opendev.org/openstack/keystone/src/commit/f9a086e16599123fba8a12c1c06bec73565a0ebc/keystone/identity/core.py#L1419-L1422

It's running the update_user method on the identity sql driver. Even though the local user doesn't exist, since it's running against the user table, it successfully finds the shadow user:

https://opendev.org/openstack/keystone/src/commit/f9a086e16599123fba8a12c1c06bec73565a0ebc/keystone/identity/backends/sql.py#L206

but then for some reason recreates the record in the local_user table:

https://opendev.org/openstack/keystone/src/commit/f9a086e16599123fba8a12c1c06bec73565a0ebc/keystone/identity/backends/sql.py#L210

So we'll need to fix that, rather than filter the output.

(For anyone else trying to reproduce this, it helps to disable keystone caching, otherwise the issue only appears intermittently.)

Changed in keystone:
status: In Progress → Triaged
importance: Undecided → Medium
Changed in keystone:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.opendev.org/687990
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=7597ecc1350eb6918c09585e4116911102acb54a
Submitter: Zuul
Branch: master

commit 7597ecc1350eb6918c09585e4116911102acb54a
Author: Pedro Martins <email address hidden>
Date: Thu Oct 10 08:51:32 2019 -0300

    Stop adding entry in local_user while updating ephemerals

    Problem description
    ===================
    Today we have a consistency problem when updating federated
    users via OpenStack. When I update a ephemeral user via OpenStack,
    a registry in the local_user table is created, making this user
    having entries in user, local_user and federated_user tables in
    the database.

    Furthermore, if I try to do some operations using this user
    (that has entries in all three tables), I get a "More than one
    user exists with the name ..." error from the OpenStack
    Keystone API. It happens because the user has an entry in both
    local_user and federated_user tables.

    I fix the persistence in the local_user table for ephemeral
    users when doing updates.

    Proposal
    ========
    I fix the problem with creating an entry in the
    local_user table while updating an ephemeral user

    Closes-Bug: #1848342

    Change-Id: I2ac6e90f24b94dc5c0d9c0758f008a388597036c

Changed in keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/741205

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/train)

Reviewed: https://review.opendev.org/c/openstack/keystone/+/741205
Committed: https://opendev.org/openstack/keystone/commit/f1c1678efe893df3fc33e5127675e8b58567da32
Submitter: "Zuul (22348)"
Branch: stable/train

commit f1c1678efe893df3fc33e5127675e8b58567da32
Author: Pedro Martins <email address hidden>
Date: Thu Oct 10 08:51:32 2019 -0300

    Stop adding entry in local_user while updating ephemerals

    Problem description
    ===================
    Today we have a consistency problem when updating federated
    users via OpenStack. When I update a ephemeral user via OpenStack,
    a registry in the local_user table is created, making this user
    having entries in user, local_user and federated_user tables in
    the database.

    Furthermore, if I try to do some operations using this user
    (that has entries in all three tables), I get a "More than one
    user exists with the name ..." error from the OpenStack
    Keystone API. It happens because the user has an entry in both
    local_user and federated_user tables.

    I fix the persistence in the local_user table for ephemeral
    users when doing updates.

    Proposal
    ========
    I fix the problem with creating an entry in the
    local_user table while updating an ephemeral user

    Closes-Bug: #1848342

    Change-Id: I2ac6e90f24b94dc5c0d9c0758f008a388597036c
    (cherry picked from commit 7597ecc1350eb6918c09585e4116911102acb54a)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone train-eol

This issue was fixed in the openstack/keystone train-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.