Comment 4 for bug 1702211

We made some good progress on this today and ended up figuring out the issue. Full logs and context can be found in IRC [0].

The crux of the issue is that keystone relies on datetime objects in MySQL, which are subject to differences in rounding depending on the MySQL version. This is especially true when precision under a second is required. The second part of the issue is that during the update of passwords, keystone asks the backend for all passwords, ordered by their creation time [1]. If a password is updated twice within a second - both passwords will have the same creation time making it possible of the old password to be returned ahead of the newly created passed.

To fix this, we shouldn't rely on native datetime formats with MySQL. Instead, keystone should store the datetime as an integer (which is more precise) and marshal the integer to a datetime object in the application. The result will be that MySQL will return passwords ordered by an integer, which is more precise, and leave the datetime conversion to keystone.

This should be done consistently in all places where keystone relies on datetime from MySQL.

[0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-08-11.log.html#t2017-08-11T14:53:24
[1] https://github.com/openstack/keystone/blob/6621c7810772eb076917ff7a959e77e13a0d6b85/keystone/identity/backends/sql_model.py#L276-L281