db sync on federation failed if there is existing data

[Jose Castro Leon]

If you have an existing entry in the identity_provider table, when updating from juno to kilo, it fails with the following error

OperationalError: (OperationalError) (1048, "Column 'remote_id' cannot be null") 'INSERT INTO idp_remote_ids (idp_id, remote_id) VALUES (%s, %s)' ('https://cern.ch/login', None)

The migrate_repo goes from 2 to 6 and it failes in step 7.

The issue is linked to a new field created on identity_provider table called 'remote_id' that it is created in step 3 and left empty.
Then on step 7 it tries to read to insert into the idp_remote_ids that does not accept null values.

2015-07-28 15:04:00.247 18168 TRACE keystone Traceback (most recent call last):
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/bin/keystone-manage", line 44, in <module>
2015-07-28 15:04:00.247 18168 TRACE keystone cli.main(argv=sys.argv, config_files=config_files)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 585, in main
2015-07-28 15:04:00.247 18168 TRACE keystone CONF.command.cmd_class.main()
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 76, in main
2015-07-28 15:04:00.247 18168 TRACE keystone migration_helpers.sync_database_to_version(extension, version)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 249, in sync_database_to_version
2015-07-28 15:04:00.247 18168 TRACE keystone _sync_extension_repo(extension, version)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 219, in _sync_extension_repo
2015-07-28 15:04:00.247 18168 TRACE keystone init_version=init_version)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/migration.py", line 79, in db_sync
2015-07-28 15:04:00.247 18168 TRACE keystone return versioning_api.upgrade(engine, repository, version)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
2015-07-28 15:04:00.247 18168 TRACE keystone return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2015-07-28 15:04:00.247 18168 TRACE keystone File "<string>", line 2, in _migrate
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
2015-07-28 15:04:00.247 18168 TRACE keystone return f(*a, **kw)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
2015-07-28 15:04:00.247 18168 TRACE keystone schema.runchange(ver, change, changeset.step)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
2015-07-28 15:04:00.247 18168 TRACE keystone change.run(self.engine, step)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
2015-07-28 15:04:00.247 18168 TRACE keystone script_func(engine)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/contrib/federation/migrate_repo/versions/007_add_remote_id_table.py", line 39, in upgrade
2015-07-28 15:04:00.247 18168 TRACE keystone remote_id_table.insert(remote_idp_entry).execute()
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/base.py", line 386, in execute
2015-07-28 15:04:00.247 18168 TRACE keystone return e._execute_clauseelement(self, multiparams, params)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1759, in _execute_clauseelement
2015-07-28 15:04:00.247 18168 TRACE keystone return connection._execute_clauseelement(elem, multiparams, params)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
2015-07-28 15:04:00.247 18168 TRACE keystone compiled_sql, distilled_params
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
2015-07-28 15:04:00.247 18168 TRACE keystone context)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception
2015-07-28 15:04:00.247 18168 TRACE keystone e, statement, parameters, cursor, context)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
2015-07-28 15:04:00.247 18168 TRACE keystone util.raise_from_cause(newraise, exc_info)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-07-28 15:04:00.247 18168 TRACE keystone reraise(type(exception), exception, tb=exc_tb)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
2015-07-28 15:04:00.247 18168 TRACE keystone context)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
2015-07-28 15:04:00.247 18168 TRACE keystone cursor.execute(statement, parameters)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2015-07-28 15:04:00.247 18168 TRACE keystone self.errorhandler(self, exc, value)
2015-07-28 15:04:00.247 18168 TRACE keystone File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-07-28 15:04:00.247 18168 TRACE keystone raise errorclass, errorvalue
2015-07-28 15:04:00.247 18168 TRACE keystone OperationalError: (OperationalError) (1048, "Column 'remote_id' cannot be null") 'INSERT INTO idp_remote_ids (idp_id, remote_id) VALUES (%s, %s)' ('https://cern.ch/login', None)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/206561

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/229647

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/206561
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1438cfeb4cf56773998e03c9801efee5f9673a75
Submitter: Jenkins
Branch: master

commit 1438cfeb4cf56773998e03c9801efee5f9673a75
Author: Marek Denis <email address hidden>
Date: Tue Jul 28 16:23:07 2015 +0200

    Skip rows with empty remote_ids

    Federation migration scripts are trying to migrate
    identity_provider.remote_id columns to a separate table remote_idps and
    so 1:N relations are possible. We should, however do any DB operations
    on identity_providers's values that have non null remote_id values.

    Change-Id: Ifbb80896969986bafedf1c879bc7474832afca60
    Closes-Bug: #1478961

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229647
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=92f7085cd32d168771ca667fe2503222ef885902
Submitter: Jenkins
Branch: stable/liberty

commit 92f7085cd32d168771ca667fe2503222ef885902
Author: Marek Denis <email address hidden>
Date: Tue Jul 28 16:23:07 2015 +0200

    Skip rows with empty remote_ids

    Federation migration scripts are trying to migrate
    identity_provider.remote_id columns to a separate table remote_idps and
    so 1:N relations are possible. We should, however do any DB operations
    on identity_providers's values that have non null remote_id values.

    Change-Id: Ifbb80896969986bafedf1c879bc7474832afca60
    Closes-Bug: #1478961

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/235215

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/235215
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=88dccd1e57770d96177d63897fa05819f73cd8a6
Submitter: Jenkins
Branch: master

commit 8dcd82fb9c76d43f26338bee293b32f4af473ad9
Author: Lance Bragstad <email address hidden>
Date: Wed Sep 9 14:09:06 2015 +0000

    Ensure token validation works irrespective of padding

    In change I674bad86ccc9027ac3b365c10b3b142fc9d73c17 we removed the padding from
    Fernet tokens. This commit makes it so that we can validate Fernet tokens that
    come in with and without padding.

    Change-Id: I60e52af6c4cb39f2d136540a487bba8505a2c6b4
    Closes-Bug: 1491926
    (cherry picked from commit 90aca980587283d26fcfd20ae6be86ca1bbe2788)

commit 4a239f23e3e599b55f6553f3b4f3290773d2ded7
Author: Tony Wang <email address hidden>
Date: Fri Aug 21 23:41:32 2015 -0400

    Show v3 endpoints in v2 endpoint list

    v3 endpoints that are not migrated from v2 endpoints, were not shown in
    v2 endpoint list. While it was more clear, separating v2 and v3
    endpoints, it was not easy for tools or services which are still using
    v2 APIs to get the endpoints created by v3 API.
    Since it won't do harm to return more information, this commit is to
    show v3 endpoints in API call to list v2 endpoints.
    NOTE: public interface is required in a v2 endpoints, so only v3
    endpoints with public interface are added into the list.

    Change-Id: Id9b3822f87e973b6eb7dd9fc715c2dba76b9fe04
    Closes-Bug: #1480270
    (cherry picked from commit 0d13a85e93737df8c9c130f194839b27d0781890)

commit ec3ceb5956aa2bab5fe03107849a7ccde27ef87c
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Oct 1 06:09:29 2015 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I621480e00e624575dd01138de260f246570aceee

commit 92f7085cd32d168771ca667fe2503222ef885902
Author: Marek Denis <email address hidden>
Date: Tue Jul 28 16:23:07 2015 +0200

    Skip rows with empty remote_ids

    Federation migration scripts are trying to migrate
    identity_provider.remote_id columns to a separate table remote_idps and
    so 1:N relations are possible. We should, however do any DB operations
    on identity_providers's values that have non null remote_id values.

    Change-Id: Ifbb80896969986bafedf1c879bc7474832afca60
    Closes-Bug: #1478961

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 9.0.0.0b1 development milestone.