OpenStack Identity (keystone)

Valid column creation in token table fails with postgres

Bug #1068181 reported by Dave Spano on 2012-10-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Identity (keystone)	Fix Released	Medium	Dirk Mueller	OpenStack Identity (keystone) 2013.1 "grizzly"

Bug Description

When migrating from the Essex to the Folsom version of Ubuntu 12.04 with postgres I get the following error when keystone tries to update the database schema.

For it to work, I needed to truncate the token table in the keystone database. If this is better posted with the Ubuntu package maintainer, please let me know.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
Setting up keystone (2012.2-0ubuntu1~cloud0) ...
Traceback (most recent call last):
  File "/usr/bin/keystone-manage", line 28, in <module>
    cli.main(argv=sys.argv, config_files=config_files)
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 164, in main
    return run(cmd, (args[:1] + args[2:]))
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 147, in run
    return CMDS[cmd](argv=args).run()
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 35, in run
    return self.main()
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 56, in main
    driver.db_sync()
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/sql.py", line 136, in db_sync
    migration.db_sync()
  File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration.py", line 52, in db_sync
    return versioning_api.upgrade(CONF.sql.connection, repo_path, version)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 159, in with_engine
    return f(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 91, in runchange
    change.run(self.engine, step)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 145, in run
    script_func(engine)
  File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migrate_repo/versions/003_token_valid.py", line 33, in upgrade
    token.create_column(valid)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/schema.py", line 426, in create_column
    column.create(table=self, *p, **kw)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/schema.py", line 528, in create
    engine._run_visitor(visitorcallable, self, connection, **kwargs)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2303, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1973, in _run_visitor
    **kwargs).traverse_single(element)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/ansisql.py", line 53, in traverse_single
    ret = super(AlterTableVisitor, self).traverse_single(elem)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 106, in traverse_single
    return meth(obj, **kw)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/ansisql.py", line 101, in visit_column
    self.execute()
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/ansisql.py", line 42, in execute
    return self.connection.execute(self.buffer.getvalue())
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1628, in _execute_text
    statement, parameters
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) column "valid" contains null values
'\nALTER TABLE token ADD valid BOOLEAN NOT NULL' {}
dpkg: error processing keystone (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
keystone
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@ha3:/etc/apt/sources.list.d# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
Setting up keystone (2012.2-0ubuntu1~cloud0) ...
Traceback (most recent call last):
  File "/usr/bin/keystone-manage", line 28, in <module>
    cli.main(argv=sys.argv, config_files=config_files)
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 164, in main
    return run(cmd, (args[:1] + args[2:]))
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 147, in run
    return CMDS[cmd](argv=args).run()
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 35, in run
    return self.main()
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 56, in main
    driver.db_sync()
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/sql.py", line 136, in db_sync
    migration.db_sync()
  File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration.py", line 52, in db_sync
    return versioning_api.upgrade(CONF.sql.connection, repo_path, version)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 159, in with_engine
    return f(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 91, in runchange
    change.run(self.engine, step)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 145, in run
    script_func(engine)
  File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migrate_repo/versions/003_token_valid.py", line 33, in upgrade
    token.create_column(valid)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/schema.py", line 426, in create_column
    column.create(table=self, *p, **kw)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/schema.py", line 528, in create
    engine._run_visitor(visitorcallable, self, connection, **kwargs)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2303, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1973, in _run_visitor
    **kwargs).traverse_single(element)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/ansisql.py", line 53, in traverse_single
    ret = super(AlterTableVisitor, self).traverse_single(elem)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 106, in traverse_single
    return meth(obj, **kw)
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/ansisql.py", line 101, in visit_column
    self.execute()
  File "/usr/lib/python2.7/dist-packages/migrate/changeset/ansisql.py", line 42, in execute
    return self.connection.execute(self.buffer.getvalue())
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1628, in _execute_text
    statement, parameters
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) column "valid" contains null values
'\nALTER TABLE token ADD valid BOOLEAN NOT NULL' {}
dpkg: error processing keystone (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:

Tags:

Revision history for this message

Joseph Heck (heckj) wrote on 2012-10-20:

Adam,

Do you have a postgres database setup for Keystone? I recall you talking about wanting to move Postgres to a first-class citizen at the summit. It looks like something we've got in one of the migration scripts isn't happy on that system in particular.

tags:	added: upgrade
Changed in keystone:
status:	New → Triaged
importance:	Undecided → Medium
assignee:	nobody → Adam Young (ayoung)

Revision history for this message

Adam Young (ayoung) wrote on 2012-10-23:

Yes, and I don't want to truncate the token table, either, as that will invalidate all outstanding tokens. Looks like we need to explicitly set the default values for 'valid', or do the table crete in multiple steps.

Revision history for this message

Dirk Mueller (dmllr) wrote on 2013-02-18:

Changing the generated SQL snippet from

ALTER TABLE token ADD valid BOOLEAN NOT NULL

ALTER TABLE token ADD valid BOOLEAN NOT NULL DEFAULT True

and executing it manually works. I'm confused why SQLAlchemy (or Migrate?) is not creating the Default part in the sql snippet.

OpenStack Infra (hudson-openstack) on 2013-02-18

Changed in keystone:
assignee:	Adam Young (ayoung) → Dirk Mueller (dmllr)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-02-19: Fix merged to keystone (master)

Reviewed: https://review.openstack.org/22215
Committed: http://github.com/openstack/keystone/commit/838069bf2239d4b3f094a644a8c77f0841a7240a
Submitter: Jenkins
Branch: master

commit 838069bf2239d4b3f094a644a8c77f0841a7240a
Author: Dirk Mueller <email address hidden>
Date: Mon Feb 18 18:03:31 2013 +0100

Workaround Migration issue with PostgreSQL

For unknown reason, SQLAlchemy-Migrate does not pass
the populate_default=True down to the ALTER TABLE statement.

Therefore, create the column in two steps, which
avoids the IntegrityError.

Fixes LP Bug 1068181

Change-Id: I30770d8b29ec442576a66fc07b2c78e2440fb328

Changed in keystone:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2013-02-21

Changed in keystone:
milestone:	none → grizzly-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2013-04-04

Changed in keystone:
milestone:	grizzly-3 → 2013.1

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.