Migrate 43 fails on old sqlalchemy

[Jamie Lennox]

When using sqlalchemy 0.7.10 running migration 043_fixup_region_description.py fails with the error:

Traceback (most recent call last):
  File "keystone/tests/test_sql_upgrade.py", line 2546, in test_upgrade_region_unique_description
    self.upgrade(43)
  File "keystone/tests/test_sql_upgrade.py", line 139, in upgrade
    self._migrate(*args, **kwargs)
  File "keystone/tests/test_sql_upgrade.py", line 156, in _migrate
    self.schema.runchange(ver, change, changeset.step)
  File "/home/jamie/.virtualenvs/keystone2/lib/python2.7/site-packages/migrate/versioning/schema.py", line 91, in runchange
    change.run(self.engine, step)
  File "/home/jamie/.virtualenvs/keystone2/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 145, in run
    script_func(engine)
  File "/home/jamie/work/keystone/keystone/common/sql/migrate_repo/versions/043_fixup_region_description.py", line 78, in upgrade
    region_table = sql.Table(_REGION_TABLE_NAME, meta, autoload=True)
  File "/home/jamie/.virtualenvs/keystone2/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 106, in __getattr__
    raise AttributeError(key)
AttributeError: values

Upgrading to sqlalchemy 0.8.5 fixes the error, however our requirements.txt file lists: SQLAlchemy>=0.7.8,<=0.8.99 so this should still be valid.

I can't quite tell when the values() function was added, i assume it was 0.8 but i'm not familiar with the migration to know exactly what is being accomplished there.

[Jamie Lennox (jamielennox)]

running: python -m testtools.run keystone.tests.test_sql_upgrade.SqlUpgradeTests.test_upgrade_region_unique_description

[Lance Bragstad (lbragstad)]

Looks like the upgrade in migration 43 is the only place where 'column.values' is being used, which would explain why we are hitting this here:

https://github.com/openstack/keystone/blob/master/keystone/common/sql/migrate_repo/versions/043_fixup_region_description.py#L80

So, regardless it looks like we are going to have to go back and alter migration 43, unless we bump the lower bound requirement on sqlalchemy to something greater than 0.7.10.

[Lance Bragstad (lbragstad)]

@Jamie, are there any other steps you took to recreate this? I've done the following;

1.) Activated venv for Keystone
2.) Installed sqlalchemy 0.7.10 https://pypi.python.org/simple/SQLAlchemy/
3.) Ran the test: 'python -m testtools.run keystone.tests.test_sql_upgrade.SqlUpgradeTests.test_upgrade_region_unique_description'

(venv)lbragstad@precise64:/opt/stack/keystone$ sudo pip show sqlalchemy
---
Name: SQLAlchemy
Version: 0.7.10
Location: /usr/local/lib/python2.7/dist-packages
Requires:
(venv)lbragstad@precise64:/opt/stack/keystone$ python -m testtools.run keystone.tests.test_sql_upgrade.SqlUpgradeTests.test_upgrade_region_unique_description
Tests running...

Ran 1 test in 1.516s
OK
(venv)lbragstad@precise64:/opt/stack/keystone$

[Dolph Mathews (dolph)]

$ pip install --upgrade "sqlalchemy==0.7.10"
Requirement already up-to-date: sqlalchemy==0.7.10 in /Users/dolph/Environments/os/lib/python2.7/site-packages
Cleaning up...

$ python -m testtools.run keystone.tests.test_sql_upgrade.SqlUpgradeTests.test_upgrade_region_unique_description
Tests running...
======================================================================
ERROR: keystone.tests.test_sql_upgrade.SqlUpgradeTests.test_upgrade_region_unique_description
----------------------------------------------------------------------
pythonlogging:'': {{{
Adding cache-proxy 'keystone.tests.test_cache.CacheIsolatingProxy' to backend.
Loading repository /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo...
Loading script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/001_add_initial_tables.py...
Script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/001_add_initial_tables.py loaded successfully
[...]
Loading script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/042_endpoint_enabled.py...
Script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/042_endpoint_enabled.py loaded successfully
Loading script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/043_fixup_region_description.py...
Script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/043_fixup_region_description.py loaded successfully
Loading script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/044_service_enabled.py...
Script /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/044_service_enabled.py loaded successfully
Repository /Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo loaded successfully
Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'keystone'), ('version_table', 'migrate_version'), ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))])
}}}

Traceback (most recent call last):
  File "keystone/tests/test_sql_upgrade.py", line 2546, in test_upgrade_region_unique_description
    self.upgrade(43)
  File "keystone/tests/test_sql_upgrade.py", line 139, in upgrade
    self._migrate(*args, **kwargs)
  File "keystone/tests/test_sql_upgrade.py", line 156, in _migrate
    self.schema.runchange(ver, change, changeset.step)
  File "/Users/dolph/Environments/os/lib/python2.7/site-packages/migrate/versioning/schema.py", line 91, in runchange
    change.run(self.engine, step)
  File "/Users/dolph/Environments/os/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 145, in run
    script_func(engine)
  File "/Users/dolph/Projects/keystone/keystone/common/sql/migrate_repo/versions/043_fixup_region_description.py", line 80, in upgrade
    len(idx.columns.values()) is 1):
  File "/Users/dolph/Environments/os/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 106, in __getattr__
    raise AttributeError(key)
AttributeError: values

Ran 1 test in 1.012s
FAILED (failures=1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/82793

[Lance Bragstad (lbragstad)]

I did some tweaking and I can recreate, the fix above works for me.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/82793
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=8a22fcd6657a5bf6a619123480126eccdf4d5bc3
Submitter: Jenkins
Branch: master

commit 8a22fcd6657a5bf6a619123480126eccdf4d5bc3
Author: Dolph Mathews <email address hidden>
Date: Tue Mar 25 07:13:22 2014 -0500

    Avoid using .values() on the indexed columns

    The length of a dict-like object should be the same as the length of the
    list of values in the object, so calling .values() should be unnecessary
    here.

    Change-Id: I625dc7dea00cc8f09c68ea104c5c64c5f39fffc0
    Closes-Bug: 1297059