migration test fails on table addition

Bug #1620764 reported by Alexander Makarov
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
David Stanek

Bug Description

If expand repo migration adds a table, corresponding unit test fails attempting to access created table with the error "Table does not exist" [0]

[0] http://logs.openstack.org/88/208488/51/check/gate-keystone-python27-db-ubuntu-xenial/81311f3/console.html#_2016-09-06_14_27_49_936937

Changed in keystone:
assignee: nobody → Ron De Rose (ronald-de-rose)
Revision history for this message
Ron De Rose (ronald-de-rose) wrote :

I'm able to add a new table without the migration tests failing:
https://review.openstack.org/#/c/366974/

I did a recheck on your patch (208488) and now its only failing MySQLOpportunisticFullMigration and PostgreSQLOpportunisticFullMigration:
http://logs.openstack.org/88/208488/51/check/gate-keystone-python27-db-ubuntu-xenial/7a9c330/testr_results.html.gz

I wonder if it has something to do with your sqlalchemy syntax.

Revision history for this message
Alexander Makarov (amakarov) wrote :

The difference is empty data and contract migrations: when I removed those two (and from the test too), gate passed. I think that's not the expected behavior.

Changed in keystone:
assignee: Ron De Rose (ronald-de-rose) → nobody
Changed in keystone:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
David Stanek (dstanek) wrote :

I was able to track this down into sqlalchemy-migrate. I'm 100% sure if this is only in tests or can actually happen during production upgrade, but I suspect that it's only tests.

If you run certain combinations of tests the table will not be created. One such example:
  "testr run keystone.tests.unit.test_sql_upgrade.MySQLOpportunisticFullMigration.test_add_delegation_tables keystone.tests.unit.test_sql_upgrade.FullMigration.test_add_delegation_tables" -- you need to be in an activated environment to run that.

I was able to fix the issue by hacking sqlalchemy-migrate like http://paste.openstack.org/show/577431/. The issue seems to be that some objects in sql-alchemy are being reused somehow. For example, I added logging to our code to print out detail of each change being applied. When I do that I can see that the change object has the correct `path` attribute value pointing to the expand repo, but the `module` property's value is a script object from the data migration repo.

Some more work needs to be done to figure this out.

Revision history for this message
David Stanek (dstanek) wrote :

the tldr; is that we are using the same names for the migration modules (e.g. expand_repo/version/001_awesome.py and data_migration_repo/versions/001_awesome.py). The confusing the script caching used by sqlalchemy-migrate. This appears to only be a test problem.

The thing that was failing in the tests is that we did the following:
  - expand to version 003
  - migration to verison 003
  - contract to version 003
  - expand to version 004 and make sure it worked

It didn't work because the script being executed was actually the data_migration the delegation table was not created and the test failed.

Somehow things broke down with the PythonScript instance caching and how it handles importing the code dynamically. I was able to prove this by renaming the 004_* files to unique names. This causes the tests to pass.

As a test-only workaround we can clear the cache in between test runs to make sure the data is properly reloaded.

I don't think the wold impact production because we are not trying to load and reload repositories like we do in tests.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/371075

Changed in keystone:
assignee: nobody → David Stanek (dstanek)
status: Confirmed → In Progress
Changed in keystone:
milestone: none → ocata-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/371075
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0818d42cb2afb5cd1ded72d9c2cc7c0ac4762a2c
Submitter: Jenkins
Branch: master

commit 0818d42cb2afb5cd1ded72d9c2cc7c0ac4762a2c
Author: David Stanek <email address hidden>
Date: Thu Sep 15 19:28:22 2016 +0000

    Ensure the sqla-migrate scripts cache is cleared

    The cache was causing problems with our multiple repositories since
    the modules all have the same names.
      e.g. expand_repo/versions/001_awesome.py and
           data_migration_repo/versions/001_awesome.py)

    Change-Id: Ib9bbeebc2149b0aee5f3d6fec963e40e90f1b743
    Closes-Bug: #1620764

Changed in keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 11.0.0.0b1

This issue was fixed in the openstack/keystone 11.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.