db migration tests falsely succeed

Bug #1406314 reported by Boris Bobrov on 2014-12-29
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Steve Martinelli

Bug Description

---[Proposed Solution (morganfainberg)]---
The solution to this bug it so eliminate all SQLite migrations in the unit tests and only test migrations with the functional test suite. This is something that will occur once David Stanek has the full functional test suite up and running.

This also implies that support for SQLite is suspect at best for all use-cases.

---[ Problem Description ]---

Some db migration tests succeed, while the same migrations, applied by db_sync fail. A good example of this is bug #1405673: migration 44 seems to be tested, but failes when applied in real life.

To illustrate the issue, http://paste.openstack.org/show/155081/ can be applied here https://github.com/openstack/keystone/blob/f5c06d619c762d5d39e8ec3bc752215ec02b2756/keystone/tests/test_sql_upgrade.py#L152-L163 . The tests will be very slow, but reliable (because the db will be in file, not in memory) and will fail. Here's the output:

(.venv)breton@bbobrov-pc ~/src/keystone (bug/1405673*) $ ./run_tests.sh SqlUpgradeTests
    test_blank_db_to_start OK 0.50
    test_downgrade_endpoint_enabled_cols FAIL
    test_downgrade_service_enabled_cols OK 20.78
    test_add_actor_id_index OK 25.44
    test_endpoint_region_downgrade_columns OK 27.31
    test_downgrade_endpoint_enabled_data FAIL
    test_id_mapping OK 21.70
    test_assignment_table_migration OK 25.05
    test_limited_trusts_upgrade OK 18.93
    test_downgrade_service_enabled_data OK 24.67
    test_downgrade_to_db_init_version FAIL
    test_project_parent_id_downgrade OK 39.04
    test_endpoint_region_migration OK 46.27
    test_endpoint_region_upgrade_columns OK 36.07
    test_region_url_downgrade OK 36.40
    test_limited_trusts_downgrade OK 31.71
    test_limited_trusts_downgrade_trusts_cleanup OK 33.84
    test_region_url_upgrade OK 38.80
    test_project_parent_id_upgrade OK 49.62
    test_project_parent_id_cleanup OK 52.02
    test_two_steps_forward_one_step_back FAIL
    test_region_migration OK 34.42
    test_remove_actor_id_index OK 48.16
    test_start_version_db_init_version OK 0.95
    test_upgrade_endpoint_enabled_cols OK 29.47
    test_region_url_cleanup OK 42.90
    test_token_user_id_and_trust_id_index_downgrade OK 96.98
    test_upgrade_endpoint_enabled_data OK 101.70
    test_token_user_id_and_trust_id_index_upgrade OK 125.88
    test_upgrade_region_unique_description FAIL
    test_upgrade_region_non_unique_description OK 91.90
    test_upgrade_add_initial_tables OK 65.13
    test_upgrade_service_enabled_data OK 97.24
    test_upgrade_service_enabled_cols OK 80.83
Ran 34 tests in 557.405s

FAILED (failures=5)

description: updated
tags: added: sql test-improvement
Morgan Fainberg (mdrnstm) wrote :

I would like to see a move to eliminate SQLite from use in Keystone instead of trying to rely on it to "test" migrations. SQLite does not mirror any real production worthy system and tends to result in weird edge-cases.

Instead of "fixing" these migration tests we should move towards the functional test suite doing tests against MySQL and pgsql (in infra-gate) directly and drop SQLite support completely.

SQL migration tests need to be re-worked as well to mirror how nova handles them for pre-upgrade/upgrade/post-upgrade checks (and downgrades).

Morgan Fainberg (mdrnstm) wrote :

Comments in #1 are reflective of SQLite support in migration testing. SQLite does not compare to any other SQL backend in migrations (and in many some cases not even for runtime)

Morgan Fainberg (mdrnstm) wrote :

So lets move all migration testing to the functional test suite and eliminate SQLite migration testing as soon as David Stanek has the functional testing up and working.

Changed in keystone:
importance: Undecided → Wishlist
description: updated
description: updated
Changed in keystone:
status: New → Triaged
Changed in keystone:
milestone: none → kilo-rc1
Changed in keystone:
milestone: kilo-rc1 → none
Viktor Serhieiev (vsergeyev) wrote :

JFYI - here is an another attempt to run DB migration tests on various backends - https://review.openstack.org/#/c/171115/

tags: added: functional-testing
Steve Martinelli (stevemar) wrote :

This is resolved now that https://blueprints.launchpad.net/keystone/+spec/tests-on-rdbmses has merged in newton 1

Changed in keystone:
milestone: none → newton-1
Boris Bobrov (bbobrov) wrote :

We now have mysql and postgresql jobs that check migrations

Changed in keystone:
status: Triaged → Fix Released
Changed in keystone:
assignee: nobody → Steve Martinelli (stevemar)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers