Database schema differs from migrations

[Oleksii Chuprykov]

As models and migrations don't have any logical relation in code, so differences are possible. Furthermore in most of cases differences exists. Possible solution -- adding a new migration and altering a model to remove differences. Next, tests for differences between models and migrations should be added.

Patch that removes diff https://review.openstack.org/#/c/113916/
Adding the sync tests https://review.openstack.org/#/c/110683/

Diff example from glance:
AssertionError: Models and migration scripts aren't in sync:
[ ( 'remove_table',
    Table('migrate_version', MetaData(bind=None), Column('repository_id', VARCHAR(length=250), table=<migrate_version>, primary_key=True, nullable=False), Column('repository_path', TEXT(), table=<migrate_version>), Column('version', INTEGER(), table=<migrate_version>), schema=None)),
  [ ( 'modify_default',
      None,
      'image_locations',
      'status',
      { 'existing_nullable': False,
        'existing_type': VARCHAR(length=30)},
      u"'active'::character varying",
      None)],
  [ ( 'modify_nullable',
      None,
      'image_locations',
      'updated_at',
      { 'existing_server_default': None,
        'existing_type': TIMESTAMP()},
      True,
      False)],
  [ ( 'modify_nullable',
      None,
      'image_members',
      'status',
      { 'existing_server_default': None,
        'existing_type': VARCHAR(length=20)},
      True,
      False)],
  [ ( 'modify_nullable',
      None,
      'image_members',
      'updated_at',
      { 'existing_server_default': None,
        'existing_type': TIMESTAMP()},
      True,
      False)],
  [ ( 'modify_nullable',
      None,
      'image_properties',
      'updated_at',
      { 'existing_server_default': None,
        'existing_type': TIMESTAMP()},
      True,
      False)],
  ( 'remove_index',
    Index('ix_image_properties_image_id_name', Column('image_id', VARCHAR(length=36), ForeignKey(u'images.id'), table=<image_properties>, nullable=False), Column('name', VARCHAR(length=255), table=<image_properties>, nullable=False))),
  ( 'add_constraint',
    UniqueConstraint(Column('image_id', String(length=36), ForeignKey('images.id'), table=<image_properties>, nullable=False), Column('name', String(length=255), table=<image_properties>, nullable=False))),
  [ ( 'modify_nullable',
      None,
      'image_tags',
      'updated_at',
      { 'existing_server_default': None,
        'existing_type': TIMESTAMP()},
      True,
      False)],
  [ ( 'modify_default',
      None,
      'images',
      'id',
      { 'existing_nullable': False,
        'existing_type': VARCHAR(length=36)},
      u"nextval('images_id_seq'::regclass)",
      None)],
  [ ( 'modify_nullable',
      None,
      'images',
      'protected',
      { 'existing_server_default': None,
        'existing_type': BOOLEAN()},
      True,
      False)],
  [ ( 'modify_nullable',
      None,
      'images',
      'updated_at',
      { 'existing_server_default': None,
        'existing_type': TIMESTAMP()},
      True,
      False)],
  [ ( 'modify_nullable',
      None,
      'tasks',
      'status',
      { 'existing_server_default': None,
        'existing_type': VARCHAR(length=30)},
      False,
      True)],
  [ ( 'modify_nullable',
      None,
      'tasks',
      'type',
      { 'existing_server_default': None,
        'existing_type': VARCHAR(length=30)},
      False,
      True)],
  [ ( 'modify_nullable',
      None,
      'tasks',
      'updated_at',
      { 'existing_server_default': None,
        'existing_type': TIMESTAMP()},
      True,
      False)]]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/119824

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113916
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=9def368d07f9db8698c3c3b8b4db92aadd78dcbb
Submitter: Jenkins
Branch: master

commit 9def368d07f9db8698c3c3b8b4db92aadd78dcbb
Author: Oleksii Chuprykov <email address hidden>
Date: Tue Sep 2 13:14:21 2014 +0300

    Alter models and add migration

    We must have correct models i.e. models that correspond
    a database schema to use sqlalchemy features.
    Update models.py and add migration script to correct
    database schema and get rid of the difference between
    schema and models.

    Partial-Bug: #1365436
    Change-Id: I9b4571906d39bcdb35048caa58d16ad5e888cce4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110683
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=e90f184143c8d69d826e7763a2503bc480927a5b
Submitter: Jenkins
Branch: master

commit e90f184143c8d69d826e7763a2503bc480927a5b
Author: Oleksii Chuprykov <email address hidden>
Date: Mon Oct 6 19:53:24 2014 +0300

    Add ModelSMigrationSync classes

    Add classes for testing correspondence between migration scripts
    and metadata.
    We need oslotest in test-requirements due to bug#1356425

    Partial-Bug: #1365436
    Change-Id: I0db68ea3557dd9d214ea50ff7c96de1a47a58d4b

[Zhi Yan Liu (lzy-dev)]

What is left for this bug regarding to above two merged patches? We need to handle metadef stuff right?

[Oleksii Chuprykov (ochuprykov)]

Yes, this one https://review.openstack.org/#/c/119824/ .

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119824
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=2968b2213ed978e980f6c8b11398c993766b395f
Submitter: Jenkins
Branch: master

commit 2968b2213ed978e980f6c8b11398c993766b395f
Author: Oleksii Chuprykov <email address hidden>
Date: Mon Sep 8 16:22:50 2014 +0300

    Add sync check for models_metadef

    Correct migrations to satisfy models_metadef.
    Add nullable=False for json_schema in metadef_object
    and metadef_property according to changes in
    Change-Id: I2c574210f8d62c77a438afab83ff80f3e5bd2fe7
    Set updated_at nullable=True due to problems
    with setting default value NOW() for datetime
    type for mysql (the same as in models.py).

    Closes-Bug: #1365436

    Change-Id: I5399923621913724914499aad9148a9410faa4ef