model-migration check fails to detect integer-biginteger mismatch

[Henry Gessau]

See https://review.openstack.org/#/c/222079/15/neutron/db/migration/alembic_migrations/versions/mitaka/expand/32e5974ada25_add_neutron_resources_table.py

Kevin had Integer in the model and BigInteger in the migration.
This should be detected by _TestModelsMigrations().

[Kyle Mestery (mestery)]

Yikes, seems like something we should be looking at fixing.

[Ann Taraday (akamyshnikova)]

I'm not sure that this is a Neutron bug. This can be alembic bug or at least oslo.db. Because types compared on alembic level.

[Bo Chi (bochi-michael)]

Considering this is a high priority, if anyone who wants to take this, please do.
I'm just thinking I may not be able to resolve this quickly

[Henry Gessau (gessau)]

Thanks Bo, I have assigned it to Ann for initial tracking down of the source of the problem.

[Ann Taraday (akamyshnikova)]

The issue here is cause that in model is used with_variant type [1] So, compare_type method see [2] and seems that it compares them as equal. This is the only case in Neutron, so there are no other places to fix this. I will propose patch for oslo.db, not sure that there is something that should be done on Neutron side.

[1] - https://github.com/openstack/neutron/blob/master/neutron/db/model_base.py#L100
[2] - http://paste.openstack.org/show/480278/

[Ann Taraday (akamyshnikova)]

Looking deeper I saw that Alembic doesn't see the difference between BigInteger and Integer at all. Will ask Mike Bayer about that.

[Mike Bayer (zzzeek)]

Hi Ann -

can you please provide a code example, or at least what backend you're referring to? Alembic type comparison rules are necessarily affected by backend since the current DB type is first the result of SQLAlchemy creating that type previously, and then round-tripping through reflection from that backend. Also comparison of types is off by default unless a value is specified for the "compare_type" flag is set on the environment.

For the typical targets of MySQL and SQLite the "BigInteger" type should be generating a type of BIGINT which would compare differently upon reflection to INTEGER. A real bug report at https://bitbucket.org/zzzeek/alembic would be most helpful thanks!

[Ann Taraday (akamyshnikova)]

The test is running on MySQL and PostgreSQL backends. On both of them difference of Integer and BigInteger is skipped. In test from oslo.db comapare_type is provided [1], and in fact difference in other types is checked. If I set String in model and Integer in migration test will fail on that. The example of code is shown here [2]. When I debug code I see that compare_type [3] gets meta_type.impl as Integer and insp_type as BIGINT and this difference was not detected. In fact when I change Integer for BigInteger in some places it was not detected either.

[1] - https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/test_migrations.py#L592
[2] - https://review.openstack.org/#/c/222079/15/neutron/db/migration/alembic_migrations/versions/mitaka/expand/32e5974ada25_add_neutron_resources_table.py
[3] - https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/test_migrations.py#L418

I will sum up all of this and report in https://bitbucket.org/zzzeek/alembic.

Thanks!

[Ann Taraday (akamyshnikova)]

Create a bug in alembic https://bitbucket.org/zzzeek/alembic/issues/341/integer-vs-biginteger-diffrence-are-not

[Ann Taraday (akamyshnikova)]

This is issue is resolved for alembic 0.8.4.

[Lajos Katona (lajos-katona)]

As I see the current situation (we use now, in Dalmatian, alembic~1.9.4) and the code it is no more an issue