mariadb 10.2 breaks server default comparison

[Mike Bayer]

I'm still triaging the exact nature of this. Using mariadb 10.2, failure in oslo.db like:

_____________________________________________________________ ModelsMigrationsSyncMysql.test_models_sync _____________________________________________________________
NOTE: Incompatible Exception Representation, displaying natively:

_StringException: Traceback (most recent call last):
  File "/home/classic/dev/redhat/openstack/oslo.db/oslo_db/sqlalchemy/test_migrations.py", line 615, in test_models_sync
    alembic.autogenerate.compare_metadata(mc, self.get_metadata()))
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/api.py", line 102, in compare_metadata
    migration_script = produce_migrations(context, metadata)
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/api.py", line 132, in produce_migrations
    compare._populate_migration_script(autogen_context, migration_script)
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/compare.py", line 22, in _populate_migration_script
    _produce_net_changes(autogen_context, upgrade_ops)
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/compare.py", line 48, in _produce_net_changes
    autogen_context, upgrade_ops, schemas
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/util/langhelpers.py", line 313, in go
    fn(*arg, **kw)
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/compare.py", line 75, in _autogen_for_tables
    inspector, upgrade_ops, autogen_context)
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/compare.py", line 176, in _compare_tables
    modify_table_ops, autogen_context, inspector):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/compare.py", line 260, in _compare_columns
    schema, tname, colname, conn_col, metadata_col
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/util/langhelpers.py", line 313, in go
    fn(*arg, **kw)
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/autogenerate/compare.py", line 762, in _compare_server_default
    rendered_conn_default
  File "/home/classic/dev/redhat/openstack/oslo.db/.tox/py27/lib/python2.7/site-packages/alembic/runtime/migration.py", line 448, in _compare_server_default
    rendered_metadata_default
  File "/home/classic/dev/redhat/openstack/oslo.db/oslo_db/sqlalchemy/test_migrations.py", line 468, in compare_server_default
    meta_def)
  File "/home/classic/dev/redhat/openstack/oslo.db/oslo_db/sqlalchemy/utils.py", line 934, in __call__
    self._url_from_target(target), target, arg, kw)
  File "/home/classic/dev/redhat/openstack/oslo.db/oslo_db/sqlalchemy/utils.py", line 971, in _dispatch_on
    return self._dispatch_on_db_driver(dbname, driver, arg, kw)
  File "/home/classic/dev/redhat/openstack/oslo.db/oslo_db/sqlalchemy/utils.py", line 1000, in _dispatch_on_db_driver
    return self._invoke_fn(fn, arg, kw)
  File "/home/classic/dev/redhat/openstack/oslo.db/oslo_db/sqlalchemy/utils.py", line 974, in _invoke_fn
    return fn(*arg, **kw)
  File "/home/classic/dev/redhat/openstack/oslo.db/oslo_db/sqlalchemy/test_migrations.py", line 490, in _compare_server_default
    return meta_def.arg != insp_def.split("'")[1]
IndexError: list index out of range

also breaks downstream projects, such as glance:

_____________________________________________________________ ModelsMigrationsSyncMysql.test_models_sync _____________________________________________________________
NOTE: Incompatible Exception Representation, displaying natively:

_StringException: Traceback (most recent call last):
  File "/home/classic/dev/redhat/openstack/glance/.tox/py27/lib/python2.7/site-packages/oslo_db/sqlalchemy/test_migrations.py", line 619, in test_models_sync
    "Models and migration scripts aren't in sync:\n%s" % msg)
  File "/home/classic/dev/redhat/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: Models and migration scripts aren't in sync:
[ [ ( 'modify_default',
      None,
      'images',
      'protected',
      { 'existing_nullable': False,
        'existing_type': TINYINT(display_width=1)},
      DefaultClause(<sqlalchemy.sql.elements.TextClause object at 0x7f73b23a3350>, for_update=False),
      DefaultClause(<sqlalchemy.sql.elements.False_ object at 0x7f73b2f25510>, for_update=False))]]

this is confirmed exactly on mariadb 10.2 only, works on mariadb 10.1, MySQL 5.6, 5.7. I think I've encountered this issue upstream as well am going to investigate further.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545958

[Mike Bayer (zzzeek)]

There are also failures in alembic upstream with server default comparison working through review: https://bitbucket.org/zzzeek/alembic/issues/483/mysql-server-default-dont-work-at-all-due

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545958
Committed: https://git.openstack.org/cgit/openstack/oslo.db/commit/?id=3b260a3bca8c9af6fc81a122e8696e09802a6881
Submitter: Zuul
Branch: master

commit 3b260a3bca8c9af6fc81a122e8696e09802a6881
Author: Mike Bayer <email address hidden>
Date: Mon Feb 19 14:01:06 2018 -0500

    Conditionally adjust for quoting in comparing MySQL defaults

    MariaDB 10.2 appears to not return server defaults with quotes
    around integer values which breaks the assumption that we have
    to de-quote server default values. Make the dequoting
    a regexp that will pass when the quotes are not present.

    Change-Id: Ie5aeffcc3c550673a7fdd82769a315821cebb272
    Closes-bug: #1750414

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.db 4.34.0 release.