neutron-db-manage fails with postgres sql

[Ryu Ishimoto]

While testing on Suse environment that uses postgres SQL, running neutron-db-manage caused this error:

root@d08-00-27-80-e3-5e:~ # /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/midonet/midonet.ini upgrade head" neutron
No handlers could be found for logger "oslo_config.cfg"
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
  Running upgrade for neutron ...
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
  OK
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
  Running upgrade for networking-midonet ...
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> start_neutron_midonet, start neutron-midonet chain
INFO [alembic.runtime.migration] Running upgrade start_neutron_midonet -> 25aeae45d4ad, add task
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 565, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 196, in do_upgrade
    desc=branch, sql=CONF.command.sql)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 108, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 397, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/site-packages/midonet/neutron/db/migration/alembic_migration/env.py", line 86, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/site-packages/midonet/neutron/db/migration/alembic_migration/env.py", line 77, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/site-packages/alembic/runtime/migration.py", line 312, in run_migrations
    step.migration_fn(**kw)
  File "/usr/lib/python2.7/site-packages/midonet/neutron/db/migration/alembic_migration/versions/25aeae45d4ad_add_task.py", line 42, in upgrade
    sa.Column('created_at', sa.DateTime(), nullable=False),)
  File "<string>", line 8, in create_table
  File "<string>", line 3, in create_table
  File "/usr/lib/python2.7/site-packages/alembic/operations/ops.py", line 1098, in create_table
    return operations.invoke(op)
  File "/usr/lib/python2.7/site-packages/alembic/operations/base.py", line 318, in invoke
    return fn(self, operation)
  File "/usr/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 101, in create_table
    operations.impl.create_table(table)
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 194, in create_table
    self._exec(schema.CreateTable(table))
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 968, in _execute_ddl
    compiled
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
oslo_db.exception.DBError: (psycopg2.ProgrammingError) type modifier is not allowed for type "text"
LINE 6: data TEXT(16777216),
              ^
 [SQL: '\nCREATE TABLE midonet_tasks (\n\tid SERIAL NOT NULL, \n\ttype VARCHAR(36), \n\tdata_type VARCHAR(36), \n\tdata TEXT(16777216), \n\tresource_id VARCHAR(36), \n\ttenant_id VARCHAR(255), \n\ttransaction_id VARCHAR(40) NOT NULL, \n\tcreated_at TIMESTAMP WITHOUT TIME ZONE NOT NULL, \n\tPRIMARY KEY (id)\n)\n\n']

[YAMAMOTO Takashi (yamamoto)]

which version are you using?
do you have I78ae899df080a7aa1bb78ed75d8e43a9bf75103e ?

[Ryu Ishimoto (ryu-midokura)]

It's liberty/1.0.0 - fcab0c549b47c7e145e60e5c31d48779a5f4ba87

But this problem comes from:

https://github.com/openstack/networking-midonet/blob/stable/liberty/midonet/neutron/db/migration/alembic_migration/versions/25aeae45d4ad_add_task.py#L38

So any version that contains this migration script will fail. Changing that type from Text to String fixes it. But that means we are changing the already-released migration script.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/292016

[YAMAMOTO Takashi (yamamoto)]

Ryu,

the migration script was fixed by the above mentioned commit.
i back-ported it to liberty.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/293026

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tony Breeds (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/293026
Reason: This review is being abandoned as the stable/kilo will go End-Of-Life (EOL) in the near future.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292016
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=67e19d9ccc3fbc48d44065754dc0e0825f7e307f
Submitter: Jenkins
Branch: stable/liberty

commit 67e19d9ccc3fbc48d44065754dc0e0825f7e307f
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Dec 10 12:56:59 2015 +0900

    Clarify our use of Text(length) and explicitly restrict it to mysql

    Specify Text length only for mysql. Add a comment to explain
    why we specify this long length.

    SQLAlchemy mysql dialect uses the length argument of Text() to
    choose a backing type. (TEXT, LONGTEXT, ...)
    However Text() usually doesn't need the length argument and
    some dialects (eg. postgresql) bail out with errors when it's given.

    Also, remove the length argument from the model because it doesn't
    serve any purposes.

    Reference: http://docs.sqlalchemy.org/en/rel_1_0/core/type_basics.html#sqlalchemy.types.Text

    Backport note: This commit was edited to exclude test_migrations part.

    Closes-Bug: #1556090
    Change-Id: I78ae899df080a7aa1bb78ed75d8e43a9bf75103e
    (cherry picked from commit 11e7099788233d7281cb4ec67ea4ea9b0dbc1de9)
    Conflicts:
     midonet/neutron/tests/unit/db/test_migrations.py

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/networking-midonet 1.1.0 release.