Alembic migrations not tested -- and they don't work for SQLite

Bug #1217156 reported by Jay Pipes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Medium
gordon chung

Bug Description

(py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/migrate_repo$ python manage.py upgrade sqlite:////home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/ceilometer/openstack/common/db/ceilometer.sqlite .
0 -> 1...
done
1 -> 2...
done
2 -> 3...
done
3 -> 4...
done
4 -> 5...
done
5 -> 6...
done
6 -> 7...
done
7 -> 8...
done
8 -> 9...
done
9 -> 10...
done
10 -> 11...
done
11 -> 12...
done
(py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/migrate_repo$ cd ../alembic/
(py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic$ alembic current
Current revision for sqlite:////home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/ceilometer/openstack/common/db/ceilometer.sqlite: None
(py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic$ alembic upgrade head
/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/util.py:163: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
  warnings.warn(msg)
Traceback (most recent call last):
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/bin/alembic", line 8, in <module>
    load_entry_point('alembic==0.6.0', 'console_scripts', 'alembic')()
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/config.py", line 294, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/config.py", line 289, in main
    self.run_cmd(cfg, options)
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/config.py", line 275, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/command.py", line 124, in upgrade
    script.run_env()
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/script.py", line 193, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/util.py", line 177, in load_python_file
    module = load_module(module_id, path)
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/compat.py", line 39, in load_module
    return imp.load_source(module_id, path, fp)
  File "/home/jpipes/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic/env.py", line 77, in <module>
    run_migrations_online()
  File "/home/jpipes/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic/env.py", line 71, in run_migrations_online
    context.run_migrations()
  File "<string>", line 7, in run_migrations
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/environment.py", line 652, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/migration.py", line 224, in run_migrations
    change(**kw)
  File "/home/jpipes/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic/versions/17738166b91_fix_meter_resource_m.py", line 46, in upgrade
    change_type(sa.Text)
  File "/home/jpipes/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic/versions/17738166b91_fix_meter_resource_m.py", line 39, in change_type
    new_column.create(meter)
AttributeError: 'Column' object has no attribute 'create'

It doesn't look like the Alembic migrations have ever been tested properly with SQLite... or anything for that matter.

I do not understand how the base storage SQLAlchemy test case is doing a conn.upgrade() in its setUp() method and tests are actually passing... none of it seems to work when you try it out.

Revision history for this message
Jay Pipes (jaypipes) wrote :

Also, things to keep in mind... Alembic does not handle a number of SQLite ALTER TABLE missing operations, namely DROP COLUMN and RENAME COLUMN:

https://bitbucket.org/zzzeek/alembic/issue/21/column-renames-not-supported-on-sqlite

We worked around this with SQLite specialized migration SQL scripts in sqlalchemy-migrate, but the same does not exist in Alembic. Instead it looks like we will need to jump through a number of hoops -- including copying table schemas and data into new table schemas.

Julien Danjou (jdanjou)
Changed in ceilometer:
status: New → Triaged
milestone: none → havana-rc1
importance: Undecided → Medium
Guangyu Suo (yugsuo)
Changed in ceilometer:
assignee: nobody → Guangyu Suo (yugsuo)
Revision history for this message
Jay Pipes (jaypipes) wrote :

Guangyu, please note I am working on removing Alembic from Ceilometer for the Havana release:

https://review.openstack.org/#/c/46841/

Best,
-jay

Revision history for this message
Guangyu Suo (yugsuo) wrote : Re: [Bug 1217156] Re: Alembic migrations not tested -- and they don't work for SQLite
Download full text (4.9 KiB)

Oh, thanks, that's greate , I will keep an eye on your pathsets.

2013/9/23 Jay Pipes <email address hidden>

> Guangyu, please note I am working on removing Alembic from Ceilometer
> for the Havana release:
>
> https://review.openstack.org/#/c/46841/
>
> Best,
> -jay
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1217156
>
> Title:
> Alembic migrations not tested -- and they don't work for SQLite
>
> Status in Ceilometer:
> Triaged
>
> Bug description:
> (py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/migrate_repo$
> python manage.py upgrade
> sqlite:////home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/ceilometer/openstack/common/db/ceilometer.sqlite
> .
> 0 -> 1...
> done
> 1 -> 2...
> done
> 2 -> 3...
> done
> 3 -> 4...
> done
> 4 -> 5...
> done
> 5 -> 6...
> done
> 6 -> 7...
> done
> 7 -> 8...
> done
> 8 -> 9...
> done
> 9 -> 10...
> done
> 10 -> 11...
> done
> 11 -> 12...
> done
> (py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/migrate_repo$
> cd ../alembic/
> (py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic$
> alembic current
> Current revision for
> sqlite:////home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/ceilometer/openstack/common/db/ceilometer.sqlite:
> None
> (py27)jpipes@uberbox:~/repos/openstack/ceilometer/ceilometer/storage/sqlalchemy/alembic$
> alembic upgrade head
>
> /home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/util.py:163:
> UserWarning: Skipping unsupported ALTER for creation of implicit constraint
> warnings.warn(msg)
> Traceback (most recent call last):
> File "/home/jpipes/repos/openstack/ceilometer/.tox/py27/bin/alembic",
> line 8, in <module>
> load_entry_point('alembic==0.6.0', 'console_scripts', 'alembic')()
> File
> "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/config.py",
> line 294, in main
> CommandLine(prog=prog).main(argv=argv)
> File
> "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/config.py",
> line 289, in main
> self.run_cmd(cfg, options)
> File
> "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/config.py",
> line 275, in run_cmd
> **dict((k, getattr(options, k)) for k in kwarg)
> File
> "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/command.py",
> line 124, in upgrade
> script.run_env()
> File
> "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/script.py",
> line 193, in run_env
> util.load_python_file(self.dir, 'env.py')
> File
> "/home/jpipes/repos/openstack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/alembic/util.py",
> line 177, in load_python_file
> module = load_module(module_id, path)
> File
> "/home/jpipes/repos/openstack/ceilometer/.tox/py2...

Read more...

Revision history for this message
Guangyu Suo (yugsuo) wrote :

Oh, thanks, that's greate , I will keep an eye on your pathsets.

Julien Danjou (jdanjou)
Changed in ceilometer:
assignee: Guangyu Suo (yugsuo) → nobody
Julien Danjou (jdanjou)
Changed in ceilometer:
milestone: havana-rc1 → none
tags: added: havana-rc-potential
Thierry Carrez (ttx)
tags: added: havana-backport-potential
removed: havana-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/71688

Changed in ceilometer:
assignee: nobody → gordon chung (chungg)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/71688
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2cf06e9ab471e9bcaa5ba954224f006607197b4b
Submitter: Jenkins
Branch: master

commit 2cf06e9ab471e9bcaa5ba954224f006607197b4b
Author: Gordon Chung <email address hidden>
Date: Thu Feb 6 18:41:44 2014 -0500

    Alembic migrations not tested

    take alembic migrations out until we can confidently make the
    switch to it once real db tests are set up.

    Closes-Bug: #1217156
    Change-Id: Ic62c5248dc92a0a0eb497b6f278d42070606abe9

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: none → icehouse-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.