heat doesn't work with SQLAlchemy >= 0.8

[Thomas Goirand]

Hi,

Since python-sqlalchemy 0.8.2 was uploaded to Debian Sid, 167 heat unit tests are failing:

  File "/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/utils.py", line 36, in __get_backend
    self.__backend = __import__(name, None, None, fromlist)
  File "/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/sqlalchemy/api.py", line 20, in <module>
    from heat.db.sqlalchemy import models
  File "/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/sqlalchemy/models.py", line 32, in <module>
    class Json(types.TypeDecorator, types.MutableType):
AttributeError: 'module' object has no attribute 'MutableType'

Indeed, the following is wrong:
class Json(types.TypeDecorator, types.MutableType):

MutableType is completely gone from SQLAlchemy, and should have never been used.

Note that this affects both the Grizzly and Havana branches of Heat, therefore leaving Sid users with no solution to run Heat. Note that reverting the python-sqlalchemy upload in Debian Sid isn't an option, and that the fix should really happen in Heat.

[Clint Byrum (clint-fewbar)]

Not sure we can get this into h2, but lets make sure it is solved by h3 as sqlalchemy 0.8 will come to all of the platforms soon enough.

Users in Debian unstable can, of course, run Heat in a virtualenv with 0.7. I know that is anathema to the Debian way, but it is a workaround.

[Lei Zhang (redheadflyundershadow)]

After checking this carefully, I can fix the bug mentioned by the bug description.
But another error still there, that is db sync, we use sqlalchemy-migrate now, the version is 0.7.2,
can not support sqlalchemy 0.8, the ref is here:
https://code.google.com/p/sqlalchemy-migrate/issues/detail?id=156
due to this error:
Traceback (most recent call last):
  File "heat/tests/test_watch.py", line 64, in setUp
    self.setUpDatabase()
  File "heat/tests/test_watch.py", line 49, in setUpDatabase
    utils.setup_dummy_db()
  File "heat/tests/utils.py", line 92, in setup_dummy_db
    migration.db_sync()
  File "heat/db/migration.py", line 28, in db_sync
    return IMPL.db_sync(version=version)
  File "heat/db/utils.py", line 40, in __getattr__
    backend = self.__get_backend()
  File "heat/db/utils.py", line 36, in __get_backend
    self.__backend = __import__(name, None, None, fromlist)
  File "heat/db/sqlalchemy/migration.py", line 54, in <module>
    from migrate.versioning import api as versioning_api
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 33, in <module>
    from migrate.versioning import (repository, schema, version,
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 10, in <module>
    from sqlalchemy import exceptions as sa_exceptions
ImportError: cannot import name exceptions

It is because sqlalchemy has removed exceptions module.

It also seems that sqlalchemy-migrate would not support sqlalchemy any more, so I suggest we use alembic as migration tools
If it is approved, I will add alembic repo and figure it out by using sqlalchemy >=0.8 in heat (like what neutron done).

What do you think?

Regards
Lei

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/40330

[Lei Zhang (redheadflyundershadow)]

Hi, Thomas
    I already commit code to https://review.openstack.org/40330
and you can test with it.

Regards
Lei

On Tue, Aug 6, 2013 at 10:15 AM, OpenStack Hudson <
<email address hidden>> wrote:

> Fix proposed to branch: master
> Review: https://review.openstack.org/40330
>
> ** Changed in: heat
> Status: Triaged => In Progress
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1199435
>
> Title:
> heat doesn't work with SQLAlchemy >= 0.8
>
> Status in Orchestration API (Heat):
> In Progress
>
> Bug description:
> Hi,
>
> Since python-sqlalchemy 0.8.2 was uploaded to Debian Sid, 167 heat
> unit tests are failing:
>
> File
> "/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/utils.py",
> line 36, in __get_backend
> self.__backend = __import__(name, None, None, fromlist)
> File
> "/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/sqlalchemy/api.py",
> line 20, in <module>
> from heat.db.sqlalchemy import models
> File
> "/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/sqlalchemy/models.py",
> line 32, in <module>
> class Json(types.TypeDecorator, types.MutableType):
> AttributeError: 'module' object has no attribute 'MutableType'
>
> Indeed, the following is wrong:
> class Json(types.TypeDecorator, types.MutableType):
>
> MutableType is completely gone from SQLAlchemy, and should have never
> been used.
>
> Note that this affects both the Grizzly and Havana branches of Heat,
> therefore leaving Sid users with no solution to run Heat. Note that
> reverting the python-sqlalchemy upload in Debian Sid isn't an option,
> and that the fix should really happen in Heat.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/heat/+bug/1199435/+subscriptions
>

[Thomas Goirand (thomas-goirand)]

Hi,
The attached patch is a backport which I did for Grizzly. It worked for me. Thanks Zhang Lei for the original patch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/40330
Committed: http://github.com/openstack/heat/commit/b071b6e73563e84a81f1eed77b05054511692db9
Submitter: Jenkins
Branch: master

commit b071b6e73563e84a81f1eed77b05054511692db9
Author: Zhang Lei (Sneeze) <email address hidden>
Date: Wed Aug 14 10:07:52 2013 +0800

    Handle heat with SQLAlchemy >= 0.8

    Lots of unit tests failed with the new SQLAlchemy(0.8.2)
    which makes build failed, this patch will fix all the related
    bug like no MutableType and changed module name in SQLAlchemy(0.8.2)

    Fixes bug #1199435

    Change-Id: I67d7588db088dd3dd88d44c32e78c6b8a4708bff