Comment 2 for bug 1355749

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

After looking at the env a bit, I see the following:

1. Nova services are actually up, but they fail to update their status in the database, so nova-manage services list shows them as disabled

2. Nova services fail to update their state in the database because SQLAlchemy refuses to open a new db connection (http://paste.openstack.org/show/93736/), as it thinks 10+30 connections are already opened (10 in the pool, 30 - overflow)

3. lsof shows that nova-conductor and other services don't have *any* open connections to MySQL

4. This looks very similar to the known SQLAlchemy issue - https://bitbucket.org/zzzeek/sqlalchemy/issue/2772 , which was fixed in 0.8.3 (http://docs.sqlalchemy.org/en/latest/changelog/changelog_08.html#change-a95b0f6765fdf5ebdf844806fb2aa122) and we are using 0.8.2 right now

5. Restart of the services helps, but that's not an option, of course

Upgrade to the latest SQLAlchemy 0.8.x release should fix this.