Session obtained by SqlAlchemy driver does not use ceilometer.conf's database_connection setting

[Eric Pendergrass]

The SqlAlchemy driver (impl_sqlalchemy.py), when getting a session from openstack's common session class, always receives a session with the default (sqlite) sql_connection configuration. The expected behavior would be for it to use database_connection from ceilometer.conf. The sqlite defaults are hard coded in the session.py.

Here is a trace showing the current behavior:
(Pdb) l
135 url = conf.database_connection
136 if url == 'sqlite://':
137 url = os.environ.get('CEILOMETER_TEST_SQL_URL', url)
138 LOG.info('connecting to %s', url)
139 #sqlalchemy_session.set_defaults('sql_connection="mysql://root:password@localhost:3306/kvs_metering"')
140 -> self.session = sqlalchemy_session.get_session(url, conf)
141
142 def upgrade(self, version=None):
143 migration.db_sync(self.session.get_bind(), version=version)
144
145 def clear(self):
(Pdb) w
...
  /opt/stack/ceilometer/ceilometer/api/hooks.py(44)before()
-> state.request.cfg)
  /opt/stack/ceilometer/ceilometer/storage/impl_sqlalchemy.py(94)get_connection()
-> return Connection(conf)
> /opt/stack/ceilometer/ceilometer/storage/impl_sqlalchemy.py(140)__init__()
-> self.session = sqlalchemy_session.get_session(url, conf)
(Pdb) s
...
> /opt/stack/ceilometer/ceilometer/openstack/common/db/sqlalchemy/session.py(447)get_engine()
-> _ENGINE = create_engine(CONF.sql_connection)
(Pdb) p CONF.sql_connection
'sqlite:////opt/stack/ceilometer/ceilometer/openstack/common/db/ceilometer.sqlite'

This is strange, because on startup the connection info appears correct:
serving on 0.0.0.0:8777, view at http://127.0.0.1:8777
2013-05-22 09:01:51,302 DEBUG [ceilometer.storage][MainThread] looking for 'mysql' driver in 'ceilometer.storage'
2013-05-22 09:01:51 DEBUG [ceilometer.storage] looking for 'mysql' driver in 'ceilometer.storage'
> /opt/stack/ceilometer/ceilometer/storage/impl_sqlalchemy.py(135)__init__()
-> url = conf.database_connection
(Pdb) c
2013-05-22 09:01:54,510 INFO [ceilometer.storage.impl_sqlalchemy][MainThread] connecting to mysql://root:password@localhost:3306/kvs_metering
2013-05-22 09:01:54 INFO [ceilometer.storage.impl_sqlalchemy] connecting to mysql://root:password@localhost:3306/kvs_metering
> /opt/stack/ceilometer/ceilometer/storage/impl_sqlalchemy.py(285)get_meters()
-> query = self.session.query(Resource)

Here's the contents of /etc/ceilometer.conf:
[DEFAULT]
os_auth_url = http://<ip address>:35357/v2.0
os_tenant_name = service
os_password = password
os_username = ceilometer
policy_file = /etc/ceilometer/policy.json
verbose = True
notification_topics = notifications
rabbit_password = password
rabbit_host = localhost
rpc_backend = ceilometer.openstack.common.rpc.impl_kombu
database_connection=mysql://root:password@localhost:3306/kvs_metering
sql_connection=mysql://root:password@localhost:3306/kvs_metering

[keystone_authtoken]
signing_dir = /var/cache/ceilometer
admin_tenant_name = service
admin_password = password
admin_user = ceilometer
auth_protocol = http

I wonder if impl_sqlalchemy's Connection.__init__() method should grab the ceilometer.conf database config info and use session.py's set_defaults() method to set the correct connection info. For this to happen, set_defaults would have to be changed to allow its sql_opts to be overridden by an input parameter.

[Julien Danjou (jdanjou)]

Oh, this is likely due the recent switch to Oslo session connection stuff for SQL I guess :(

[Eric Pendergrass (eap-x)]

Thanks for the comment, JD. Is this something we'll want to fix for the current release? Depending on my workload, I might be able to contribute a patch, but not sure.

[Julien Danjou (jdanjou)]

Yes, it seems quite critical to me. Bonus point if you get merged before next Thursday for havana-1.

[Tong Li (litong01)]

The problem is that the session.py looks for configuration sql_connection, however in ceilometer configuration
there is no mention of sql_connection, only database_connection, so the database_connection configuration
was never used. Thus the dbsync never did anything.

[Eric Pendergrass (eap-x)]

It seems session.py should use database_connection instead of sql_connection since database_connection seems to be the standard name, if I'm not mistaken.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/30752
Committed: http://github.com/openstack/ceilometer/commit/9f6e3639fb4a0b6701b65e2d7ffba80db0b935c3
Submitter: Jenkins
Branch: master

commit 9f6e3639fb4a0b6701b65e2d7ffba80db0b935c3
Author: Gordon Chung <email address hidden>
Date: Tue May 28 11:48:43 2013 -0400

    Session does not use ceilometer.conf's database_connection

    use [database]connection setting from openstack.common sqlalchemy session

    Change-Id: I582d4cb7d51d6ddecc27ae5fcdedcfd19dc898af
    Fixes: bug #1183106