For reference, we found the bug that seems same cause.
[ Issue ]
When we ran the ceilometer-expirer and following DB Erroro has occurred.
2014-03-03 18:06:36.734 31990 CRITICAL ceilometer [-] (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`ceilometer`.`meter`, CONSTRAINT `fk_meter_project_id` FOREIGN KEY (`project_id`) REFERENCES `project` (`id`))') 'DELETE FROM project WHERE project.id = %s' ('e6e689cd30e049c1a0f66b21d6c183e8',)
2014-03-03 18:06:36.734 31990 TRACE ceilometer Traceback (most recent call last):
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/bin/ceilometer-expirer", line 10, in <module>
2014-03-03 18:06:36.734 31990 TRACE ceilometer sys.exit(expirer())
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/lib/python2.7/dist-packages/ceilometer/storage/__init__.py", line 167, in expirer
2014-03-03 18:06:36.734 31990 TRACE ceilometer cfg.CONF.database.time_to_live)
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/lib/python2.7/dist-packages/ceilometer/storage/impl_sqlalchemy.py", line 344, in clear_expired_metering_data
2014-03-03 18:06:36.734 31990 TRACE ceilometer for res_obj in query.all():
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all
2014-03-03 18:06:36.734 31990 TRACE ceilometer return list(self)
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2226, in __iter__
2014-03-03 18:06:36.734 31990 TRACE ceilometer self.session._autoflush()
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1127, in _autoflush
2014-03-03 18:06:36.734 31990 TRACE ceilometer self.flush()
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/lib/python2.7/dist-packages/ceilometer/openstack/common/db/sqlalchemy/session.py", line 594, in _wrap
2014-03-03 18:06:36.734 31990 TRACE ceilometer raise exception.DBError(e)
2014-03-03 18:06:36.734 31990 TRACE ceilometer DBError: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`ceilometer`.`meter`, CONSTRAINT `fk_meter_project_id` FOREIGN KEY (`project_id`) REFERENCES `project` (`id`))') 'DELETE FROM project WHERE project.id = %s' ('e6e689cd30e049c1a0f66b21d6c183e8',)
2014-03-03 18:06:36.734 31990 TRACE ceilometer
[ Reproduce ]
1. ceilometer-expirer TTL is set to 10sec. (for generating the error, use short TTL)
2. Register 300 images in Glance.
3. CentralAgent is running and 600 cases registered in meter (result meter)
4. When the next CentralAgent's polling starts, run the ceilometer-expirer
[ Note ]
I think, Ceilometer isn't handling a conflict processing appropriate.
I saw review https://review.openstack.org/#/c/72414/ .
So I have understood, you have the plan of changing data model.
I think it's really important.
Could you share the progress of changing data model?
For reference, we found the bug that seems same cause.
[ Issue ]
When we ran the ceilometer-expirer and following DB Erroro has occurred.
2014-03-03 18:06:36.734 31990 CRITICAL ceilometer [-] (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`ceilometer` .`meter` , CONSTRAINT `fk_meter_ project_ id` FOREIGN KEY (`project_id`) REFERENCES `project` (`id`))') 'DELETE FROM project WHERE project.id = %s' ('e6e689cd30e04 9c1a0f66b21d6c1 83e8',) bin/ceilometer- expirer" , line 10, in <module> lib/python2. 7/dist- packages/ ceilometer/ storage/ __init_ _.py", line 167, in expirer database. time_to_ live) lib/python2. 7/dist- packages/ ceilometer/ storage/ impl_sqlalchemy .py", line 344, in clear_expired_ metering_ data lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2115, in all lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2226, in __iter__ _autoflush( ) lib/python2. 7/dist- packages/ sqlalchemy/ orm/session. py", line 1127, in _autoflush lib/python2. 7/dist- packages/ ceilometer/ openstack/ common/ db/sqlalchemy/ session. py", line 594, in _wrap DBError( e) .`meter` , CONSTRAINT `fk_meter_ project_ id` FOREIGN KEY (`project_id`) REFERENCES `project` (`id`))') 'DELETE FROM project WHERE project.id = %s' ('e6e689cd30e04 9c1a0f66b21d6c1 83e8',)
2014-03-03 18:06:36.734 31990 TRACE ceilometer Traceback (most recent call last):
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/
2014-03-03 18:06:36.734 31990 TRACE ceilometer sys.exit(expirer())
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/
2014-03-03 18:06:36.734 31990 TRACE ceilometer cfg.CONF.
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/
2014-03-03 18:06:36.734 31990 TRACE ceilometer for res_obj in query.all():
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/
2014-03-03 18:06:36.734 31990 TRACE ceilometer return list(self)
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/
2014-03-03 18:06:36.734 31990 TRACE ceilometer self.session.
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/
2014-03-03 18:06:36.734 31990 TRACE ceilometer self.flush()
2014-03-03 18:06:36.734 31990 TRACE ceilometer File "/usr/local/
2014-03-03 18:06:36.734 31990 TRACE ceilometer raise exception.
2014-03-03 18:06:36.734 31990 TRACE ceilometer DBError: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`ceilometer`
2014-03-03 18:06:36.734 31990 TRACE ceilometer
[ Reproduce ]
1. ceilometer-expirer TTL is set to 10sec. (for generating the error, use short TTL)
2. Register 300 images in Glance.
3. CentralAgent is running and 600 cases registered in meter (result meter)
4. When the next CentralAgent's polling starts, run the ceilometer-expirer
[ Note ]
I think, Ceilometer isn't handling a conflict processing appropriate.
I saw review https:/ /review. openstack. org/#/c/ 72414/ .
So I have understood, you have the plan of changing data model.
I think it's really important.
Could you share the progress of changing data model?