Gnocchi

stop expunging on every metricd worker

Bug #1583820 reported by gordon chung
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi
Fix Released
Medium
gordon chung
Gnocchi 2.2.0

Bug Description

right now every metricd worker attempts to clear db of delete records every single cycle. if we scale up metricd workers to 'production' size, it starts throwing connections error. i can increase connections but it still doesn't make sense that every worker (i have 32) is constantly querying db whether it should delete the same set of data. we should move cleanup to it's own single worker like reporting

2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage [-] Unexpected error during deleting metrics
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage Traceback (most recent call last):
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/root/dev/gnocchi/gnocchi/storage/__init__.py", line 193, in process_background_tasks
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage self.expunge_metrics(index, sync) [0/1262]
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/root/dev/gnocchi/gnocchi/storage/__init__.py", line 201, in expunge_metrics
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage metrics_to_expunge = index.list_metrics(status='delete')
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/root/dev/gnocchi/gnocchi/indexer/sqlalchemy.py", line 426, in list_metrics
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage with self.facade.independent_reader() as session:
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return self.gen.next()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 759, in _transaction_scope
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage allow_async=self._allow_async) as resource:
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return self.gen.next()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 491, in _session
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage bind=self.connection, mode=self.mode)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 272, in _create_session
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage self._start()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage engine_args, maker_args)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage sql_connection=sql_connection, **engine_kwargs)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage test_conn = _test_connection(engine, max_retries, retry_interval)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 326, in _test_connection
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return engine.connect()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return self._connection_cls(self, **kwargs)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage if connection is not None else engine.raw_connection()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage self.pool.unique_connection, _connection)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage e, dialect, self)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage util.raise_from_cause(newraise, exc_info)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage reraise(type(exception), exception, tb=exc_tb)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return fn()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return _ConnectionFairy._checkout(self)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage fairy = _ConnectionRecord.checkout(pool)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage rec = pool._do_get()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage self._dec_overflow()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage compat.reraise(exc_type, exc_value, exc_tb)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return self._create_connection()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return _ConnectionRecord(self)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage self.connection = self.__connect()
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage connection = self.__pool._invoke_creator(self)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return dialect.connect(*cargs, **cparams)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage return self.dbapi.connect(*cargs, **cparams)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage conn = _connect(dsn, connection_factory=connection_factory, async=async)
2016-05-19 18:02:24.818 4844 ERROR gnocchi.storage OperationalError: (psycopg2.OperationalError) FATAL: remaining connection slots are reserved for non-replication superuser connections

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to gnocchi (master)

Fix proposed to branch: master
Review: https://review.openstack.org/319383

Changed in gnocchi:
assignee: nobody → gordon chung (chungg)
status: New → In Progress
Julien Danjou (jdanjou)
Changed in gnocchi:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to gnocchi (master)

Reviewed: https://review.openstack.org/319383
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=24d2854f604a4b4f1178aa90b677ae3256a2da9c
Submitter: Jenkins
Branch: master

commit 24d2854f604a4b4f1178aa90b677ae3256a2da9c
Author: gord chung <email address hidden>
Date: Fri May 20 13:26:11 2016 -0400

    separate cleanup into own worker

    Change-Id: I5938cbb641cc746ac15ff5b864fa7e7657863676
    Closes-Bug: #1583820

Changed in gnocchi:
status: In Progress → Fix Committed
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/gnocchi 2.2.0

This issue was fixed in the openstack/gnocchi 2.2.0 release.

Julien Danjou (jdanjou)
Changed in gnocchi:
status: Fix Committed → Fix Released
milestone: none → 2.2.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.