Marconi hangs when the backend mongoDB becomes unavailable.

[Mike Panetta]

If the Catalog DB becomes unavailable for whatever reason, marconi will hang and stop servicing requests. It seems to retry the connection a few times and then just gives up. Backtrace below.

2014-02-25 16:43:54.296 14180 ERROR marconi.queues.storage.mongodb.utils [-] No replica set members available for query with ReadPreference SECONDARY_PREFERRED
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils Traceback (most recent call last):
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils File "/usr/lib/python2.6/site-packages/marconi/queues/storage/mongodb/utils.py", line 252, in wrapper
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils return func(*args, **kwargs)
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils File "/usr/lib/python2.6/site-packages/marconi/queues/storage/mongodb/utils.py", line 321, in next
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils item = next(self.cursor)
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 904, in next
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils if len(self.__data) or self._refresh():
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 848, in _refresh
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils self.__uuid_subtype))
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 782, in __send_message
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils res = client._send_message_with_response(message, **kwargs)
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils File "/usr/lib64/python2.6/site-packages/pymongo/mongo_replica_set_client.py", line 1631, in _send_message_with_response
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils raise AutoReconnect(msg, errors)
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils AutoReconnect: No replica set members available for query with ReadPreference SECONDARY_PREFERRED
2014-02-25 16:43:54.296 14180 TRACE marconi.queues.storage.mongodb.utils
2014-02-25 16:43:54.306 14183 CRITICAL marconi [-] ConnectionError

[Feilong Wang (flwang)]

Hi Mike, please take a look at this patch https://review.openstack.org/#/c/75165/ I think it can fix your issue. Please let me know if it works. Thanks.