MongoDB connection failures when entire test suite is run

[Terri Yu]

When I run the Ceilometer test suite with the command "tox -e py27", I get 159 ConnectionFailure errors for the MongoDB tests.

The traceback for all of these errors is:

Traceback (most recent call last):
  File "/opt/stack/ceilometer/ceilometer/tests/base.py", line 71, in skip_if_not_implemented
    return func(*args, **kwargs)
  File "/opt/stack/ceilometer/tests/storage/test_storage_scenarios.py", line 1685, in setUp
    super(EventTestBase, self).setUp()
  File "/opt/stack/ceilometer/ceilometer/tests/db.py", line 43, in setUp
    self.conn = storage.get_connection(cfg.CONF)
  File "/opt/stack/ceilometer/ceilometer/storage/__init__.py", line 81, in get_connection
    return get_engine(conf).get_connection(conf)
  File "/opt/stack/ceilometer/ceilometer/storage/impl_mongodb.py", line 76, in get_connection
    return Connection(conf)
  File "/opt/stack/ceilometer/ceilometer/storage/impl_mongodb.py", line 301, in __init__
    self.conn = self.CONNECTION_POOL.connect(url)
  File "/opt/stack/ceilometer/ceilometer/storage/impl_mongodb.py", line 154, in connect
    safe=True)
  File "/opt/stack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 352, in __init__
    raise ConnectionFailure(str(e))
ConnectionFailure: connection closed

There is also an odd message "No module named deprecated" at the end of the test results.

It seems that there are two ways to get around this:

1) If I increase the maximum number of MongoDB connections from 128 to 256 in the --maxConns argument to mongod in run-tests.sh, the errors go away, but I still get the "No module named deprecated" message.

2) If I only run a part of the test suite (e.g. "tox -e py27 -- storage"), the errors also disappear, though I still get the "No module named deprecated" message.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/44465
Committed: http://github.com/openstack/ceilometer/commit/2152627f1aea1bd8c976b49c7c7a4decd3b22721
Submitter: Jenkins
Branch: master

commit 2152627f1aea1bd8c976b49c7c7a4decd3b22721
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Aug 30 13:52:41 2013 +0200

    Disable the pymongo pooling feature for tests

    Because pymongo doesn't close connection on .close() when we use its
    pool, we disable the MongoClient pooling feature for running tests.

    This allow to use a normal number of connection in mongod for test.

    Fixes bug #1218488

    Change-Id: Ie4c74620937816ed0592f5ac72de99dee3173ad8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/45202

[Julien Danjou (jdanjou)]

Actually appeared again http://logs.openstack.org/74/44774/3/check/gate-ceilometer-python26/96cc90f/console.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45202
Committed: http://github.com/openstack/ceilometer/commit/d17dde508bf1fd3033e911fc503595f1f6320b11
Submitter: Jenkins
Branch: master

commit d17dde508bf1fd3033e911fc503595f1f6320b11
Author: Alexei Kornienko <email address hidden>
Date: Thu Sep 5 12:36:40 2013 +0300

    Improved MongoClient pooling to avoid out of connections error

    Use 1 MongoClient instance per database server determined by host:port +
    connection options

    Fixes bug #1218488

    Change-Id: If06844f6bf09674216b029310c1a5f445c4476fe

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45642
Committed: http://github.com/openstack/ceilometer/commit/84c571acb99bc8dff99c925e8afd589ef7e2a5bb
Submitter: Jenkins
Branch: master

commit 84c571acb99bc8dff99c925e8afd589ef7e2a5bb
Author: Julien Danjou <email address hidden>
Date: Mon Sep 9 11:04:34 2013 +0200

    run-tests: fix MongoDB start wait

    There's actually 2 lines that matches the current rule:

      [websvr] admin web console waiting for connections on port 30000
      [initandlisten] waiting for connections on port 29000

    Let's make it more tighten.

    Related-Bug: #1218488

    Change-Id: I4ba56a7c2406a9851aa8087b4cabb0aa31698763