Fullstack test db fixture usage conflicts with retargetable db fixture usage

[Maru Newby]

When the fullstack sanity test runs ahead of the retargetable example, the retargetable test fails with the error that follows.

Better coordination is needed between tests that rely on oslo_db's MySqlOpportunisticTestCase and Neutron's SqlFixture to avoid this kind of failure.

2015-04-04 00:58:35.918 | 2015-04-04 00:58:35.845 | {1} neutron.tests.fullstack.test_sanity.TestSanity.test_sanity [3.159554s] ... ok
2015-04-04 00:58:35.937 | 2015-04-04 00:58:35.865 | {1} neutron.tests.retargetable.test_example.TestExample.test_network_lifecycle(Ml2Plugin) [0.013102s] ...

Traceback (most recent call last):
  File "neutron/tests/retargetable/base.py", line 80, in setUp
    self.useFixture(self.client)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 679, in useFixture
    reraise(*exc_info)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 666, in useFixture
    fixture.setUp()
  File "neutron/tests/retargetable/client_fixtures.py", line 75, in setUp
    self.useFixture(testlib_api.SqlFixture())
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/fixtures/fixture.py", line 192, in useFixture
    fixture.setUp()
  File "neutron/tests/unit/testlib_api.py", line 64, in setUp
    engine = db_api.get_engine()
  File "neutron/db/api.py", line 39, in get_engine
    facade = _create_facade_lazily()
  File "neutron/db/api.py", line 32, in _create_facade_lazily
    _FACADE = session.EngineFacade.from_config(cfg.CONF, sqlite_fk=True)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 880, in from_config
    retry_interval=conf.database.retry_interval)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 796, in __init__
    **engine_kwargs)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 413, in create_engine
    _test_connection(engine, max_retries, retry_interval)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 583, in _test_connection
    return engine.connect()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1890, in connect
    return self._connection_cls(self, **kwargs)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 60, in __init__
    self.__connection = connection or engine.raw_connection()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 225, in raw_connection
    self.pool.unique_connection, _connection)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 219, in _wrap_pool_connect
    e, dialect, self)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 324, in _handle_dbapi_exception_noconnection
    six.reraise(type(newraise), newraise, exc_info[2])
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 215, in _wrap_pool_connect
    return fn()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 280, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 645, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 440, in checkout
    rec = pool._do_get()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 964, in _do_get
    return self._create_connection()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 285, in _create_connection
    return _ConnectionRecord(self)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 411, in __init__
    self.connection = self.__connect()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 539, in __connect
    connection = self.__pool._creator()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 200, in connect
    raise original_exception
sqlalchemy.exc.OperationalError: (OperationalError) (1049, "Unknown database 'kvljuegtta'") None None

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/170915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/170915
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b7ac8501a0a4753be09b37525ec1665bafacdd8b
Submitter: Jenkins
Branch: master

commit b7ac8501a0a4753be09b37525ec1665bafacdd8b
Author: Maru Newby <email address hidden>
Date: Mon Apr 6 16:28:00 2015 +0000

    Skip example retargetable functional test

    The example retargetable test that previously ran as part of the
    functional suite is now skipped due to the fullstack example's db
    fixture usage causing the test to fail if it the fullstack example
    runs first on the same worker.

    Change-Id: I0a34f9ba04c53a4291698be819070c66009c8b4a
    Related-Bug: #1440797

[John Schwarz (jschwarz)]

The problem lies in the fact that prior to every full-stack test case, a new DB is created and the 'connection' option is changed to reflect this new address. After the test, this string is not reverted back to normal.

There are 2 solutions I see for this issue:

1. Having the base Retargetable class inherit from the base Fullstack class. This will let us remove the dependency of the retargetable tests in SqlDBFixture (full stack will take care of creating a new DB which is populated with tables).

2. Reverting back the connection string after each test. afaik, oslo.db depends on this string and clears the DB at that address, so reverting it back will ensure that the retargetable tests will work 'as before'.

Option one doesn't look right - it will lead us down the path where a lot of db-based functional/api tests will have to be integrated into full stack, and that's just plain wrong. Option two sounds like the better lead though I'm not sure it will be smooth-sailing in regards to fixing the problem.

I'll upload a new patchset to fix this problem according to option 2, though I won't have to follow up on this (it's actually a holiday here and I shouldn't be working). It will be simple enough to understand the way forward if the code fails though.

John.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/172129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172129
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e4095758868f6debdddb5a7cd65f8c0a244bee66
Submitter: Jenkins
Branch: master

commit e4095758868f6debdddb5a7cd65f8c0a244bee66
Author: John Schwarz <email address hidden>
Date: Thu Apr 9 18:41:06 2015 +0300

    Revert connection option post full-stack tests

    The full-stack framework overrides the database connection string before
    every test is started, but after the test it doesn't revert the string
    back to what it was originally. Since after the test the database is
    deleted, the string is not actually valid once the test finished, and
    this conflicts with tests which are ran on the same job (specifically
    the retargetable tests - see associated bug). The proposed patch saves
    the original connection string and reverts it after the test finishes.

    Change-Id: I96c01483009084cbc2b81588a1283e84e6bcb4c4
    Closes-bug: #1440797

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072