db layer test resets interact badly with slave appserver instances

[Robert Collins]

External appservers like those started by ZopelessAppserverLayer talk to the same test DB as the test being executed.

If there are two tests in the layer, and one of them triggers a db reset (e.g. by doing a commit() rather than a rollback where we can just reset the db sequences), then the slave appserver is forcibly disconnected from the database (as is the librarian and other helpers).

Subsequent requests to the appserver will trigger oopses (which will retry so the thing -works- but we have an oops logged. And, because the next requestor is the next test, that test will see 'spurious' oopses if and only if it is run directly after a test that caused the testdb to be dropped (by committing).

Note that it is entirely appropriate for the test appserver to raise OOPSes - in production we care a great deal about noticing such events.

oops-3: {{{
Oops-Id: OOPS-ec2ee8085482c92c96e7e288cb8ca88e
Exception-Type: AssertionError
Exception-Value: Bug #504291: Store left in a disconnected state.
Date: 2011-10-31T00:30:10.023000+00:00
Page-Id:
Branch: useoops
Revision: 14111
User: unauthenticated, Anonymous, Anonymous, Anonymous User
URL: http://launchpad.dev:8085
Duration: -1
Oops-Reporter: T

CHANNEL_CREATION_TIME=1320021009.97
CONTENT_LENGTH=286
CONTENT_TYPE=text/xml
GATEWAY_INTERFACE=CGI/1.1
HTTP_HOST=xmlrpc-private.launchpad.dev:8087
HTTP_USER_AGENT=xmlrpclib.py/1.0.1 (by www.pythonware.com)
PATH_INFO=/codehosting
REMOTE_ADDR=127.0.0.1
REQUEST_METHOD=POST
SCRIPT_NAME=
SERVER_NAME=lucid-test-lp
SERVER_PORT=8087
SERVER_PROTOCOL=HTTP/1.0
SERVER_SOFTWARE=zope.server.http (PrivateXMLRPC)
wsgi.input=%3Chidden%3E
xmlrpc args=%5B%27+launchpad-services%27, %27//%7Eperson-name-425389/product-name-425384/.bzr/branch-format%27%5D
zc.zservertracelog.interfaces.ITraceLog=%3Chidden%3E

Traceback (most recent call last):
  Module canonical.launchpad.webapp.publication, line 762, in endRequest
    "Bug #504291: Store left in a disconnected state.")
AssertionError: Bug #504291: Store left in a disconnected state.
}}}

[Robert Collins (lifeless)]

(there are also oopses like
oops-4: {{{
Oops-Id: OOPS-a1fd7ae1c2a945a8867824c5e636b891
Exception-Type: DisconnectionError
Exception-Value: terminating connection due to administrator command server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
Date: 2011-10-31T00:30:10.114000+00:00
Page-Id:
Branch: useoops
Revision: 14111
User: unauthenticated, Anonymous, Anonymous, Anonymous User
URL: http://launchpad.dev:8085
Duration: 0.00250387191772
Oops-Reporter: T

CHANNEL_CREATION_TIME=1320021010.11
CONTENT_LENGTH=266
CONTENT_TYPE=text/xml
GATEWAY_INTERFACE=CGI/1.1
HTTP_HOST=xmlrpc-private.launchpad.dev:8087
HTTP_USER_AGENT=xmlrpclib.py/1.0.1 (by www.pythonware.com)
PATH_INFO=/codehosting
REMOTE_ADDR=127.0.0.1
REQUEST_METHOD=POST
SCRIPT_NAME=
SERVER_NAME=lucid-test-lp
SERVER_PORT=8087
SERVER_PROTOCOL=HTTP/1.0
SERVER_SOFTWARE=zope.server.http (PrivateXMLRPC)
wsgi.input=%3Chidden%3E
xmlrpc args=%5B%27+launchpad-services%27, %27//%7Eperson-name-425389/.bzr/branch-format%27%5D
zc.zservertracelog.interfaces.ITraceLog=%3Chidden%3E

00000-00000@SQL-main-master SELECT FeatureFlag.date_modified, FeatureFlag.flag, FeatureFlag.priority, FeatureFlag.scope, FeatureFlag."value" FROM FeatureFlag ORDER BY FeatureFlag.flag, FeatureFlag.priority DESC

Traceback (most recent call last):
  Module zope.publisher.publish, line 131, in publish
    obj = request.traverse(obj)
  Module zope.publisher.http, line 455, in traverse
    ob = super(HTTPRequest, self).traverse(obj)
  Module zope.publisher.base, line 250, in traverse
    publication.callTraversalHooks(self, obj)
  Module canonical.launchpad.webapp.publication, line 574, in callTraversalHooks
    notify(BeforeTraverseEvent(ob, request))
  Module zope.event, line 23, in notify
    subscriber(event)
  Module zope.component.event, line 26, in dispatch
    zope.component.subscribers(event, None)
  Module zope.component._api, line 138, in subscribers
    return sitemanager.subscribers(objects, interface)
  Module zope.component.registry, line 323, in subscribers
    return self.adapters.subscribers(objects, provided)
  Module zope.interface.adapter, line 579, in subscribers
    subscription(*objects)
  Module lp.services.profile.profile, line 176, in before_traverse
    if getFeatureFlag('profiling.enabled'):
  Module lp.services.features, line 222, in getFeatureFlag
    return features.getFlag(flag)
  Module lp.services.features.flags, line 261, in getFlag
    return self._known_flags.lookup(flag)
  Module lp.services.features.flags, line 183, in lookup
    v = self._calc(key)
  Module lp.services.features.flags, line 264, in _checkFlag
    self._needRules()
  Module lp.services.features.flags, line 298, in _needRules
    self._rules = self.rule_source.getAllRulesAsDict()
  Module lp.services.features.rulesource, line 56, in getAllRulesAsDict
    for (flag, scope, priority, value) in self.getAllRulesAsTuples():
  Module lp.services.features.rulesource, line 135, in getAllRulesAsTuples
    for r in rs:
  Module storm.store, line 997, in __iter__
    result = self._store._connection.execute(self._get_select())
  Module storm.databases.postgr...