test failure: test_pgbouncer_stopped

[David Britton]

So far, I have seen this on precise, not sure if it's an error introduced by a new dependency, ask me for more troubleshooting if you need it. I have repeated it across a couple of environments including one where I just did dist-upgrade as of today (19 Mar, 20:30 UTC) followed by a fresh boot.

======================================================================
ERROR: test_pgbouncer_stopped (tests.databases.postgres.PostgresDisconnectionTestWithPGBouncer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/jenkins/workspace/storm/label/precise/tests/mocker.py", line 102, in test_method_wrapper
    result = test_method()
  File "/var/jenkins/workspace/storm/label/precise/tests/databases/postgres.py", line 696, in test_pgbouncer_stopped
    "SELECT current_database()")
  File "/usr/lib/python2.7/unittest/case.py", line 471, in assertRaises
    callableObj(*args, **kwargs)
  File "/var/jenkins/workspace/storm/label/precise/storm/databases/postgres.py", line 266, in execute
    return Connection.execute(self, statement, params, noresult)
  File "/var/jenkins/workspace/storm/label/precise/storm/database.py", line 238, in execute
    raw_cursor = self.raw_execute(statement, params)
  File "/var/jenkins/workspace/storm/label/precise/storm/databases/postgres.py", line 276, in raw_execute
    return Connection.raw_execute(self, statement, params)
  File "/var/jenkins/workspace/storm/label/precise/storm/database.py", line 322, in raw_execute
    self._check_disconnect(raw_cursor.execute, *args)
  File "/var/jenkins/workspace/storm/label/precise/storm/database.py", line 371, in _check_disconnect
    return function(*args, **kwargs)
DatabaseError: error with no message from the libpq

----------------------------------------------------------------------

pgbouncer 1.4.2-2build1
libpq5 9.1.3-2
postgresql 9.1+129
postgresql-8.4 8.4.10-1
postgresql-9.1 9.1.3-2

[William Grant (wgrant)]

It's the new version of psycopg2. See also http://psycopg.lighthouseapp.com/projects/62710/tickets/82 and https://github.com/ask/kombu/issues/110.

[William Grant (wgrant)]

In the case of a pgbouncer disconnection during a rollback, psycopg2 2.2 raises a ProgrammingError with no message. That's hacked around in Storm:

        if isinstance(exc, disconnection_errors):
            # When the connection is closed by a termination of pgbouncer, a
            # ProgrammingError is raised. If the raw connection is closed we
            # assume it's actually a disconnection.
            if isinstance(exc, ProgrammingError):
                if self._raw_connection.closed:
                    return True

But 2.4.x (with the same libpq5) now returns a DatabaseError('error with no message from the libpq',), which isn't caught here. We might just be able to sensibly catch DatabaseError here.

[William Grant (wgrant)]

I'll extend the check to catch DatabaseError and undisable the tests that were disabled in bug #900702.