ZEO test failures on Mac OS X 10.4 PPC

[Philipp von Weitershausen]

ZEO tests from ZODB 3.7 fail on MacOSX (ChrisM confirms that, as well as the fact that they pass on Linux). The failures I get are these:

Error in test testNOK (ZEO.tests.testAuth.PlainTextAuth)
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 270, in run
    self.tearDown()
  File "src/ZEO/tests/testAuth.py", line 54, in tearDown
    self.__super_tearDown()
  File "src/ZEO/tests/ConnectionTests.py", line 124, in tearDown
    forker.shutdown_zeo_server(adminaddr)
  File "src/ZEO/tests/forker.py", line 181, in shutdown_zeo_server
    s.connect(adminaddr)
  File "<string>", line 1, in connect
error: (22, 'Invalid argument')

and many more of them. I tried running the tests or ZODB 3.6 but couldn't get the tests to pick up ZEO tests:

  philipp@bender:~/dev/ZODB36$ python setup.py build
  ...
  philipp@bender:~/dev/ZODB36$ python test.py -ps ZEO
  Running tests from build/lib.darwin-8.7.0-Power_Macintosh-2.4
  Total: 0 tests, 0 failures, 0 errors

Those commands work in ZODB 3.7, except for the test failures I get, obviously.

[Jim Fulton (jim-zope)]

I just checked in a fix that I'd made on the trunk and forgotten to check into the branch. Could you see if it makes the tests pass for you on PPC now?

[Philipp von Weitershausen (philikon)]

They almost pass now. This is the only failure I get:

philipp@Vigor235:~/dev/ZODB37$ python test.py -ps ZEO
Running tests from src
Running unit tests:
  Running:
    14/162 (8.6%)

Failure in test checkReconnectUpgrade (ZEO.tests.testConnection.FileStorageReconnectionTests)
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 260, in run
    testMethod()
  File "src/ZEO/tests/ConnectionTests.py", line 857, in checkReconnectUpgrade
    self.pollUp()
  File "src/ZEO/tests/ConnectionTests.py", line 232, in pollUp
    self.fail("timed out waiting for storage to connect")
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 301, in fail
    raise self.failureException, msg
AssertionError: timed out waiting for storage to connect

The following test left new threads behind:
checkReconnectUpgrade (ZEO.tests.testConnection.FileStorageReconnectionTests)
New thread(s): [<ConnectThread(Connect([(2, ('localhost', 27408))]), started daemon)>]

  Ran 162 tests with 1 failures and 0 errors in 818.417 seconds.

[Jim Fulton (jim-zope)]

Have you tried running the tests more than once?

Was there anything else going on on the machine at the time?

FWIW, my macbook runs these tests in 232 seconds.

I might be able to run these on a PPC mac mini tomorrow.

[Philipp von Weitershausen (philikon)]

I ran them twice again now. The time out error didn't occur anymore. Instead, I twice got this:

philipp@Vigor234:~/dev/ZODB37$ python test.py -ps ZEO
Running tests from src
Running unit tests:
  Running:
    33/162 (20.4%)

Error in test checkVerificationInvalidationPersists (ZEO.tests.testConnection.MappingStorageConnectionTests)
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 260, in run
    testMethod()
  File "src/ZEO/tests/ConnectionTests.py", line 531, in checkVerificationInvalidationPersists
    self._storage = self.openClientStorage('test')
  File "src/ZEO/tests/ConnectionTests.py", line 183, in openClientStorage
    realm=realm)
  File "src/ZEO/ClientStorage.py", line 321, in __init__
    self._wait(wait_timeout)
  File "src/ZEO/ClientStorage.py", line 342, in _wait
    assert self._connection.is_async()
AttributeError: 'NoneType' object has no attribute 'is_async'

  Ran 162 tests with 0 failures and 1 errors in 811.086 seconds.

The machine wasn't doing anythign else in particular, just reading emails, browsing the web, etc.

[Christian Theune (ctheune)]

Changes: submitter email, edited transcript, importance (3.3 Release => critical), new comment

When reviewing the list of blocking things, Jim noted that this only happens on PPC and shouldn't be release-blocking. I'm moving it to critical.

[Jim Fulton (jim-zope)]

I'll keep plugging away at this as I have time, but I'm not very hopeful.

I'm tempted to just remove the assertion.

I have a PPC machine that I can use at home to run the tests. On that machine, I'm getting another error. Note that both errors go away if I run the tests individually. :(

[Philipp von Weitershausen (philikon)]

I agree, it's not release blocking in general, it's just blocking me from making releases (I don't feel very comfortable creating releases when my tests don't all pass).

[Christian Theune (ctheune)]

Well, to assure you, those tests run on other platforms. :)
I'm really keen on getting another beta release (before the RC). *Really*.

[Russ Ferriday (russf)]

Here's a tiny clue...

If you change the 3 'test' parameters to 'testxxx', the test runs fine on OSX.

There are other calls to openClientStorage('test' ...) in previous tests.

Is there something left from a previous test that causes this error? Why only on OSX?

[Russ Ferriday (russf)]

I spotted the need for the patch below while tracking this one.
And, I notice that self.caches ends up with ['trace', 'trace'] which does not seem right.

*** SOMEONE APPLY THIS PLEASE ***

[streak:src/ZEO/tests] russ% svn diff
Index: ConnectionTests.py
===================================================================
--- ConnectionTests.py (revision 69562)
+++ ConnectionTests.py (working copy)
 -132,7 +132,7 @@
         for c in self.caches:
             for i in 0, 1:
                 for ext in "", ".trace":
- base = "%s-%s.zec%s" % (c, "1", ext)
+ base = "%s-%s.zec%s" % (c, i, ext)
                     path = os.path.join(tempfile.tempdir, base)
                     # On Windows before 2.3, we don't have a way to wait for
                     # the spawned server(s) to close, and they inherited

[Russ Ferriday (russf)]

1. Let me rephrase my *** message *** below to : *** Would somebody see if this patch makes sense ***.

2. I'm not sure about the relevance of 'trace' being in self.caches twice. Seems like that should be benign.

3. Changing 3 occurences of openClientStorage('test') to openClientStorage('testx') in checkVerificationInvalidationPersists makes the test run. But I don't know why.

[Baiju Muthukadan (baijum)]

Looks like ZODB issue. Added ZODB to also affects list.

[Jim Fulton (jim-zope)]

This is pretty old. The lines where Philipp last reported failures don't exist any more, I don't think we care that much about PPC anyway.