bzr selftest failures and hanging unit tests?

[Bob Tanner]

In my attempt to troubleshoot my own problem documented at https://bugs.launchpad.net/bzr/+bug/137681, I decided to run the selftest. It looks like the selftest won't pass on osx+python-2.5, details below.

$ bzr selftest
testing: /Users/tanner/projects/bzr.dev/bzr
   /Users/tanner/projects/bzr.dev/bzrlib (0.91.0dev0 python2.5.1.final.0)

XFAIL: test_bundle.V08BundleTester.test_unicode_bundle
Mac OSX doesn't preserve unicode combining characters.

XFAIL: test_bundle.V09BundleKnit1Tester.test_unicode_bundle
Mac OSX doesn't preserve unicode combining characters.

XFAIL: test_bundle.V09BundleKnit2Tester.test_unicode_bundle
Mac OSX doesn't preserve unicode combining characters.

XFAIL: test_bundle.V4BundleTester.test_unicode_bundle
Mac OSX doesn't preserve unicode combining characters.

FAIL: test_bundle.V4WeaveBundleTester.test_unicode_bundle
    not equal:
a = [(u'with Dod\xe9', 'withdod-id', 'file')]
b = []

ERROR: test_msgeditor.MsgEditorTest.test_edit_commit_message
    'ascii' codec can't encode character u'\u1234' in position 0: ordinal not in range(128)

FAIL: test_options.TestOptionDefinitions.test_option_grammar
    The following options don't match the style guide:
cvsps-import use-rcs Use rcs to extract texts. (default)

ERROR: test_permissions.TestPermissions.test_new_files
    [Errno 1] Operation not permitted: '.bzr'

FAIL: test_plugins.TestPlugins.test_trivial_plugin_get_path
    not equal:
a = '/tmp/testbzr-XvmPzj.tmp/tmppQKSSf/work/plugin.py'
b = '/private/tmp/testbzr-XvmPzj.tmp/tmppQKSSf/work/plugin.py'

FAIL: test_selftest.TestTestCaseInTempDir.test_home_is_not_working
    not equal:
a = '/tmp/testbzr-XvmPzj.tmp/tmp0GEhX9/work'
b = u'/private/tmp/testbzr-XvmPzj.tmp/tmp0GEhX9/work'

FAIL: test_selftest.TestTestCaseWithMemoryTransport.test_cwd_is_TEST_ROOT
    not equal:
a = '/tmp/testbzr-XvmPzj.tmp'
b = u'/private/tmp/testbzr-XvmPzj.tmp'

FAIL: test_sftp_transport.SFTPTransportTestRelative.test__remote_path
    not equal:
a = '/tmp/testbzr-XvmPzj.tmp/tmpsZIQKl/work/relative'
b = '/private/tmp/testbzr-XvmPzj.tmp/tmpsZIQKl/work/relative'

[3499/8273 in 369s, 2 errors, 6 failed, 4 known failures, 91 skipped, 6 missingerror: uncaptured python exception, closing channel <medusa.ftp_server.passive_acceptor listening 127.0.0.1:64804 at 0x44818c8> (<class 'socket.error'>:(24, 'Too many open files') [/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asyncore.py|read|68] [/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asyncore.py|handle_read_event|384] [build/bdist.macosx-10.3-fat/egg/medusa/ftp_server.py|handle_accept|832] [/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asyncore.py|accept|321] [/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/socket.py|accept|167])

At this point the tests just hang. No CPU utilization from python, nothing in top, and the process looks to stay in the sleep state.

20356 p2 S+ 2:07.71 /Library/Frameworks/Python.framework/Versions/2.5/Res

Maybe I need to pass a special parameter to the selftest to run on osx?

[Andrew Bennetts (spiv)]

Could you please rerun with -v, i.e. "bzr selftest -v", to identify the hanging test?

From the log so far I'd guess it's an FTP test, but it would be good to know the exact one.

Thanks!

[Bob Tanner (tanner)]

Huge logfile of 'bzr selftest -v'

[Martin Pool (mbp)]

That's certainly huge - 5.1GB when uncompressed!

The log shows a zillion repetitions of

Traceback (most recent call last):
  File "/Users/tanner/projects/bzr.dev/bzrlib/transport/sftp.py", line 811, in run
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/socket.py", line 167, in accept
socket.error: (24, 'Too many open files')

The errors about paths in /tmp are already known in another bug, and it's mac os specific.

[Vincent Ladeuil (vila)]

The 'Too many open files' bug(s) should have been fixed in bzr.dev, could you try again with a recent bzr ?
The errors about paths in /tmp should also be fixed, what remain to be fixed are some tests that fail under /tmp because of permissions problems.
To work around these ones, do:
cd
mkdir bzr-tests
TMPDIR=$HOME/bzr-tests bzr selftest

[Vincent Ladeuil (vila)]

I nominate this bug for the weirdest bug of the year contest.

It has symptoms related to:
- interacting tests,
- erratic tests,
- being able to produce multiple log files in /tmp that get all updated concurrently (as in all the log files get the same content which is itself related to several tests),
- etc

[Vincent Ladeuil (vila)]

I forgot to mention that the 'Too many open files' is surely related to the log file aliasing problem.

[Vincent Ladeuil (vila)]

The root cause was that, when medusa (ftp server) was not available, the setup for the tests requiring it was failing.

During that setup, a new log file was created and more importantly a new logger was configured to write to that file.

Since the setup was failing, the tearDown method was never called, leaving the logger installed.

Subsequent tests were then adding their own log file and logger but were still using the dandling one.

The accumulation of tests requiring medusa and leaving their loggers installed lead to the 'Too many open files' errors and the surprising aliased log files effect.

A fix has been posted to the mailing list.