push hyperlinks and not mere bug numbers into trac

[Zooko Wilcox-O'Hearn]

launchpad pushed a note onto my trac:

http://allmydata.org/trac/tahoe/ticket/402#comment:23

That's cool! But would you please push hyperlinks to the appropriate launchpad page e.g. https://bugs.launchpad.net/pyopenssl/+bug/236190 and not just the bug number?

Thanks!

[zooko (zooko-tahoe-trac)]

Ah, well I tried making Tahoe require pyOpenSSL >= 0.6, != 0.7, but the pyOpenSSL-0.6 tarball is not {{{easy_install}}}'able, as described here:

https://bugs.launchpad.net/pyopenssl/+bug/236190

If the pyOpenSSL maintainers fix the 0.6 tarball's permission bits as I submitted in that ticket, then this will start working.

[zooko (zooko-tahoe-trac)]

If this gets fixed so that pyOpenSSL can be {{{easy_install}}}'ed (provided that {{{OpenSSL}}} is installed), then this will reduce the need for #282 (more detailed and targeted docs about installing from source).

[zooko (zooko-tahoe-trac)]

I've requested that JP give me admin privs for the pyOpenSSL sf.net project so that I can upload a pyOpenSSL-0.6.tar.gz which works around this problem.

[zooko (zooko-tahoe-trac)]

Now, as far as we know the combination of Tahoe-1.1+Twisted-8.1+pyOpenSSL-0.7 doesn't lead to any bad behavior except for a vast number of unit tests failing doing to failure to close connections. One workaround, if we can't get an {{{easy_install}}}'able {{{pyOpenSSL-0.6.tar.gz}}} would be to code up an explicit "skip-test-like" behavior in our Makefile or perhaps in our test code to skip all these numerous failing tests if pyOpenSSL v0.7 is detected. I'm not sure exactly how that would be implemented. It also feels a little bit uncomfortable to deploy Tahoe-1.1+Twisted-8.1+pyOpenSSL-0.7 because I'm not entirely sure that the bug wouldn't lead to other problems for Tahoe users. (Doubtless pyOpenSSL-0.6 is more buggy that pyOpenSSL-0.7, but at least we have experience with it and there are no known anomalies which could be explained by bugs in pyOpenSSL-0.6.)

[zooko (zooko-tahoe-trac)]

Okay this whole issue is now {{{foolscap}}}'s problem -- Tahoe doesn't actually require {{{pyOpenSSL}}} at all. Tahoe requires foolscap-with-secure-connections, and (currently) foolscap-with-secure-connections requires pyOpenSSL. So I'm closing this ticket as "fixed" and further work will be done in #438 (get foolscap to declare its dependency on pyOpenSSL) and http://foolscap.lothar.com/trac/ticket/66 (install requires pyOpenSSL (for secure mode)).

[zooko (zooko-tahoe-trac)]

Oh wait, resolving this as "fixed" is a bit premature. Until there is a foolscap release that does this, and Tahoe specifies that it requires such a foolscap release, then this is still an open issue for Tahoe.

Also, there is a judgment call to be made as to what version of foolscap Tahoe should require.

[zooko (zooko-tahoe-trac)]

Okay, Brian is planning to release foolscap v0.2.8 which declares an "extra" dependency. If you specify that your project depends on foolscap "with the extra feature of secure connections", then foolscap will require pyOpenSSL.

He is not specifying that it requires a version of pyOpenSSL other than v0.7, which means that if Tahoe requires foolscap, and foolscap foolscap causes pyOpenSSL to be installed, and the version of pyOpenSSL that gets installed is version 0.7, then the Tahoe unit tests will all get ERRORs due to connections not being shut down properly.

So now I want to figure out how to make those ERRORs not happen when people install Tahoe and run {{{make test}}}.

[warner (warner-tahoe-trac)]

Apparently we don't fully understand the combination of versions that trigger this problem. On my debian/sid system, I see timeout/reactor-unclean failures of tahoe's test_system (and of several foolscap unit tests). This system has:
 * python-twisted-8.1.0-1
 * python-openssl-0.7-1
 * libssl0.9.8g-10.1

However, an Ubuntu/Hardy system we just set up does *not* fail tests, when using what we believe to be twisted-8.1.0, pyopenssl-0.7, and libssl0.9.8g-4ubuntusomething.

If this really only causes failures on sid (but hardy is ok), we're willing to push it out a release.
We still want to get it fixed, but it will probably require the pyopenssl maintainers to
fix [http://twistedmatrix.com/trac/ticket/3218 twisted#3218].

sid users are still advised to hold python-openssl at 0.6-5 .

[exarkun (exarkun-tahoe-trac)]

Are you sure it doesn't fail in that configuration? The problem includes a race condition dependent on timing of network operations and Python calls. It may just be that the race is biased towards going the wrong way frequently in one environment and the right way in the other.

[zooko (zooko-tahoe-trac)]

I just ran the Tahoe unit tests on Mac OS 10.4 on a PowerPC G4 867 MHz laptop, and this failure did not occur.

[zooko (zooko-tahoe-trac)]

This is important because currently there are two workarounds, each of which is unacceptable to one of the Tahoe developers:

workaround #1: leave "secure_connections" out of the requirements that Tahoe needs from foolscap, so that installations of Tahoe, which trigger installations of foolscap, do ''not'' trigger installations of pyOpenSSL. This works around the problem because if you happen to have pyOpenSSL already installed, but invisible to setuptools, and it is a version of pyOpenSSL that doesn't trigger this bug, then everything works including no bogus test failures. However, this is unacceptable to Zooko because if you do not already have the right version of pyOpenSSL installed then you will get a runtime exception and you'll have to manually installed pyOpenSSL. It is unacceptable to Zooko to require users to manually install pyOpenSSL.

workaround #2: leave "secure_connections" in the requirements. Then you won't have to manually install anything, and if you happen to get a combination of Twisted and pyOpenSSL which do not trigger this bug, you won't get any bogus test failures. However, this is unacceptable to Brian, because if you get a combination of Twisted and pyOpenSSL and your development platform which triggers this bug then you'll get bogus test failures. People seeing bogus test failures are unacceptable to Brian (and his development platform -- sid -- is the one which incurs this failure).

Here is a work-around which is kind of ugly but at least it isn't unacceptable: write a tearDown() method to reach inside the reactor and clean off outstanding delayed calls and open sockets. Also we would have to change the Tahoe unit tests to not wait for connection cleanup before passing the tests.

A *good* solution to this would, of course, be to fix this bug in Twisted and/or pyOpenSSL. Maybe we could contribute some time to that. I vaguely recall that there is now a unit test for the problem...

[zooko (zooko-tahoe-trac)]

Looks like the Twisted folks have been making progress on this issue:

http://twistedmatrix.com/trac/ticket/3218

[warner (warner-tahoe-trac)]

zooko says that the twisted folks say that this may only happen with the select reactor.. so another easy workaround is to use the pollreactor instead. I'll test this and report back.

[zooko (zooko-tahoe-trac)]

Okay this is fixed by [2803], [2804], [2805], [2806], [2807]. (Changes to the build system sometimes take multiple patches, because I use the buildbot to try out my changes on all of our platforms at once. If the buildbot "try this out but don't commit it to trunk" feature were working and I knew how to use it then I would do that instead.)

The fix is to set --reactor=poll on linux. (So this is in a sense a work-around instead of a fix, but on the other hand there's no reason for us to prefer the select reactor on linux, so this is fine.)

[zooko (zooko-tahoe-trac)]

Hooray -- the Twisted folks have fixed this issue:

http://twistedmatrix.com/trac/ticket/3218

[warner (warner-tahoe-trac)]

Excellent. Now we just need to wait for them to make a release, and add advice in the README to avoid the combination of Twisted in (8.0.1 .. 8.1.0) and pyOpenSSL-0.7 .

incidentally: I've tested twisted-8.0.1 and 8.1.0 (against pyopenssl-0.7) and saw test failures. I don't know about 8.0.0 . I see some failures against twisted-2.5.0, and different (non-ssl-related) ones against twisted-2.4.0 . So the versionspace to avoid might be Twisted in (8.0.0 .. 8.1.0) and pyopenssl-0.7 .. don't know yet.

Or, we could just be satisfied with always using the pollreactor. But if we ever want to simplify the Makefile and remove that platform-detection / reactor-choosing code, we could force users to go with a post-8.1.0 release of twisted instead.

[zooko (zooko-tahoe-trac)]

I'm not aware of any reason to prefer a select reactor over a poll reactor if there is a poll reactor on your platform, so I'm satisfied with our current reactor chooser.

I think I'll suggest to the Twisted folks (via their issue tracker) that Twisted could make poll reactor the default reactor on platforms that support it.

[zooko (zooko-tahoe-trac)]

http://twistedmatrix.com/trac/ticket/2234 # Select default reactor based on platform and available libraries

[launchpad (launchpad-tahoe-trac)]

Updating Launchpad bug reference

[Zooko Wilcox-O'Hearn (zooko)]

launchpad pushed a note onto my trac:

http://allmydata.org/trac/tahoe/ticket/402#comment:23

That's cool! But would you please push hyperlinks to the appropriate launchpad page e.g. https://bugs.launchpad.net/pyopenssl/+bug/236190 and not just the bug number?

Thanks!

[Diogo Matsubara (matsubara)]

Sounds like a good enhancement. Thanks for the report.

[Graham Binns (gmb)]

Actually, updating the Launchpad bug reference has added a link to the Launchpad bug in the Trac ticket summary at the top of the ticket page: http://allmydata.org/trac/tahoe/ticket/402

However, there's no reason why we couldn't update the Trac plugin to record the new bug reference as a hyperlink. That functionality is specifically with the Trac plugin, so I'm retargetting this bug appropriately.

[Tim Hatch (timhatch)]

Okay, I think the best way to do this is to change the comment from "Updating Launchpad bug reference" to "See [bug link] in Launchpad." Is that what you expected to happen?

IIRC the "* launchpad_bug set to 236190" portion is text only, autogenerated from higher up, so we can't put the link in there.

[Zooko Wilcox-O'Hearn (zooko)]

That sound fine.

[Tim Hatch (timhatch)]

I pushed this change a couple of minutes ago.