funny permission bits in pyOpenSSL-0.6.tar.gz confuse tarfile.py

[Zooko Wilcox-O'Hearn]

As reported on the setuptools roundup -- http://bugs.python.org/setuptools/issue16 -- if one tries to install pyOpenSSL-0.6.tar.gz using Python's tarfile.py (instead of GNU tar) to unpack the tarball, then you get an exception that ends with this:

"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/tarfile.py",
line 1656, in _extract_member
    self.chmod(tarinfo, targetpath)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/tarfile.py",
line 1777, in chmod
    raise ExtractError("could not change mode of %s to mode %o (octal),
exception: %s" % (targetpath, tarinfo.mode, e))
tarfile.ExtractError: could not change mode of
/tmp/easy_install-4yCggq/pyOpenSSL-0.6 to mode 2755 (octal), exception: [Errno
1] Operation not permitted: '/tmp/easy_install-4yCggq/pyOpenSSL-0.6'

Inspection shows that the pyOpenSSL-0.6.tar.gz has the g+s bit set on all of its directories. Unsetting that bit and generating a new tarball makes this problem stop happening. Resetting that bit and regenerating a new tarball makes this problem come back.

Exarkun mentioned on IRC that he wasn't going to generate a new pyOpenSSL-0.6.tar.gz, so I made one for him, attached.

Also attached is the result of find . -type f | sha256sum, which does not differ between this tarball and the original one.

[Zooko Wilcox-O'Hearn (zooko)]

As reported on the setuptools roundup -- http://bugs.python.org/setuptools/issue16 -- if one tries to install pyOpenSSL-0.6.tar.gz using Python's tarfile.py (instead of GNU tar) to unpack the tarball, then you get an exception that ends with this:

"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/tarfile.py",
line 1656, in _extract_member
    self.chmod(tarinfo, targetpath)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/tarfile.py",
line 1777, in chmod
    raise ExtractError("could not change mode of %s to mode %o (octal),
exception: %s" % (targetpath, tarinfo.mode, e))
tarfile.ExtractError: could not change mode of
/tmp/easy_install-4yCggq/pyOpenSSL-0.6 to mode 2755 (octal), exception: [Errno
1] Operation not permitted: '/tmp/easy_install-4yCggq/pyOpenSSL-0.6'

Inspection shows that the pyOpenSSL-0.6.tar.gz has the g+s bit set on all of its directories. Unsetting that bit and generating a new tarball makes this problem stop happening. Resetting that bit and regenerating a new tarball makes this problem come back.

Exarkun mentioned on IRC that he wasn't going to generate a new pyOpenSSL-0.6.tar.gz, so I made one for him, attached.

Also attached is the result of find . -type f | sha256sum, which does not differ between this tarball and the original one.

[zooko (zooko-tahoe-trac)]

Ah, well I tried making Tahoe require pyOpenSSL >= 0.6, != 0.7, but the pyOpenSSL-0.6 tarball is not {{{easy_install}}}'able, as described here:

https://bugs.launchpad.net/pyopenssl/+bug/236190

If the pyOpenSSL maintainers fix the 0.6 tarball's permission bits as I submitted in that ticket, then this will start working.

[zooko (zooko-tahoe-trac)]

If this gets fixed so that pyOpenSSL can be {{{easy_install}}}'ed (provided that {{{OpenSSL}}} is installed), then this will reduce the need for #282 (more detailed and targeted docs about installing from source).

[Zooko Wilcox-O'Hearn (zooko)]

http://allmydata.org/trac/tahoe/ticket/402 (bug in Twisted, triggered by pyOpenSSL-0.7) and the related documentation issues for Tahoe (linked from issue #402 there) are blocked on this issue.

[Zooko Wilcox-O'Hearn (zooko)]

Here's an idea: give me admin privs on the pyOpenSSL sf.net project and I'll do this.

[zooko (zooko-tahoe-trac)]

I've requested that JP give me admin privs for the pyOpenSSL sf.net project so that I can upload a pyOpenSSL-0.6.tar.gz which works around this problem.

[zooko (zooko-tahoe-trac)]

Now, as far as we know the combination of Tahoe-1.1+Twisted-8.1+pyOpenSSL-0.7 doesn't lead to any bad behavior except for a vast number of unit tests failing doing to failure to close connections. One workaround, if we can't get an {{{easy_install}}}'able {{{pyOpenSSL-0.6.tar.gz}}} would be to code up an explicit "skip-test-like" behavior in our Makefile or perhaps in our test code to skip all these numerous failing tests if pyOpenSSL v0.7 is detected. I'm not sure exactly how that would be implemented. It also feels a little bit uncomfortable to deploy Tahoe-1.1+Twisted-8.1+pyOpenSSL-0.7 because I'm not entirely sure that the bug wouldn't lead to other problems for Tahoe users. (Doubtless pyOpenSSL-0.6 is more buggy that pyOpenSSL-0.7, but at least we have experience with it and there are no known anomalies which could be explained by bugs in pyOpenSSL-0.6.)

[zooko (zooko-tahoe-trac)]

Okay this whole issue is now {{{foolscap}}}'s problem -- Tahoe doesn't actually require {{{pyOpenSSL}}} at all. Tahoe requires foolscap-with-secure-connections, and (currently) foolscap-with-secure-connections requires pyOpenSSL. So I'm closing this ticket as "fixed" and further work will be done in #438 (get foolscap to declare its dependency on pyOpenSSL) and http://foolscap.lothar.com/trac/ticket/66 (install requires pyOpenSSL (for secure mode)).

[zooko (zooko-tahoe-trac)]

Oh wait, resolving this as "fixed" is a bit premature. Until there is a foolscap release that does this, and Tahoe specifies that it requires such a foolscap release, then this is still an open issue for Tahoe.

Also, there is a judgment call to be made as to what version of foolscap Tahoe should require.

[Zooko Wilcox-O'Hearn (zooko)]

Fixed by uploading a pyOpenSSL-0.6.tar.gz without the group sticky bits on the directories.

[Zooko Wilcox-O'Hearn (zooko)]

For reference the original tarball had sha256sum

5f8bee99cc78f1e6c01832582ed1ea5cf344d629d55100d6a0a430a1545dc0a1

and this tarball is now stored on sf.net as "pyOpenSSL-0.6-original.tar.gz".

The new tarball has sha256sum

58ee9dd10fbd97d65347626f60a38aafdcb05d19ad90c883c4284266ba7ec959

[Zooko Wilcox-O'Hearn (zooko)]

For reference the original tarball had md5sum

6200b71d3eb294a312d52c4825fc71c5

and the new one has md5sum

af67fda6d2125cae9c56ee50430a40c7

[Jean-Paul Calderone (exarkun)]

Some of the sf mirrors have not yet updated.

[zooko (zooko-tahoe-trac)]

Okay, Brian is planning to release foolscap v0.2.8 which declares an "extra" dependency. If you specify that your project depends on foolscap "with the extra feature of secure connections", then foolscap will require pyOpenSSL.

He is not specifying that it requires a version of pyOpenSSL other than v0.7, which means that if Tahoe requires foolscap, and foolscap foolscap causes pyOpenSSL to be installed, and the version of pyOpenSSL that gets installed is version 0.7, then the Tahoe unit tests will all get ERRORs due to connections not being shut down properly.

So now I want to figure out how to make those ERRORs not happen when people install Tahoe and run {{{make test}}}.

[warner (warner-tahoe-trac)]

Apparently we don't fully understand the combination of versions that trigger this problem. On my debian/sid system, I see timeout/reactor-unclean failures of tahoe's test_system (and of several foolscap unit tests). This system has:
 * python-twisted-8.1.0-1
 * python-openssl-0.7-1
 * libssl0.9.8g-10.1

However, an Ubuntu/Hardy system we just set up does *not* fail tests, when using what we believe to be twisted-8.1.0, pyopenssl-0.7, and libssl0.9.8g-4ubuntusomething.

If this really only causes failures on sid (but hardy is ok), we're willing to push it out a release.
We still want to get it fixed, but it will probably require the pyopenssl maintainers to
fix [http://twistedmatrix.com/trac/ticket/3218 twisted#3218].

sid users are still advised to hold python-openssl at 0.6-5 .

[exarkun (exarkun-tahoe-trac)]

Are you sure it doesn't fail in that configuration? The problem includes a race condition dependent on timing of network operations and Python calls. It may just be that the race is biased towards going the wrong way frequently in one environment and the right way in the other.

[Zooko Wilcox-O'Hearn (zooko)]

I think all the sf mirrors have updated to have the new one now.

Feel free to give me admin privs of whatever kind are needed to close tickets.

[zooko (zooko-tahoe-trac)]

I just ran the Tahoe unit tests on Mac OS 10.4 on a PowerPC G4 867 MHz laptop, and this failure did not occur.

[zooko (zooko-tahoe-trac)]

This is important because currently there are two workarounds, each of which is unacceptable to one of the Tahoe developers:

workaround #1: leave "secure_connections" out of the requirements that Tahoe needs from foolscap, so that installations of Tahoe, which trigger installations of foolscap, do ''not'' trigger installations of pyOpenSSL. This works around the problem because if you happen to have pyOpenSSL already installed, but invisible to setuptools, and it is a version of pyOpenSSL that doesn't trigger this bug, then everything works including no bogus test failures. However, this is unacceptable to Zooko because if you do not already have the right version of pyOpenSSL installed then you will get a runtime exception and you'll have to manually installed pyOpenSSL. It is unacceptable to Zooko to require users to manually install pyOpenSSL.

workaround #2: leave "secure_connections" in the requirements. Then you won't have to manually install anything, and if you happen to get a combination of Twisted and pyOpenSSL which do not trigger this bug, you won't get any bogus test failures. However, this is unacceptable to Brian, because if you get a combination of Twisted and pyOpenSSL and your development platform which triggers this bug then you'll get bogus test failures. People seeing bogus test failures are unacceptable to Brian (and his development platform -- sid -- is the one which incurs this failure).

Here is a work-around which is kind of ugly but at least it isn't unacceptable: write a tearDown() method to reach inside the reactor and clean off outstanding delayed calls and open sockets. Also we would have to change the Tahoe unit tests to not wait for connection cleanup before passing the tests.

A *good* solution to this would, of course, be to fix this bug in Twisted and/or pyOpenSSL. Maybe we could contribute some time to that. I vaguely recall that there is now a unit test for the problem...

[zooko (zooko-tahoe-trac)]

Looks like the Twisted folks have been making progress on this issue:

http://twistedmatrix.com/trac/ticket/3218

[warner (warner-tahoe-trac)]

zooko says that the twisted folks say that this may only happen with the select reactor.. so another easy workaround is to use the pollreactor instead. I'll test this and report back.

[zooko (zooko-tahoe-trac)]

Okay this is fixed by [2803], [2804], [2805], [2806], [2807]. (Changes to the build system sometimes take multiple patches, because I use the buildbot to try out my changes on all of our platforms at once. If the buildbot "try this out but don't commit it to trunk" feature were working and I knew how to use it then I would do that instead.)

The fix is to set --reactor=poll on linux. (So this is in a sense a work-around instead of a fix, but on the other hand there's no reason for us to prefer the select reactor on linux, so this is fine.)

[zooko (zooko-tahoe-trac)]

Hooray -- the Twisted folks have fixed this issue:

http://twistedmatrix.com/trac/ticket/3218

[warner (warner-tahoe-trac)]

Excellent. Now we just need to wait for them to make a release, and add advice in the README to avoid the combination of Twisted in (8.0.1 .. 8.1.0) and pyOpenSSL-0.7 .

incidentally: I've tested twisted-8.0.1 and 8.1.0 (against pyopenssl-0.7) and saw test failures. I don't know about 8.0.0 . I see some failures against twisted-2.5.0, and different (non-ssl-related) ones against twisted-2.4.0 . So the versionspace to avoid might be Twisted in (8.0.0 .. 8.1.0) and pyopenssl-0.7 .. don't know yet.

Or, we could just be satisfied with always using the pollreactor. But if we ever want to simplify the Makefile and remove that platform-detection / reactor-choosing code, we could force users to go with a post-8.1.0 release of twisted instead.

[zooko (zooko-tahoe-trac)]

I'm not aware of any reason to prefer a select reactor over a poll reactor if there is a poll reactor on your platform, so I'm satisfied with our current reactor chooser.

I think I'll suggest to the Twisted folks (via their issue tracker) that Twisted could make poll reactor the default reactor on platforms that support it.

[zooko (zooko-tahoe-trac)]

http://twistedmatrix.com/trac/ticket/2234 # Select default reactor based on platform and available libraries