tahoe-lafs can't start because foolscap packaging metadata doesn't declare the fact that it supports secure_connections

[Zooko Wilcox-O'Hearn]

foolscap has accidentally dropped a build dependency on python-setuptools so the requires.txt is not installed.
This leads to software breaking which use this setuptools feature to check if a package provides the [secure_connections] feature and a missing dependency on python-twisted.

This is fixed by lp:~jtaylor/ubuntu/natty/foolscap/foolscap-fix-782414

TESTCASE:
> python -c 'import pkg_resources; print pkg_resources.require("foolscap[secure_connections]")'
What should happen:
[foolscap 0.6.1 (/usr/lib/python2.7/dist-packages), pyOpenSSL 0.10 (/usr/lib/python2.7/dist-packages), Twisted 10.2.0 (/usr/lib/python2.7/dist-packages)]
What happens:
$ python -c 'import pkg_resources; print pkg_resources.require("foolscap[secure_connections]")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 654, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 557, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2180, in requires
    "%s has no such extra feature %r" % (self, ext)
pkg_resources.UnknownExtra: foolscap 0.6.1 has no such extra feature 'secure_connections'

the dependency tahoe-lafs is broken by this
(it is also broken by bug 782461 and bug 769935 unrelated to foolscap)

original description:

The foolscap setup.py file (http://foolscap.lothar.com/trac/browser/setup.py?annotate=blame&rev=7c7b1185b68076cc6176ea41b5fc677445e1e45b#L75 ) says that if a package that requires foolscap requires an extra feature named "secure_connections" then foolscap requires pyOpenSSL.

Other Python packages can then specify that they depend on "foolscap with its extra feature of 'secure_connections'". This is spelled "foolscap[secure_connections]". Tahoe-LAFS does this:

http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/_auto_deps.py?annotate=blame&rev=4976#L21

Currently the packaging of foolscap in Natty claims that it doesn't know about the "secure_connections" feature:

$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.require('foolscap')
[foolscap 0.6.1 (/usr/lib/python2.7/dist-packages)]
>>> pkg_resources.require('foolscap[secure_connections]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 654, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 557, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2180, in requires
    "%s has no such extra feature %r" % (self, ext)
pkg_resources.UnknownExtra: foolscap 0.6.1 has no such extra feature 'secure_connections'

This means that, for example, the tahoe-lafs executable that is packaged in Natty cannot start:

$ tahoe --version
Traceback (most recent call last):
  File "/usr/bin/tahoe", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2671, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 654, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 557, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2180, in requires
    "%s has no such extra feature %r" % (self, ext)
pkg_resources.UnknownExtra: foolscap 0.6.1 has no such extra feature 'secure_connections'
$ type tahoe
tahoe is hashed (/usr/bin/tahoe)
$ dpkg --search /usr/bin/tahoe
tahoe-lafs: /usr/bin/tahoe
$ apt-cache policy tahoe-lafs
tahoe-lafs:
  Installed: 1.8.2-0ubuntu1
  Candidate: 1.8.2-0ubuntu1
  Version table:
 *** 1.8.2-0ubuntu1 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ natty/universe i386 Packages
        100 /var/lib/dpkg/status

By inspecting the foolscap setup.py, you can see that the stanza that declares this extra feature is inside a test for the existence of setuptools: http://foolscap.lothar.com/trac/browser/setup.py?annotate=blame&rev=7c7b1185b68076cc6176ea41b5fc677445e1e45b#L65

So the easiest way to fix this is probably to configure the Ubuntu packaging of foolscap to indicate that setuptools is a build-time requirement of foolscap, and then rebuild. This should produce the metadata file (named /usr/share/pyshared/foolscap-0.6.1.egg-info/requires.txt ) that indicates that this distribution of foolscap knows how to provide secure_connections, as long as it is in turn provided with pyOpenSSL.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: python-foolscap 0.6.1-1
ProcVersionSignature: Ubuntu 2.6.38-8.42-virtual 2.6.38.2
Uname: Linux 2.6.38-8-virtual i686
Architecture: i386
Date: Fri May 13 20:23:17 2011
Ec2AMI: ami-ccf405a5
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: t1.micro
Ec2Kernel: aki-407d9529
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: foolscap
UpgradeStatus: Upgraded to natty on 2011-05-13 (0 days ago)

[Zooko Wilcox-O'Hearn (zooko)]

In this comment I suggest an alternate fix.

In my opinion this is a flaw in the packaging of foolscap for Ubuntu, and will cause this trouble for any code which tries to use foolscap and to specify that it requires foolscap's "secure_connections" feature. In my opinion the easiest way to fix this is to add the Python packaging metadata to foolscap, as described above.

However, if for some reason that doesn't work, an alternate approach would be to change Tahoe-LAFS as packaged in Ubuntu, remove the part of its code (in http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/_auto_deps.py?annotate=blame&rev=4976#L21 ) that says that it depends on 'foolscap["secure_connections"]' and replace it with code that says it depends on just 'foolscap'. At the same time, add code that says that it depends on pyOpenSSL. Here is a patch to Tahoe-LAFS that will do that.

diff -rN -u old-pristine/src/allmydata/_auto_deps.py new-pristine/src/allmydata/_auto_deps.py
--- old-pristine/src/allmydata/_auto_deps.py 2011-05-13 16:16:49.000000000 -0600
+++ new-pristine/src/allmydata/_auto_deps.py 2011-05-13 16:16:50.000000000 -0600
@@ -23,7 +23,21 @@
     # of size N.
     # foolscap < 0.6 is incompatible with Twisted 10.2.0.
     # foolscap 0.6.1 quiets a DeprecationWarning.
- "foolscap[secure_connections] >= 0.6.1",
+ "foolscap >= 0.6.1",
+
+ # foolscap requires pyOpenSSL in order to provide secure
+ # connections, but unfortunately we can't rely on distributions of
+ # foolscap to declare that they provide the "secure_connections"
+ # feature even when they do (see
+ # http://foolscap.lothar.com/trac/ticket/174 ).
+ # Therefore, we specify here that Tahoe-LAFS depends on pyOpenSSL.
+ # Note that this is a fiction--Tahoe-LAFS itself does not require
+ # pyOpenSSL for anything. If a future version of foolscap provided
+ # secure connections without using pyOpenSSL, or required a newer
+ # version of pyOpenSSL, or required a different library than
+ # pyOpenSSL, then Tahoe-LAFS users who are provided with that
+ # newer version of foolscap would need to edit this declaration.
+ "pyOpenSSL",

     "Nevow >= 0.6.0",

[Zooko Wilcox-O'Hearn (zooko)]

This is a regression from Maverick to Natty. Below is a transcript of the equivalent process on Maverick, including tahoe-lafs working on Maverick. So you could look at the difference in the procedure for building foolscap on Maverick vs. building foolscap on Natty. I assume that the salient difference is that the new build is done without setuptools being present.

zooko@ip-10-117-66-104:~$ python
Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resourcers
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pkg_resourcers
>>> import pkg_resources
>>> pkg_resources.require("foolscap")
[foolscap 0.5.1 (/usr/lib/pymodules/python2.6), Twisted 10.1.0 (/usr/lib/python2.6/dist-packages)]
>>> pkg_resources.require("foolscap[secure_connections]")
[foolscap 0.5.1 (/usr/lib/pymodules/python2.6), pyOpenSSL 0.10 (/usr/lib/pymodules/python2.6), Twisted 10.1.0 (/usr/lib/python2.6/dist-packages)]
>>>
zooko@ip-10-117-66-104:~$ dpkg --listfiles python-foolscap | grep requires.txt
/usr/share/pyshared/foolscap-0.5.1.egg-info/requires.txt
zooko@ip-10-117-66-104:~$ cat /usr/share/pyshared/foolscap-0.5.1.egg-info/requires.txt
twisted >= 2.4.0

[secure_connections]
zooko@ip-10-117-66-104:~$ tahoe --version
allmydata-tahoe: 1.7.1, foolscap: 0.5.1, pycryptopp: 0.5.17, zfec: 1.4.5, Twisted: 10.1.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.6.6, platform: Linux-Ubuntu_10.10-i686-32bit_ELF, sqlite: 3.7.2, simplejson: 2.1.1, pyOpenSSL: 0.10, setuptools: 0.6, pysqlite: 2.4.1

[Julian Taylor (jtaylor)]

the dependency on python-setuptools somehow got lost on the transition to dh_python2
I'll fix it as soon as possible.

[Julian Taylor (jtaylor)]

fix commited in debian svn http://svn.debian.org/viewsvn/python-modules?view=rev&revision=17008

[Daira Hopwood (daira)]

OK, the foolscap packaging error is fixed in the package for foolscap 0.6.1-2. But the Tahoe-LAFS package doesn't require foolscap 0.6.1-2, does it? So a user of Tahoe-LAFS can still hit the error if that package dependency isn't updated, IIUC.

[Julian Taylor (jtaylor)]

Tahoe is only has a runtime dependency on foolscap.
So when the foolscap package is updated this tahoe problem will be solved without further action.

note that tahoe has further problems: see bug 782379 and bug 782461

[Launchpad Janitor (janitor)]

This bug was fixed in the package foolscap - 0.6.1-3

---------------
foolscap (0.6.1-3) unstable; urgency=low

  * Standards-version bumped to 3.9.2 (no changes needed)
  * wrap-and-sort debian/
  * remove ${python:Breaks}
  * add python-setuptools to b-d again
    was dropped in 0.6.1-1 by accident (LP: #782414)
 -- Ubuntu Archive Auto-Sync <email address hidden> Thu, 19 May 2011 06:24:29 +0000

[Micah Gersten (micahg)]

I got this error when building locally and installing this on natty with the new build-dep:
sudo dpkg -i ../python-foolscap_0.6.1-1ubuntu0.1_all.deb
(Reading database ... 189330 files and directories currently installed.)
Preparing to replace python-foolscap 0.6.1-1 (using .../python-foolscap_0.6.1-1ubuntu0.1_all.deb) ...
Unpacking replacement python-foolscap ...
dpkg: error processing ../python-foolscap_0.6.1-1ubuntu0.1_all.deb (--install):
 unable to open '/usr/share/pyshared/foolscap-0.6.1.egg-info/PKG-INFO.dpkg-new': No such file or directory
Processing triggers for man-db ...
Errors were encountered while processing:
 ../python-foolscap_0.6.1-1ubuntu0.1_all.deb

Did I do something wrong here?

[Martin Pitt (pitti)]

Julian provided a fix for the egg issue in his MP, setting back to triaged.

[Martin Pitt (pitti)]

Uploaded to natty-proposed queue.

[Martin Pitt (pitti)]

Accepted foolscap into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Zooko Wilcox-O'Hearn (zooko)]

I installed python-foolscap_0.6.1-1ubuntu0.1_all.deb from natty-proposed. It installed cleanly and Tahoe-LAFS passes its unit tests using this version of foolscap.

[Julian Taylor (jtaylor)]

there are still some issues with this update when twisted meta package is not installed as the requires.txt still lists it as a dependency, also a runtime dependency on setuptools is required as the zope egg requires it but does not depend on it.

without the twisted meta package:
# python -c 'import pkg_resources; print pkg_resources.require("foolscap[secure_connections]")'
Traceback (most recent call last):
...
pkg_resources.DistributionNotFound: twisted>=2.4.0

with twisted and without setuptools:
# python -c 'import pkg_resources; print pkg_resources.require("foolscap[secure_connections]")'
Traceback (most recent call last):
...
pkg_resources.DistributionNotFound: distribute

these issues are not really important as tahoe, the only user of this feature, depends on both missing packages.

[Martin Pitt (pitti)]

Thanks for testing! Julian, I understand that the issues you brought up are not regressions from the -proposed package, so the proposed update is an improvement overall?

[Launchpad Janitor (janitor)]

This bug was fixed in the package foolscap - 0.6.1-1ubuntu0.1

---------------
foolscap (0.6.1-1ubuntu0.1) natty-proposed; urgency=low

  * add build dependency python-setup to create proper egg (LP: #782414)
    - remove old egg symlinks in a preinst to ensure a correct upgrade
    - pydist-override twisted -> python-twisted-core
 -- Julian Taylor <email address hidden> Mon, 30 May 2011 14:00:49 +0200

[Julian Taylor (jtaylor)]

yes the issues are not regressions. The feature does not work at all with the version in natty even if you have those packages installed.

the missing setuptools dependency issue really needs fixing in python-defaults.
It also affects all zope packages (bug 563060)

It is fixed in 2.7.2-2 in debian experimental:
- remove setuptools from requires.txt (it is replaced with
       python-pkg-resources Debian dependency)