too high version dependency on pycryptopp

[Julian Taylor]

Binary package hint: tahoe-lafs

when then executing tahoe in a clean natty chroot with python-mock (bug 782455) and a fixed python-foolscap (bug 782414 see below) installed:

$ apt-get install python-mock
$ tahoe
Traceback (most recent call last):
  File "/usr/bin/tahoe", line 9, in <module>
    load_entry_point('allmydata-tahoe==1.8.2', 'console_scripts', 'tahoe')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 305, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2244, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1954, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/pymodules/python2.7/allmydata/__init__.py", line 369, in <module>
    check_all_requirements()
  File "/usr/lib/pymodules/python2.7/allmydata/__init__.py", line 367, in check_all_requirements
    raise PackagingError(get_error_string(errors, debug=True))
allmydata.PackagingError:
PackagingError: We require pycryptopp >= 0.5.20, but could only find version 0.5.17.
The version we found is from '/usr/lib/pymodules/python2.7'.
To resolve this problem, uninstall that version, either using your
operating system's package manager or by moving aside the directory.

the fixed foolscap it is available here from svn:
svn://svn.debian.org/python-modules/packages/foolscap/trunk/
revision 17008

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: tahoe-lafs 1.8.2-0ubuntu1
ProcVersionSignature: Ubuntu 2.6.38-9.43-generic 2.6.38.4
Uname: Linux 2.6.38-9-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Sat May 14 01:52:13 2011
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Alpha amd64 (20110329.1)
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_US:en
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: tahoe-lafs
UpgradeStatus: No upgrade log present (probably fresh install)

[Zooko Wilcox-O'Hearn (zooko)]

Here's where the upstream Tahoe-LAFS specifies what version of pycryptopp it needs:

http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/_auto_deps.py?annotate=blame&rev=4976#L64

It says:

 64 if platform.machine().lower() in ['i386', 'x86_64', 'amd64', 'x86', '']:
 65 # pycryptopp v0.5.20 fixes bugs in SHA-256 and AES on x86 or amd64
 66 # (from Crypto++ revisions 470, 471, 480, 492). The '' is there
 67 # in case platform.machine is broken and this is actually an x86
 68 # or amd64 machine.
 69 install_requires.append("pycryptopp >= 0.5.20")
 70 else:
 71 # pycryptopp v0.5.13 had a new bundled version of Crypto++
 72 # (v5.6.0) and a new bundled version of setuptools (although that
 73 # shouldn't make any difference to users of pycryptopp).
 74 install_requires.append("pycryptopp >= 0.5.14")

So it sounds like it is potentially important to upgrade pycryptopp in Ubuntu from v0.5.17 to v0.5.20. Doing so fixes a bug which causes incorrecvt SHA-256 and AES computations on x86 and amd64 architectures.

[Julian Taylor (jtaylor)]

as far as I can tell the pycryptopp package uses the system crypto++ package which is not so hopelessly outdated as pycryptopp.
So it should not be affected by this issue and the dependency is to high.

@ Zooko
as you are the maintainer of pycryptopp please update it!
it is 12 releases behind upstream...
If you do not have the time, orphan it so someone else can take over.

[Zooko Wilcox-O'Hearn (zooko)]

Agreed, since Ubuntu's distribution of pycryptopp uses the system (Ubuntu distribution of) Crypto++ then it is not vulnerable to that issue.

Now we either have to patch Tahoe-LAFS so that its src/allmydata/_auto_deps.py file does not require such a new pycryptopp or upgrade pycryptopp. I will work on the latter, but I haven't done this before and I'm not sure how long it will take me.

[Daira Hopwood (daira)]

Paul Hummer wrote about this bug on the tahoe-dev mailing list, and I replied:

On 07/06/11 01:17, Paul Hummer wrote:
> > Hi folks-
> >
> > The context of this email is here:
> >
> > https://code.launchpad.net/~jtaylor/ubuntu/oneiric/tahoe-lafs/tahoe-lafs-fix/+merge/63631
> >
> >
> > Some of the autodeps sort of stuff I purposefully patch out specifically
> > because I make that sort of stuff static in the packaging metadata, and the
> > setup usually ends up complicating things (and going out to the internet,
> > which the Ubuntu buildds explicitly aren't allowed to do).
> >
> > I'm curious about stepping back on pycryptopp. I'm curious why we can't
> > just update the pycryptopp instead of downgrading the pycryptopp library.
There is a good reason why Tahoe depends on at least pycryptopp 0.5.20
(on x86 and x86-64). It's documented in src/allmydata/_auto_deps.py:

# pycryptopp v0.5.20 fixes bugs in SHA-256 and AES on x86 or amd64
# (from Crypto++ revisions 470, 471, 480, 492).

Now, in
<https://bugs.launchpad.net/ubuntu/+source/tahoe-lafs/+bug/782461/comments/4>
Zooko said:
# ... since Ubuntu's distribution of pycryptopp uses the system (Ubuntu
# distribution of) Crypto++ then it is not vulnerable to that issue.

But hang on, how do we know that Tahoe will use the Ubuntu distribution
of pycryptopp? It will use the pycryptopp that is first on the sys.path
in the Tahoe process. The sys.path depends in an impossibly difficult-
to-predict way on the history of installations of Python packages (by
setuptools, other Python-specific installers, and OS package managers),
including Python packages that might seem completely unrelated. When Tahoe
starts, it will sanity-check that the version of the pycryptopp it imported
is >= 0.5.20. However, there is no direct check on which version of Crypto++
the imported pycryptopp is using.

I don't see what the difficulty is in upgrading the Ubuntu pycryptopp
package to 0.5.20. That is the simplest way for Tahoe to be as sure as
it can be (against accidental misconfigurations) that it is getting a
pycryptopp that will use a Crypto++ that fixes these SHA-256 and AES bugs.

[Julian Taylor (jtaylor)]

pycryptopp 0.5.29-1 was uploaded which fixes this issue

[Daira Hopwood (daira)]

If I understand correctly, the fact that pycryptopp 0.5.29-1 has been uploaded means that the change in http://bazaar.launchpad.net/~jtaylor/ubuntu/oneiric/tahoe-lafs/tahoe-lafs-fix/revision/9 is no longer required or desired.

[Julian Taylor (jtaylor)]

yes, the commit is obsolete for oneiric, I have deleted the branch and made a new one for the mock issue (bug 769935)