build Nevow without importing Nevow

Bug #812537 reported by Zooko Wilcox-O'Hearn
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Tahoe-LAFS
Unknown
Unknown
nevow
New
Undecided
Zooko Wilcox-O'Hearn

Bug Description

This looks like a good patch:

http://bazaar.launchpad.net/~divmod-dev/divmod.org/build-nevow-without-importing-2699/revision/2601

But why do the execfile hack only when setuptools is present? Why not do that same thing in both cases?

Revision history for this message
Jean-Paul Calderone (exarkun) wrote :

Why not? What misbehavior do you observe?

Changed in nevow:
assignee: nobody → Zooko O'Whielacronx (zooko)
Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Only because it would be simpler if it did the same thing in both cases instead of doing different things. Why not?

I don't observe any misbehavior, but I haven't tried. I remember that py2exe used to have problems packaging Nevow because the version number that Nevow would generate during the packaging process would be determined by importing nevow, which would get a different version of nevow (from the sys.modules of the py2exe Python process) than the one that was being built and packaged. Inspecting this patch (but not trying it) I wonder if this code would have the same problem if used under py2exe in the absence of setuptools.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Ivan Kozik asked me on Stack Overflow to explain why it could matter, for Python packages in general, whether they import themselves when building. I managed to explain the ways that it can be a problem better this time around than before:

http://stackoverflow.com/questions/458550/standard-way-to-embed-version-into-python-package/7071358#7071358

Below I'll copy and edit what I wrote there:

If your setup.py is being executed in a freshly created Python process, and your current working directory is the directory that contains the version of your project that it is building, and any dependencies that your project imports when it is imported are already installed, then it will work.

This is the common way that programmers run setup.py, so they often think it is the only way. But there are other ways that setup.py gets evaluated! Tools like py2exe run a single Python process and then load and execute the setup.py scripts from one project after another (while packing them all together). Suppose that some of the Python code that ran before your setup.py script imported a different version of your module. In that case, "import yourniftymodule" in your setup.py would find that *different* version in your sys.modules. (This has happened to me with Nevow.)

Another problem is if the user is using a tool like setuptools or pip to install your package. Such a tool needs to do two things for packages that declare dependencies: 1. build this package, 2. install (first building, if necessary) its dependencies. Now, what order should the tool do this in?

It can't (in the setuptools/pip/virtualenv system of today) even know what the deps are until it evaluates your setup.py. Also, if it tried to do full depth-first and finish installing all deps before it builds this package, it would get stuck if there were circular deps. But if it tries to build this package before installing the dependencies, then if you import your package from your setup.py, it will not necessarily be able to import its deps, or the right versions of its deps.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

A user reported a failure to build/install Tahoe-LAFS, and the output they provided looks like it might be this issue. Attaching their output below.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

I just learned that I can solve my problem by switching from Nevow to twisted.web.templates!

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1963

Perhaps you could help others by posting an announcement suggesting twisted.web.templates.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Here is a recipe for this sort of thing (getting a version number into your Python package's build system at build-time without importing your package):

http://stackoverflow.com/questions/458550/standard-way-to-embed-version-into-python-package/7071358#7071358

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Note: this issue is still causing problems for users of Tahoe-LAFS. The Tahoe-LAFS developers are making some progress on replacing our use of Nevow with twisted.web.template (https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1963), but it will probably be at least a year or two before that work gets into the hands of the majority of our current users, and they would benefit sooner from a new stable release of Nevow which fixes this issue.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

I forget what is blocking progress on this issue. One thing we would need is someone who has write-permissions sufficient to patch Nevow's canonical trunk. Who can do that? I know exarkun can do that, but if I recall correctly he said that he wasn't not willing to do so for this issue, since this issue doesn't affect his use of Nevow. Who else has that write-permission?

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :
Revision history for this message
Daira Hopwood (daira) wrote :

I don't understand why no-one is prepared to make a release of Nevow *with the fix that has already been committed to trunk over 4 years ago*. Am I missing something?

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :
Revision history for this message
Daira Hopwood (daira) wrote :

I was mistaken, and zooko is correct. That patch is not on trunk.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Looking at https://github.com/twisted/nevow/blob/4643891d8ec4a2aab3c8515c4d82ba8924952e4e/setup.py (which I believe is the version of setup.py in Nevow 0.11.1), it no longer imports nevow (or anything else other than versioneer or setuptools) in setup.py. Yay! Problem fixed.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

This ticket has been moved to here: https://github.com/twisted/nevow/issues/7

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.