buildout does not honour platform for python executable

[gweis]

In my buildout I build a local python distribution which will be used by almost all eggs installed by buildout.
For this buildout runs with the system wide python installation and sets the python/executable option to the locally built python if necessary.

The problem is, that the platform configuration for the local python differs from the system python. This is probably an OSX only issue where you can build universal or for instance i386 only binaries.

In my case I tried to build lxml with zc.recipe.egg, which builds fine as lxml-2.2.2-py2.4-macosx-10.6-i386.egg but buildout then looks for lxml-2.2.2-py2.4-macosx-10.6-universal.egg which cam't be found and the whole buildout fails.

The attached patch determines the platform settings, for a specific python executable. In my case this patch works well, (Sorry no tests written) but I am sure it could be done in a more elegant/performant way.

regards

[gweis (gerhard-weis)]

rediffed faulty patch

[nutjob (kelly-seankelly)]

This is biting me, too.

1. The buildout is bootstrapped with the system python (macosx-10.3-fat).

2. A part in my buildout uses hexagonit.recipe.cmmi to build python 2.6 (macosx-10.3-i386).

3. Another part in my buildout uses zc.recipe.egg:custom to build PIL, with python set to the built python, generating PIL-1.1.6-py2.6-macosx-10.3-i386.egg.

The build succeeds but finds no distributions because the Environment uses the bootstrap python's platform (-fat) instead of the specified python's platform (-i386).

However, rather than adding a "platform" option, easy_install.py could just invoke self._executable with the import of setuputils.pkg_resources.get_supported_platform, invoke it, and snarf its output.

[gweis (gerhard-weis)]

That's basically what the patch does. On initialisation of the Install class in easy_install.py it first ensures that setuptools is available for self._executable and then it invokes self._executable and catches the output of setuputils.pkg_resources.get_supported_platform.

However, the determined platform name needs to be passed into every pgk_resources.Environment instance so that buildout can build the correct list of available packages.

So "platform" is not an additional option for buildout, but an attribute of the Install class and it's value is determined during runtime.

(Btw. I am building PIL too with hexagonit.recipe.cmmi and with tho attached patch it works flawless.)

[nutjob (kelly-seankelly)]

Yes, quite.

And that means it also affects not just zc.recipe.egg:custom, but also the other zc.recipe.egg recipes.

And indeed, I just hit that this morning. My buildout builds a specialized python, some custom eggs, and a final egg whose script I want with the other eggs baked in. But the final egg (using zc.recipe.egg) ignores the built custom eggs in develop-eggs since they're all "-i386" eggs and not "-fat" eggs, despite a setting of "python" to the custom python.

[gweis (gerhard-weis)]

nutjob: Did you try the attached patch? It hooks into the zc.buildout easy_install method, and should fix the mentioned issue for any eggs installed via easy_install.

[Guy Rozendorn (guy-rozendorn)]

the patch doesn't work with 1.5.2.

one not-so-safe way is to pass platform=None to Environment.
another way is to get pkg_resources.get_supported_platforms() with sys._executable, but, you'll need to install setuptools for this other python first.

why wasn't the patch added to the sourcetree?

[Guy Rozendorn (guy-rozendorn)]

As I noted earlier, there are two solutions for this issue.

The solution suggested in this patch, is to pass platform=None to to pkg_resources.Environment.
This solution will work unless the python that ran bootsrap.py and the python-executable mentioned in the buildout configuration share the same version but don't share the share platform.

In my use-case, I'm running bootstrap.py with python2.6 i386, and pointing to python2.7 x86_64 in the configuration, so this patch is good enough for me.

The better solution is to get the platform of the destination python, just like you get its version (see _get_version).
However, getting the platform is not simple as getting the version.
The default value for the platform kwarg is pkg_resources.get_supported_platform(), which is part of setuptools.
That means we need to launch the destination python with that code printed to stdout, read it and return it. but we cannot assume that setuptools is available for that python. so we need to do this own our own, and with our that can run under the "-c" command-line argument. I have a patch for this solution as well, but I didn't test if it runs on Windows, yet.

[Guy Rozendorn (guy-rozendorn)]

did some testing, couldn't install coverage and pywin32,
when doing Popen in get_plaftorm and get_version, use shell=False on Windows

updated patch attached

[Guy Rozendorn (guy-rozendorn)]

did some testing, couldn't install coverage and pywin32,
when doing Popen in get_plaftorm and get_version, use shell=False on Windows

updated patch attached

[Guy Rozendorn (guy-rozendorn)]

did some testing, couldn't install coverage and pywin32,
when doing Popen in get_plaftorm and get_version, use shell=False on Windows

updated patch attached

[Guy Rozendorn (guy-rozendorn)]

Hi,

A month has passed since I uploaded the patch.
Can someone please review it and merge it to the source?

[Jim Fulton (jim-zope)]

Normally, my first comment would be that the patch needs tests.
Otherwise, it could be a long time before it gets merged.

I really think we need to take a step back and think more about this
use case.

In general, I'd like to phase out support for using multiple Python
interpeters in the same buildout. It makes the code and tests really
complicated. I really want the code to be much simpler.

I think this is a different case though. Here, you only want to use
one Python, but you want to build it during the bootstrapping
process. Is that a correct assessment? If so, we should brainstorm
this use case (or family of use cases) on the distutils-sig list.

[Guy Rozendorn (guy-rozendorn)]

I need to look at it again, but I think that this patch is covered by the current suite of tests (whether the buildout fails or not). If I'm wrong, I will add tests that cover this.

Let me describe the use-case in full.
We develop applications that are meant to run on various flavors of Windows, Linux, Solaris and other UNIXes. We do not rely on the system-level python, because there isn't such on all the platforms we support.
So we have a build system that compiles Python on our platforms in such a way that it makes it completely relocatable (with the shared libraries and all) on the file system for that particular OS.
We use buildout for doing this, but the build process itself is irrelevant to this issue (The code is on github: https://github.com/grzn/relocatable-python).

Now that we have this Python, we want our applications to use this one, and the way we currently do it is this:
* We use some version of python to run the bootstrap (we don't care which Python this is, because we will stop using it shortly
* We download the our pre-built Python interpreter
* From now on, we use the new interpreter.

All of this is done on our development hosts. In this way, each of our apps has its own isolated interpreter, which we can upgrade when necessary without risking compatibility with other apps.

For deployment, we pack everything into an archive (incl. the interpreter), and extract it at customer's site. Since the archive already contains our Python interpreter, we use the same one both for bootstrapping and other things.

What does this has to do with this issue? If the interpreter used for the bootstrapping is 32bit, and our pre-built Python is 64bit, and there's a need to install a platform-dependent or python-version-specific pre-built module (e.g. pywin32), buildout will install an incompatible version to our pre-built interpreter.

We've been working on this use-case for quite some time, I would be happy to further discuss this.

[Guy Rozendorn (guy-rozendorn)]

There's a simple test case for this, but then again, it is not "unit test" in a sense because it is platform-specific:
* you need to run buildout and install a platform-dependent (c-extension)
* use another interpreter inside the buildout that is in different architecture than the one being used by the bootstrapper.

That's for testing this specific use-case.
I don't understand how design of the tests in this project. could you please elaborate on how to execute them?

Since I don't know how to run the tests I cannot confirm, but I'm pretty sure that the existing tests cover the changes in this patch (the change effects every buildout run).

and not that is an acceptable argument, but, I'm using this build internally on linux/windows/osx/sunos and its solid.
I would be happy to work on a complete test case, but again, I'm having trouble understanding how the tests are written and how to execute them.