Buildout -- Software for automating application assembly

a per-framework shared eggs directory

Reported by Martijn Faassen on 2009-02-18
2
Affects Status Importance Assigned to Milestone
Buildout
Undecided
Unassigned
grok
Undecided
Unassigned

Bug Description

grokproject, a tool to create projects using the Grok web framework, currently inspects .buildout and if there is no default.cfg, creates a line which says:

[buildout]
eggs-directory = /path/to/home/.buildout/eggs

This so that users are set up with a shared eggs directory which greatly cuts down on download times when people create new projects or add a dependency to an existing project that they had used elsewhere. We don't do this in grokproject-generated buildouts directly as that would set hard paths in buildout.cfg, and that would make them not interchangeable. It also introduces a single convention on where to look for shared eggs.

For Grok, which restricts which versions of eggs to download from pypi using the 'versions' construct in buildout, this approach works pretty well. Some other projects however rely on a custom index to restrict which versions of packages are installed. If a shared eggs directory is in use, the wrong eggs will be pulled in from this shared directory and the installation procedure fails. This is a problem whenever someone adds a line like this to their .buildout/default.cfg, not just in the case of Grok.

Globally sharing eggs can be useful. It also breaks the isolation assumption that some buildouts have. It would be nice if we could have another level of isolation: isolation per "framework" (or perhaps per index). eggs-directory (and perhaps the download cache as well) could then be different for each framework. Grok could then happily share eggs in ~buildout/.eggs-grok, and nobody else would be bothered by it.

One solution to do this with existing buildout technology would be to create a default.cfg which has something
like this:

[grok-dir]
path = /home/faassen/.buildout/eggs-grok

and then in a specific grokproject-generated buildout.cfg we'd do something like this:

[buildout]
eggs-directory = ${grok-dir:path}

(not sure whether the grok-dir part needs to be on the parts line here)

This would fail however if there were *no* default.cfg that defined grok-dir. We cannot move the [grok-dir] section into the project's buildout either as this would create a hardcoded path inside buildout.cfg.

We could resolve this by adding something like 'shared-eggs-directory-base' and 'shared-eggs-directory'.
In default.cfg, we'd have:

[buildout]
shared-eggs-directory-base = /path/to/.buildout/eggs

In our own buildout.cfg we'd have:

[buildout]
...
shared-eggs-directory = grok

this would cause the system to look for /path/to/.buildout/eggs-grok (shared-eggs-directory-base + '-' + shared-eggs-directory) and assign this to eggs-directory. If no shared-eggs-directory-base can be found it'll ignore shared-eggs-directory and just use the default for eggs-directory (eggs local to the buildout), so that the buildout will still work without a default.cfg in place. If an explicit eggs-directory is found this will always trump shared-eggs-directory.

This way, Grok will put all its eggs in a shared eggs directory shared by all other grok installations on the system, but other buildouts won't be bothered by it, as there'd be no reason to place an egg-directory in buildout.cfg anymore.

Chris McDonough (chrism-plope) wrote :

This problem would also be made slightly less painful if the eggs-directory contained and managed separate subdirectories full of eggs, one per index used. Then even if there was a shared eggs directory, it would be partitioned into index-specific buckets and only the bucket related to the index in the zc.recipe.egg section could be consulted.

I still can't fathom why grokproject itself needs to add anything to some global location as opposed to some sufficiently advanced user doing the same thing by hand, but that's a different argument.

On Wed, Feb 18, 2009 at 3:17 PM, Chris McDonough <email address hidden> wrote:
> I still can't fathom why grokproject itself needs to add anything to
> some global location as opposed to some sufficiently advanced user doing
> the same thing by hand, but that's a different argument.

I'm trying to solve the problem that we can all agree on first. :)

Download full text (3.9 KiB)

You make some good points.

I'd be inclined to add a feature that causes the location of the eggs
directory to depend on the index and find-links option -- possibly by
using a hash of these.

Jim

On Feb 18, 2009, at 8:54 AM, Martijn Faassen wrote:

> Public bug reported:
>
> grokproject, a tool to create projects using the Grok web framework,
> currently inspects .buildout and if there is no default.cfg, creates a
> line which says:
>
> [buildout]
> eggs-directory = /path/to/home/.buildout/eggs
>
> This so that users are set up with a shared eggs directory which
> greatly
> cuts down on download times when people create new projects or add a
> dependency to an existing project that they had used elsewhere. We
> don't
> do this in grokproject-generated buildouts directly as that would set
> hard paths in buildout.cfg, and that would make them not
> interchangeable. It also introduces a single convention on where to
> look
> for shared eggs.
>
> For Grok, which restricts which versions of eggs to download from pypi
> using the 'versions' construct in buildout, this approach works pretty
> well. Some other projects however rely on a custom index to restrict
> which versions of packages are installed. If a shared eggs directory
> is
> in use, the wrong eggs will be pulled in from this shared directory
> and
> the installation procedure fails. This is a problem whenever someone
> adds a line like this to their .buildout/default.cfg, not just in the
> case of Grok.
>
> Globally sharing eggs can be useful. It also breaks the isolation
> assumption that some buildouts have. It would be nice if we could have
> another level of isolation: isolation per "framework" (or perhaps per
> index). eggs-directory (and perhaps the download cache as well) could
> then be different for each framework. Grok could then happily share
> eggs
> in ~buildout/.eggs-grok, and nobody else would be bothered by it.
>
> One solution to do this with existing buildout technology would be
> to create a default.cfg which has something
> like this:
>
> [grok-dir]
> path = /home/faassen/.buildout/eggs-grok
>
> and then in a specific grokproject-generated buildout.cfg we'd do
> something like this:
>
> [buildout]
> eggs-directory = ${grok-dir:path}
>
> (not sure whether the grok-dir part needs to be on the parts line
> here)
>
> This would fail however if there were *no* default.cfg that defined
> grok-dir. We cannot move the [grok-dir] section into the project's
> buildout either as this would create a hardcoded path inside
> buildout.cfg.
>
> We could resolve this by adding something like 'shared-eggs-
> directory-base' and 'shared-eggs-directory'.
> In default.cfg, we'd have:
>
> [buildout]
> shared-eggs-directory-base = /path/to/.buildout/eggs
>
> In our own buildout.cfg we'd have:
>
> [buildout]
> ...
> shared-eggs-directory = grok
>
> this would cause the system to look for /path/to/.buildout/eggs-grok
> (shared-eggs-directory-base + '-' + shared-eggs-directory) and assign
> this to eggs-directory. If no shared-eggs-directory-base can be found
> it'll ignore shared-eggs-directory and just use the default for eggs-
> directory (eggs local to the buildout)...

Read more...

On Feb 18, 2009, at 9:17 AM, Chris McDonough wrote:

> This problem would also be made slightly less painful if the eggs-
> directory contained and managed separate subdirectories full of eggs,
> one per index used. Then even if there was a shared eggs directory,
> it
> would be partitioned into index-specific buckets and only the bucket
> related to the index in the zc.recipe.egg section could be consulted.

I like this. You'd want to have a subdirectory for each find-link
too. You'd then search and store in different egg directories bases
on which were used.

Jim

--
Jim Fulton
Zope Corporation

On Wed, Feb 18, 2009 at 3:51 PM, Jim Fulton <email address hidden> wrote:
> I'd be inclined to add a feature that causes the location of the eggs
> directory to depend on the index and find-links option -- possibly by
> using a hash of these.

I think that would work for Grok. Grok relies on the standard pypi
index, and would therefore share eggs with any other application that
uses the pypi index. If another application uses another index, pypi
index eggs would never interfere with these, so Chris should be all
right.

Changed in grok:
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers