grokproject should pin versions

[Martijn Faassen]

With Grok, we go through a lot of trouble to make sure versions are pinned down, to avoid new releases breaking Grok itself. We don't do this however for grokproject, which is on the critical path to installing Grok itself...

I think we should pin down the (non-test) dependencies in grokproject to actual versions. This can be easily done for PasteScript. I'm a bit more concerned about pinning down zc.buildout - I wonder what the interactions are to installing and using buildout here. This needs to be investigated.

In addition, I think we should also pin down the buildout eggs that grokproject generated buildout.cfg relies on.

So:

* pin down dependencies of grokproject

* pin down recipe dependencies of buildout.cfg generated by grokproject

[Maurits van Rees (maurits-vanrees)]

I pinned the grok version in the generated setup.py; committed in r87875.

As for the zc.buildout dependency of grokproject itself: in my own (Plone) buildouts I never pin zc.buildout or setuptools in the version.cfg as a bootstrap will always get the newest versions of those two and will then give a conflict error when running bin/buildout after that. So I am inclined to not pin those in the setup.py of grokproject or a generated project either.

As for PasteScript: when I change the dependency from "PasteScript>=1.6" to "PasteScript==1.6.3" I get this error when running bin/buildout in my grokproject checkout:

===================================
While:
  Installing grokproject.
Error: There is a version conflict.
We already have: PasteScript 1.6.2
but grokproject 0.8dev requires 'PasteScript==1.6.3'.
===================================

Ah, but of course when running "bin/buildout -nv" it gets the new requirement. So that should be fine; committed in r87877.

[Martijn Faassen (faassen)]

Hi there,

On Tue, Jul 1, 2008 at 12:06 PM, maurits <email address hidden> wrote:
> I pinned the grok version in the generated setup.py; committed in
> r87875.

I don't understand the need for this. We already pin Grok itself in
the versions file, after all. Could this be reverted? We should have
only one place where this is pinned down.

This issue is only about pinning things for grokproject and buildout.

[Martijn Faassen (faassen)]

On Tue, Jul 1, 2008 at 12:06 PM, maurits <email address hidden> wrote:
> As for the zc.buildout dependency of grokproject itself: in my own
> (Plone) buildouts I never pin zc.buildout or setuptools in the
> version.cfg as a bootstrap will always get the newest versions of those
> two and will then give a conflict error when running bin/buildout after
> that. So I am inclined to not pin those in the setup.py of grokproject
> or a generated project either.

Yes, we've been doing this for grok as well, but I actually consider
this to be the wrong behavior. It's quite possible for a broken
buildout version or recipe version to be released. For buildout, I
know a broken buildout version was released not long ago, for
instance.

I realize that this doesn't have an easy solution, but the goals are:

* release grokproject and be sure that when it is installed, it will
only use well-known fixed versions of all its dependencies

* be able to install a grokproject generated grok project and be sure
it will only use a well-known fixed version of buildout and its
recipes.

The goal is repeatability. We don't want a situation where we have a
working grokproject, someone releases a botched recipe, paste or
buildout, and suddenly the process fails.

(I agree versions in a grokproject generated buildout should not be
pinned down in the setup.py, by the way. We never should generate
anything with grokproject that pins things down in setup.py.)

P.S. This is not a critical issue, as we work in the less reliable way
already. I just would like to think about a solution.

[Maurits van Rees (maurits-vanrees)]

I have removed the dependency on zc.buildout as we already have code in place (in grokproject/utils.py:run_buildout) that downloads zc.buildout if it is not available.

We could make a small change there to require a known-good version:

- ws.require('zc.buildout')
+ ws.require('zc.buildout==1.0.6')

We could make the same change in grokproject/template/bootstrap.py; we already have some customizations there to use the grok-eggs-*.tgz.

The same might have to be done for setuptools.

On the grok-dev list Philipp makes a case for not pinning PasteScript. If you do not agree, discussion can continue there, where I think it is handier.

For recipes it might also be dangerous to pin them; for example infrae.subversion has had some (development) releases that were removed from the cheeseshop later. Me grok smash package thieves. ;-)

We use 6 recipes in the generated buildout.cfg now. We could let grokproject append pinned versions of them at the end of versions.cfg.

Hm, maybe we should pin the python version. ;-)

[Martijn Faassen (faassen)]

It's far more dangerous not to pin things than it is to pin them in
the face of package thieves. If a package is gone, the error will be
pretty clear. If a package is subtly broken after a random amount of
time, we'll have far more trouble.

[Martijn Faassen (faassen)]

Is anything in particular blocking this?

I think the principle is that when you install grok, it should get as many pinned versions as possible. Preferably those versions aren't pinned in setup.py as then it cannot be overridden, but perhaps for grokproject itself this cannot be helped.

Modifying bootstrap.py scares me a bit as we already were bitten by some weird error messages that resulted from the modification.

[Maurits van Rees (maurits-vanrees)]

I prefer not to touch the setup.py either, but perhaps it indeed cannot be helped -- we can rapidly upload a new grokproject version if a version in setup.py starts giving problems. I have at least changed the versions.cfg templates on trunk:

* Pinned all package versions in the generated versions.cfg to the
  latest released ones. Especially pinned z3c.recipe.eggbasket to
  the most recent one, 0.4.1.

[Maurits van Rees (maurits-vanrees)]

As far as I can see we have pinned down everything in the correct way now. At least a grok project generated with trunk now reports no picked versions when using this one-liner: 'bin/buildout -Novvvvv |sed -ne 's/^Picked: //p' | sort | uniq'