Can not use buildout to configure BLOB storage

[Steve Schmechel]

Adding blob-dir = ${buildout:parts-directory}/data/blobs to the [data] section of buildout.cfg does not have any effect.
The functionality was probably lost when switching to paster-support in grokproject.
Some possible solutions were discussed at: http://www.nabble.com/Can-not-use-buildout-to-configure-BLOB-storage-td24472937.html

The right default configuration for grokproject needs to be determined and the relevant documentation needs to be updated.

[Reinout van Rees (reinout)]

Needed: verification if it has been fixed (in one of the indicated ways) already in grokproject trunk.

And probably some documentation update.

[Steve Schmechel (steveschmechel)]

I am not sure if you were looking for a response to your comment on this bug, or who you would like to answer the question you asked?

I looked at the grokproject trunk code and nothing has changed since the 1.0a4 released version that would address this issue. Neither the paster template or the buildout templates were changed.

I think Uli wanted to take time to consider the right way to fix it for the majority of grokproject users. I would defer to whatever he decides.

Should you just assign the issue to him, or am I missing a step in the workflow? I would be happy to validate the fix from the trunk once someone checks something in.

I don't have SVN access or a wide enough experience with how people use buildout/paste/grokproject or I would be happy to help in other ways.

Thanks for looking at this and let me know if there is some way I can assist.

Steve

--- On Fri, 8/14/09, Reinout van Rees <email address hidden> wrote:

> From: Reinout van Rees <email address hidden>
> Subject: [Bug 407458] Re: Can not use buildout to configure BLOB storage
> To: <email address hidden>
> Date: Friday, August 14, 2009, 6:08 AM
> Needed: verification if it has been
> fixed (in one of the indicated ways)
> already in grokproject trunk.
>
> And probably some documentation update.
>
> ** Also affects: grok/1.0
>    Importance: Undecided
>        Status: New
>
> --
> Can not use buildout to configure BLOB storage
> https://bugs.launchpad.net/bugs/407458
> You received this bug notification because you are a direct
> subscriber
> of the bug.
>

[Uli Fouquet (uli-gnufix)]

Thanks Steve and Reinout for bringing this up!

Honestly, I am not sure, what the best way is.

The main problem is, that with the switch to paster, we need the (generated) zope.conf in a different location (same as deploy.ini) and we cannot use zc.recipe.zope3recipe anymore (please correct me if I'm wrong!).

Then, we left the [data] and [log] section in buildout.cfg, merely to create the directories parts/data/ and parts/log when buildout runs. This is hardly proper use of zc.recipe.filestorage but, well, okay. Maybe we can live with it.

I see the following options:

1) In zope.conf.in we replace the upper part (before <eventlog>) with something like:

     ${data:zconfig}

That would mean, that setting "blob-dir" in buildout.cfg and rerunning buildout would work again. For several reasons we could not easily drop the zope.conf.in template completely. We would also lose the ZEO config example which currently is part of zope.conf.in.

2) In zope.conf.in we could add a (commented) blob-storage definition as proposed by Steve. In buildout.cfg we could at the same time leave a comment where the blob-storage can be enabled (in zope.conf.in).

As you both are apparently the guys with a need for the blob-dir-option (or some replacement), you have to decide what way would suit your needs best. Maybe you also have an example configuration you would like to see in standard grokprojects? Otherwise I'd put something in like you, Steve, proposed on grok-dev.

Please note, that I hadn't heard of that option before your posts and therefore didn't mind when the whole configuration of grokproject changed (thanks for the hint!).

What do you think?

[Reinout van Rees (reinout)]

I've got some company work on monday that relates to this (zopeconf product configuration that also needs to end up in here somewhere), so I'll at least take a look at in on monday.

[Uli Fouquet (uli-gnufix)]

Just added a commented blob-storage def in grokproject paster template.

What's still missing are hints for blobs-and-ZEO configurations, but this might also give an overload of example configs. Maybe we should setup a page describing all the ZODB options somwhere on grok.zope.org? Or is there already such a page? I currently dig component.xml in ZODB repository to lookup all that stuff.

[Steve Schmechel (steveschmechel)]

Is there a reason to not just make this the default configuration?

I created a new, no-site-packages virtualenv and new grokproject, changed the zope.conf.in to use the BLOB storage configuration, and it runs with no problems and the tests all pass.

You do get an extra line when you start the project. (Maybe this could be suppressed by specifying the layout.):

  INFO [ZODB.blob] (26577) Blob directory `= /opt/grok/atest/blobtest/parts/data/Data.fs/blobs` has layout marker set. Selected `bushy` layout.

I'm not saying that this is the way to go - I'm just curious what the down-side would be to having it able to "just work" when you add the necessary packages to setup.py and rerun buildout.

It is unfortunate that the BLOB storage configuration is entangled with the ZODB file storage configuration in the first place.

It might actually be nicer if all the database and storage related configuration occurred in the paster templates as it is likely that one would have different paths for development and deployment anyway.

That being said, I am OK with the solution as it stands right now.

I will try grokproject and buildout from the trunk code later today. I did my tests with 1.0a4 and did have some problems with the changes made in changeset 9926 (the double $$ in front of variables). Buildout passed
one of the $'s into the output files, so the paths were broken like:

  path $/opt/grok/atest/blobtest/parts/data/Data.fs

Maybe this is fixed somewhere else in the trunk.

Steve

--- On Sun, 8/16/09, Uli Fouquet <email address hidden> wrote:

> From: Uli Fouquet <email address hidden>
> Subject: [Bug 407458] Re: Can not use buildout to configure BLOB storage
> To: <email address hidden>
> Date: Sunday, August 16, 2009, 7:29 AM
> Just added a commented blob-storage
> def in grokproject paster template.
>
> What's still missing are hints for blobs-and-ZEO
> configurations, but
> this might also give an overload of example configs. Maybe
> we should
> setup a page describing all the ZODB options somwhere on
> grok.zope.org?
> Or is there already such a page? I currently  dig
> component.xml in ZODB
> repository to lookup all that stuff.
>
> --
> Can not use buildout to configure BLOB storage
> https://bugs.launchpad.net/bugs/407458
> You received this bug notification because you are a direct
> subscriber
> of the bug.
>

[Uli Fouquet (uli-gnufix)]

> Is there a reason to not just make this the default configuration?

Only the don't-change-too-much-at-once policy.

> I created a new, no-site-packages virtualenv and new grokproject,
> changed the zope.conf.in to use the BLOB storage configuration, and it
> runs with no problems and the tests all pass.

Fine, thanks for looking into this :-)

> You do get an extra line when you start the project. (Maybe this could
> be suppressed by specifying the layout.):
>
> INFO [ZODB.blob] (26577) Blob directory `=
> /opt/grok/atest/blobtest/parts/data/Data.fs/blobs` has layout marker
> set. Selected `bushy` layout.

Is there a way to suppress this message?

Beside this, this looks erraneous to me: the path should be data/blobs,
not data/Data.fs/blobs, right? Maybe we have to switch to something like
${buildout:parts-directory}/blobs.

We _could_ also set the blobs-dir to
${buildout:parts-directory}/data/blobs, but we cannot rely, that this
directory (data/) really exists. This depends on the name of the [data]
section in buildout.cfg.

> I'm not saying that this is the way to go - I'm just curious what the
> down-side would be to having it able to "just work" when you add the
> necessary packages to setup.py and rerun buildout.

Sure, we can make this setting the (enabled) default. As you certainly
used blobs more often you can better tell what downsides could result
from that.

> It is unfortunate that the BLOB storage configuration is entangled with
> the ZODB file storage configuration in the first place.
>
> It might actually be nicer if all the database and storage related
> configuration occurred in the paster templates as it is likely that one
> would have different paths for development and deployment anyway.

That's currently not possible as (AFAIK) there is no way to forward
settings from paster to zope and vice versa. And we're yet bound to
ZConfig configurations (this XML-like format) for basic Zope
configuration. I think someone was working on support for a real WSGI
pipeline in Zope and I started to build a ZConfig parser that allows
normal Python OptionParser syntax like in paster. But both pieces are
quite complex. To have this would really be nice and main configuration
parts could go into deploy.ini then.

> That being said, I am OK with the solution as it stands right now.
>
> I will try grokproject and buildout from the trunk code later today. I
> did my tests with 1.0a4 and did have some problems with the changes
> made in changeset 9926 (the double $$ in front of variables).
> Buildout passed one of the $'s into the output files, so the paths
> were broken like:
>
> path $/opt/grok/atest/blobtest/parts/data/Data.fs
>
> Maybe this is fixed somewhere else in the trunk.

Hm, this should not happen. The grokproject templates contain double
dollars to quote the dollar character. When grokproject runs, it turns
$${buildout:directory} into ${buildout:directory} which should be
written into the etc/ template files.

Then, when you run buildout of the generated project,
${buildout:directory} turns into a real path.

I normally check this by running 'buildout' in the checked-out
grokproject branch, then run bin/grokproject...

[Steve Schmechel (steveschmechel)]

Seems to work using grokproject-1.0a5dev from trunk with the new configuration.

(Don't worry about my ramblings about the $'s. I see now that the first one is removed when grokproject runs and creates the *.conf.in templates. The second is removed when buildout creates the *.conf files.)

--- On Sun, 8/16/09, Steve Schmechel <email address hidden> wrote:

> From: Steve Schmechel <email address hidden>
> Subject: Re: [Bug 407458] Re: Can not use buildout to configure BLOB storage
> To: <email address hidden>
> Date: Sunday, August 16, 2009, 8:57 AM
> Is there a reason to not just make
> this the default configuration?
>
> I created a new, no-site-packages virtualenv and new
> grokproject,
> changed the zope.conf.in to use the BLOB storage
> configuration, and it
> runs with no problems and the tests all pass.
>
> You do get an extra line when you start the project.
> (Maybe this could
> be suppressed by specifying the layout.):
>
>   INFO [ZODB.blob] (26577) Blob directory `=
> /opt/grok/atest/blobtest/parts/data/Data.fs/blobs` has
> layout marker
> set. Selected `bushy` layout.
>
> I'm not saying that this is the way to go - I'm just
> curious what the
> down-side would be to having it able to "just work" when
> you add the
> necessary packages to setup.py and rerun buildout.
>
> It is unfortunate that the BLOB storage configuration is
> entangled with
> the ZODB file storage configuration in the first place.
>
> It might actually be nicer if all the database and storage
> related
> configuration occurred in the paster templates as it is
> likely that one
> would have different paths for development and deployment
> anyway.
>
> That being said, I am OK with the solution as it stands
> right now.
>
> I will try grokproject and buildout from the trunk code
> later today.  I did my tests with 1.0a4 and did have
> some problems with the changes made in changeset 9926 (the
> double $$ in front of variables).  Buildout passed
> one of the $'s into the output files, so the paths were
> broken like:
>
>   path $/opt/grok/atest/blobtest/parts/data/Data.fs
>
> Maybe this is fixed somewhere else in the trunk.
>
> Steve
>
> --- On Sun, 8/16/09, Uli Fouquet <email address hidden>
> wrote:
>
> > From: Uli Fouquet <email address hidden>
> > Subject: [Bug 407458] Re: Can not use buildout to
> configure BLOB storage
> > To: <email address hidden>
> > Date: Sunday, August 16, 2009, 7:29 AM
> > Just added a commented blob-storage
> > def in grokproject paster template.
> >
> > What's still missing are hints for blobs-and-ZEO
> > configurations, but
> > this might also give an overload of example configs.
> Maybe
> > we should
> > setup a page describing all the ZODB options somwhere
> on
> > grok.zope.org?
> > Or is there already such a page? I currently  dig
> > component.xml in ZODB
> > repository to lookup all that stuff.
> >
> > --
> > Can not use buildout to configure BLOB storage
> > https://bugs.launchpad.net/bugs/407458
> > You received this bug notification because you are a
> direct
> > subscriber
> > of the bug.
> >
>
> --
> Can not use buildout to configure BLOB storage
> https://bugs.launchpad.net/bugs/407458
> You received ...

[Reinout van Rees (reinout)]

I thought there was a pretty strong convention to put data that has to stay stored in ${buildout:directory}/var/something and regeneratable stuff in parts/. I might be mistaken :-) Just thought to mention it now that there's some discussion on perhaps changing the blob directory path.

[Uli Fouquet (uli-gnufix)]

We never followed that in Grok, did we? So it looks a bit like a zope2-ism to me. On the other hand, the convention to put data and similar in var/ makes sense to me. Let's discuss this on the list, okay?

[Reinout van Rees (reinout)]

Yeah, still used to plone's buildout layout :-) There's all sorts of current examples where the data is put into parts/, so it is probably OK once you get used to it.

I set up a sample project this morning to look at the differences with our current zopectl setup. It all seems to work fine, apart from missing "zopectl run xyz.py" functionality, but that's not grok's problem.

One addition that seems handy: an 'extra =' option in the zope_conf part that's copied verbatim to the end of the generated zope.conf. For specifying custom entries (like <productconf> items). Don't know what the best way to do that is. Just a comment? Already add it?

Using the filestorage recipe just for letting it create a log directory... that's dirty. I know no alternative right now, though.

[Uli Fouquet (uli-gnufix)]

This seems to work now for the paster template in grokproject.

Thanks Reinout!