plainbox needs a way to share private scripts within one provider namespace

[Zygmunt Krynicki]

With the introduction of multiple providers scripts became a private resource owned by a particular provider. Currently many providers want to "share" their scripts and cross-call them freely. PlainBox needs to setup the PATH of executed scripts so that all providers sharing one namespace are added together.

[Sylvain Pineau (sylvain-pineau)]

So far only resources scripts need to be shared between V1 providers (with two other scripts, filter_template and run_template that could be part of the resource provider btw).

What if plainbox could extend the lookup path with the resource provider path for all providers (v1)?

I don't see other use cases yet and it would give us a working test runner.

[Zygmunt Krynicki (zyga)]

This is a hacky solution that does not give us either containment (from other providers) or a way to share scripts (between providers) and I'd like to avoid that.

The two choices I was pondering were:

1) Compute a unified PATH out of the providers sharing one namespace. The problem here is that PATH ordering gives implicit resolution to conflicts that might cause extremely confusing and hard to debug problems. It would make sharing an all-or-nothing question and would also probably force everyone join the flat checkbox namespace to use a single script, thus defeating the concept entirely.

2) Create a fixed per-namespace PATH lookup field that a provider actually installs scripts to. This would solve conflicts at packaging level, avoids the need to create very large PATHs but it is unclear how it would work with under-development scripts (what the path should be then?) and still doesn't solve the problem of providers being in another namespace that want to use a single script from some provider (nor does it solve the associated public vs private API problem).

I also considered a new helper tool, "plainbox-run-script provider script", that could be used in jobs, directly, to "import" scripts from any provider. This still does not solve the public vs private API problem directly unless we, again, have private and public bin folder inside providers.

While I'm currently preparing the groundwork for any solution I'm leaning towards:

1) Single path for intra-namespace script sharing, everything is namespace-private (no public API requirement), plainbox would compute a pathname where any provider using a particular namespace, should have its executables. Would work okay for packaged providers, would be a bit more tricky for non-packaged providers (those that live in arbitrary location). Not sure if this can be solved though.

2) plainbox-run-script as a solution for arbitrary sharing, would pull provider from a public bin directory of the provider. This ignores namespaces though.

[Zygmunt Krynicki (zyga)]

So I'd like to document the current, proposed solution. Effectively we're back to one flat namespace where each job sees scrips from every provider. This returns us to the status-quo and unbreaks plainbox. The way I've implemented the solution though, allows us to easily improve that to only share scripts from one provider namespace (needs discussion, trivial to implement) or even to offer per-job "import statement" that pulls a particular script from each provider (perhaps with a way to rename the name to resolve conflicts) but this needs additional syntax support that we don't want to do until checkbox-old goes away.

The way the solution works is by constructing a temporary directory for each execution of any job, with symlinks to all of the executables found in all of the providers. In theory this is a little slower but I haven't observed any measurable slowdown (a temporary directory, likely in tmpfs with a few symlinks, is cheap).

If you want to know the details have a look at https://code.launchpad.net/~zkrynicki/checkbox/path-bug/+merge/195157