provide canonical way to install templates and configuration files into $SNAP_* dirs

[Christian Ehrhardt ]

We have had a ML discussion about it, but I think it is worth to track it properly:

I was asking about the preferred way to handle conffiles and such.
There seems to be some new functionality incoming, quoting from the ML:

"> The only snap-centric artifact about it I found was [1]. But that feels
> broken/outdated as there is no "snappy" command anymore (and snap has no
> "config" subcommand).

Something similar will come back. From what I've heard there will be a
apply-config hook which you can implement in your snap and a user can
call from the outside with a simple 'snap set
snap.name.confkey=confvalue' or similar."

But since it seems that for the former there is already work in progress (although I don't know the bug), there was something related brought up that is important just as well - and this is what this bug is about.
Quoting Nick Moffit from the Mail:

"Is there a good facility for install-time copying of template configs or
other writeable-from-initial-state data into $SNAP_USER_DATA?

The way snaps are invoked do a great job of hijacking the $*PATH
variables used to find executables and shared objects, but often we need
to hack in a lot of shuffling around for /usr/share and /etc and /var/
to aim them at snappy's special directories."

Currently I see myself and others implement something like "first call handlers" that copy content packaged in the snap (no reason to be there) to their intended place (user exposed configuration e.g. in $SNAP_DATA).

I personally would think about something like:
support for that in the copy plugin like supporting:
parts:
 defaultconf:
   plugin: copy
   files:
       etc/example.conf: $SNAP_USER_DATA/etc/foo.conf

Surely there is some more magic glue involved as at "snap building time" the path of $SNAP_USER_DATA and such isn't fix IIRC (could change per setup and in future).
Therefore I'll file it for snapcraft & snappy right at the beginning.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nick Moffitt (nick-moffitt)]

This is the general pattern I settled on for wrapper binaries (after giving up on Makefiles because I'm just odd that way):

    #!/bin/sh

    test -e $SNAP_USER_DATA/foo.conf || cp $SNAP/etc/foo.conf $SNAP_USER_DATA/
    exec $SNAP/bin/foo

[Christian Ehrhardt  (paelzer)]

Other examples of hacks currently used:

https://github.com/ogra1/packageproxy/blob/master/run-approx
https://github.com/ogra1/upnp-server

Thanks to ogra for providing these, although for completeness it is only fait to quote his opinion (against this):
"no, SNAP_USER_DATA is a runtime, not a build time thing ... you have to
do it from your wrapper script that you most likely have to use anyway
for other stuff to set up the environment for your binary (also there
are not really any install time hooks, we are not debs with maintainer
scripts (luckily))"

[Christian Ehrhardt  (paelzer)]

I understand the opinion as quoted before, but IMHO packaging will require something like that the more things that will adopt snaps.
And then I'd really prefer a architectured unified approach than to replicate this or similar hacks over and over again.

[Christian Ehrhardt  (paelzer)]

Note - depending on the preference of the snappy/snapcraft architecture Team a solution for this bug could be implemented via 1611638.

[Evan (ev)]

I think hack is an unnecessarily strong term.

In the Cassandra snap, the config is copied into $SNAP_DATA at runtime (if not already present) [1]. Helper programs are provided for getting, setting, and environment variable lookup (for templating) [2].

Having native support for config will bring consistency with other snapped applications, but otherwise this works quite well.

1: https://github.com/evandandrea/cassandra-snap/blob/master/wrapper-cassandra
2: https://lists.ubuntu.com/archives/snapcraft/2016-July/000362.html

[Christian Ehrhardt  (paelzer)]

Sorry, I didn't mean to use harsh words on young and innocent programs :-)

Our current workaround (better word?) isn't so different then:
https://gitlab.com/paelzer/ntpsec/blob/snapify-ntpsec/etc/snap/ntpd.sh

[Nick Moffitt (nick-moffitt)]

Whether it's hackish or not, it is a step we're taking to work around an obstacle snappy throws up but provides no mechanisms to deal with.

I think we're all taking roughly the same approaches, and that's encouraging that people are unlikely to get this wrong if we can't fix it. But it does make for a rather un-DRY experience that is a bit jarring.

I would support adding some sort of mechanism to make these sorts of operations first-class and consistent.

[Kyle Fazzari (kyrofa)]

I think an install hook is a more appropriate way to handle this type of initial runtime setup.

[Christian Ehrhardt  (paelzer)]

I've read here (bug 1611638) that the configure hooks will be called on "initial install, upgrade, and when `snap set` is called".

If that is true and shall be the solution to this bug it should be reflected in https://github.com/snapcore/snapd/blob/master/docs/hooks.md what it does on install.

Also it is not written there if those hooks will have all the usual per-snap ENVs like $SNAP_USER_DATA and such - will it?

[Kyle Fazzari (kyrofa)]

> [...] it should be reflected in https://github.com/snapcore/snapd/blob/master/docs/hooks.md what it does on install.

Agreed, please log a bug for that. The documentation there is a little out of date.

I mentioned this on the other bug, but since you asked here as well I'll say for completeness: hooks have the same environment and confinement as apps within the snap. They have the same environment variables defined ($SNAP, $SNAP_VERSION, etc.), and can use plugs the same way as apps.

[Christian Ehrhardt  (paelzer)]

On Fri, Oct 28, 2016 at 5:59 PM, Kyle Fazzari <email address hidden> wrote:

> > [...] it should be reflected in
> https://github.com/snapcore/snapd/blob/master/docs/hooks.md what it does
> on install.
>
> Agreed, please log a bug for that. The documentation there is a little
> out of date.
>

Will do so, thanks for the "pre-ack" to give me the confidence to do so.
Opened bug 1637934 for that.

> I mentioned this on the other bug, but since you asked here as well I'll
> say for completeness: hooks have the same environment and confinement as
> apps within the snap. They have the same environment variables defined
> ($SNAP, $SNAP_VERSION, etc.), and can use plugs the same way as apps.
>

I was unsure on the scope where the question would fit most.
Thanks a lot!

[Command Prompt, Inc. (commandpromptinc)]

Besides just putting a file into a place, there's a problem of filesystem paths inside configuration file.

It's quite typical for almost any configuration file to have a setting similar to the following:

logfile=/var/log/app/logfile.log

which may eventually become:

logfile=/home/joe/snap/app/x1/var/log/app/logfile.log

or simply

logfile=/home/joe/snap/app/x1/logfile.log

etc.

This $HOME directory will change depending on which system user runs an app. So, there's also a need to somehow process a configuration file and set all such filesystem paths correctly before application is run.

Ivan

[Kyle Fazzari (kyrofa)]

Ivan, I don't think it's possible to do such a thing in a generic manner, i.e. neither snapd nor snapcraft will be able to help you, since neither know what your intentions are regarding that path. Neither tool even knows how to parse your config file. That's why snapd defines the various environment variables that it defines-- so you can use them in such files.

[Command Prompt, Inc. (commandpromptinc)]

Just to elaborate further, I think it is very convenient to be able to have multiple copies of the same application running on the same host.

Whether people are doing this or not is unknown to me, but it is obvious that with snap packages it is as easy as running an app as several different users. It is not so hard to imagine that some people will want to run apps as normal, unprivileged users as opposed to system-wide daemons run as root and thus configuration files will have to reference different home directories.

Sure, it can be done with some basic sed and using custom variables like for example %#FSPATH#% to ensure sed works only on something unique like that but since we're talking about snapcraft handling configuration files in a way that doesn't require a wrapper, I think there's no escaping from having snapcraft or some plug-in processing configuration files and setting correct filesystem paths.

Ivan

[Command Prompt, Inc. (commandpromptinc)]

Kyle, as you well know from your experience with redis not all programs can take advantage of that. I'm currently working on pgbouncer snap and like redis it doesn't have a concept of expandable variables in a configuration file. In fact, my guess is that most programs don't.

Ivan

[Kyle Fazzari (kyrofa)]

> [...] configuration files will have to reference different home directories.

Then they should be using $SNAP_USER_DATA or $SNAP_USER_COMMON. Those will be user-specific.

> Kyle, as you well know from your experience with redis not all programs can take advantage of that.

Are you referring to my blog post[1]? If so, then you also may have noticed that while redis doesn't support environment variables contained within its config files, it still provided a way for me to use environment variables.

[1]: https://kyrofa.com/posts/snapping-nextcloud-the-memcache

[Kyle Fazzari (kyrofa)]

I'm going to go ahead and invalidate the snapcraft task here. As the original report asked: "Is there a good facility for install-time copying of template configs or other writeable-from-initial-state data into $SNAP_USER_DATA?" Snapcraft is build-time; this is outside its purview.