organize keyword does not accept globs to organize into a target directory

[Christian Ehrhardt ]

Hi,
bug 1597600 got closed by adding the dump plugin, but I have found challenges to migrate from copy to dump/nil in my case.

The case:
So far I used the copy plugin (now deprecated) to achive the following:
- copy a file from the local source (source: .) into the snap to be installed
- copy it into the snap under a different name (using the file feature of copy)

Here a slimmed down snippet of my old snapcraft.yaml
  defaultconf:
    plugin: copy
    files:
        etc/example.conf: etc/ntp.conf
        etc/use*: etc/

Here (one of many) tries using dump:

  defaultconf:
    plugin: dump
    source: etc
    organize:
        etc/example.conf: etc/ntp.conf
        etc/use*: etc/

I had various similar tries with/without the "filesets" and "snap" keys as well as trying the nil plugin to only use filesets/organize features.

But none worked.
Most of the times I end up with the system complaining that the files are not at .../install/.. like:
FileNotFoundError: [Errno 2] No such file or directory: '/home/paelzer/work/snapsomething/ntpsec/parts/defaultconf/install/etc/example.conf' -> '/home/paelzer/work/snapsomething/ntpsec/parts/defaultconf/install/etc/ntp.conf'

But that is just what I used the copy plugin for before - adding extra files that got not added by the app's "make&install" step.

I'm sure it is not too hard, but it seems I can't get there by trial and error.
And neither doc nor IRC got it resolved, hence the bug report.

[Kyle Fazzari (kyrofa)]

Yeah you've just hit a bit of a limitation in the `organize` keyword, where you can't use globbing like the copy plugin supported. You need to be more explicit, e.g.

        plugin: dump
        source: etc
        organize:
          example.conf: etc/ntp.conf
          user1.conf: etc/user1.conf
          user2.conf: etc/user2.conf

Note also that since `source: etc` organize refers to the files relative to source. I also made some dumb assumptions about what the etc/use* files were actually named.

These limitations should definitely be resolved in order for the dump plugin to actually replace the copy plugin in functionality.

[Cris Dywan (kalikiana)]

Also one will likely want to use snap: here to avoid pulling in unexpected files. Particularly when copying files from the toplevel of the source folder it otherwise snaps the whole folder.

[Christian Ehrhardt  (paelzer)]

I'm agreeing to use the snap statement snap later to reduce files, but it has to work first :-)
Thanks already Kyle and Christian for the hints.

For now I must report that - just as in my former tries - even when dropping the grobbing it fails.
Again it complains that it can't find the files "to organize" at the install stage - which is right, they are just from the source tree.

  defaultconf:
    plugin: dump
    source: etc
    organize:
      etc/example.conf: etc/ntp.conf
      use-country-pool: etc/use-country-pool
      use-distro-pool: etc/use-distro-pool
      use-gpsd-json: etc/use-gpsd-json
      use-gpsd-shm: etc/use-gpsd-shm
      use-minimal-logging: etc/use-minimal-logging
      use-no-remote-configuration: etc/use-no-remote-configuration
      use-performance-logging: etc/use-performance-logging

snapcraft cleanbuild
[...]
'install' finished successfully (0.099s)
Staging defaultconf
[Errno 2] No such file or directory: '/root/parts/defaultconf/install/etc/example.conf'
Command '['lxc', 'exec', 'snapcraft-unvexatiously-unendeared-dorene', '--', 'snapcraft', 'snap', '--output', 'ntpsec_daily_amd64.snap']' returned non-zero exit status 1

I also experimented with filesets, snap and stage again, to add them to stage explicitly or so, But they all want it "from the install stage" which isn't my case.
I feel I just overlook something on the creation process stages, but never the less I need your help to overcome it.

[Sergio Schvezov (sergiusens)]

Thanks, I changed the title to reflect the actual bug.
If there are more, please log specific bugs to avoid the meta bug problem.

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/784