The nodejs plugin downloads packages during the build

[Leo Arias]

The build step of the nodejs and gulp plugins involves running npm install. This makes launchpad builds fail, because they only allow downloads during the pull phase:

Building assets
npm install -g gulp-cli
npm ERR! Linux 3.13.0-93-generic
npm ERR! argv "/build/snapweb/parts/assets/npm/bin/node" "/build/snapweb/parts/assets/npm/bin/npm" "install" "-g" "gulp-cli"
npm ERR! node v4.4.4
npm ERR! npm v2.15.1
npm ERR! code ENOTFOUND
npm ERR! errno ENOTFOUND
npm ERR! syscall getaddrinfo

[Leo Arias (elopio)]

Sergio suggests that instead of changing the plugins, launchpad should allow downloads during the build phase too. I've added the launchpad task so we can start the discussion to see where is the best place to fix this.

[Colin Watson (cjwatson)]

We're quite reluctant to do this because it basically opens the floodgates to having all snaps be able to download anything whenever they like. I understand that this would have some usability benefits, but it imposes a strong requirement on us to do much more monitoring of our build farm than we currently have the capacity to do. Anyone at all can ask Launchpad to build a snap for them, and that means that anyone can cause the Launchpad build farm to issue fairly arbitrary web requests: what happens if somebody uses us to execute a denial of service attack against some other service? What if that causes that service to blacklist us, affecting anyone else who wants to use the Launchpad build farm to perform legitimate snap builds?

The benefit of doing things in the pull phase is that it can be (at least almost entirely) declarative, rather than giving people largely-unrestricted access to issue more or less whatever requests they like through our proxy.

As such, we'd push back quite hard against a proposal to open this up to the build phase. Please consider doing this in pull instead.

[Kyle Fazzari (kyrofa)]

Thanks for the reply, Colin. I see where you're coming from, but I also see two problems with it:

1) The floodgates are already open. If someone wants to get around the blockade, all they have to do is use a local plugin that does whatever they want in the pull phase, like so:

    http://pastebin.ubuntu.com/23047975/

2) You're trying to avoid internet access in the build step. But by asking us to "do this in pull instead," you're literally asking us to combine the build and pull step. Which means we have internet access in the build step anyway, but we lose our pull->build->stage->prime lifecycle for the plugins in question. For _everyone_, not just users of the Launchpad snap builders.

With these things in mind, it seems to us that blocking internet access in all but the pull step doesn't do much to protect anything, but definitely gets in the way of legitimate use.

[Leo Arias (elopio)]

Could we have firewall rules that deny excessive downloads to the same URL to forbid DOS attacks?
Those rules should be put in place in the pull phase now, as Kyle points out.

There are all kinds of crazy build scripts out there. Forcing them to do all the downloads in the pull is one more step for upstreams to adopt snaps, and sometimes it's not even doable.

It sounds really nice, because you could pull in one machine, copy everything to another and then do the build in there completely isolated. But from what I've seen these months building snaps, it might be too much to ask.

[Sergio Schvezov (sergiusens)]

Related bug LP: #1616421

[Sergio Schvezov (sergiusens)]

Moved the gulp work to LP: #1618622

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/762

[Bruno Nova (brunonova)]

I just tried building a snap that uses gradle in Launchpad, and it failed: first when downloading the gradle wrapper and then, after switching to the system "gradle" command, when downloading the dependencies from Maven.

This makes the gradle plugin completely useless in Launchpad.
And I suppose maven will also fail.

[Sergio Schvezov (sergiusens)]

plugin no longer supported