launchpad-buildd

Bug #1702130
Comment #9

Comment 9 for bug 1702130

Revision history for this message

Colin Watson (cjwatson) wrote on 2018-06-10:

I've just merged https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545. In local testing, with launchpad-buildd running in a suitably-firewalled VM and a local deployment of snap-proxy, this fixes a number of problems in this area: for example, the cases reported in bug 1690834 and bug 1753340 now build cleanly, as does jabref. In short, anything whose problem is just that it doesn't pass through proxy authentication details now works, which deals with quite a few disparate build systems.

Ondra, I don't think your opengrok problem really has much to do with this bug; I suspect that the downloading part of it was dealt with by improvements to snapcraft's ant plugin. In https://launchpadlibrarian.net/340504790/buildlog_snap_ubuntu_xenial_amd64_opengrok-snap_BUILDING.txt.gz I see that it's failing because dist/opengrok-*.tar.gz doesn't exist, and my local-snap-proxy branch doesn't change that. The targets in ant-build-targets don't appear to be being run at all; is the install script perhaps run before them?

The problems with maven-ant-tasks, as used by Cassandra, appear to be deeper. This build system is deprecated and unmaintained upstream, so digging through it to work out what's going on is challenging to say the least. Even after hacking snapcraft a bit to emit a Maven settings file as well as an Ant one, I found one problem with maven-ant-tasks which I would consider fatal: as far as I can see, it only supports picking up a single active proxy from its settings file, so unless you explicitly configure things at the project level (which obviously doesn't work in this sort of environment), you can have an HTTP proxy or an HTTPS proxy but not both. But despite being a fatal problem I don't think this is even what's going on here! I've straced the build, and I can see it reading the Maven settings file and then later in the same process going ahead and making unproxied requests anyway.

None of this appears to have anything to do with specifically *authenticated* proxies, and we're not the only ones to have this problem; for instance http://grokbase.com/t/ant/user/166fc12046/maven-ant-tasks-proxy-issue is about the same kind of thing and doesn't come to a useful conclusion. Given the notice on https://maven.apache.org/ant-tasks/, Cassandra would probably be well-advised to switch to something else that's considered well-maintained in the Java world.

Anyway, I have a cunning plan which should help even with this terrible situation, if I can get it to work. redsocks is a daemon that can tunnel arbitrary TCP connections from uncooperative programs via a proxy: it listens on a local port and you can use iptables to redirect selected connections to it, which it then wraps up in a CONNECT stream and sends to the configured proxy. This seems like exactly the kind of thing we need here, and would let us avoid having to debug our way through thousands of lines of unfamiliar and tangled build systems. At the moment this is at the level of thought experiment, but I'll play around and see if I can get something like this to work.

I've just merged https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545.  In local testing, with launchpad-buildd running in a suitably-firewalled VM and a local deployment of snap-proxy, this fixes a number of problems in this area: for example, the cases reported in bug 1690834 and bug 1753340 now build cleanly, as does jabref.  In short, anything whose problem is just that it doesn't pass through proxy authentication details now works, which deals with quite a few disparate build systems.

Ondra, I don't think your opengrok problem really has much to do with this bug; I suspect that the downloading part of it was dealt with by improvements to snapcraft's ant plugin.  In https://launchpadlibrarian.net/340504790/buildlog_snap_ubuntu_xenial_amd64_opengrok-snap_BUILDING.txt.gz I see that it's failing because dist/opengrok-*.tar.gz doesn't exist, and my local-snap-proxy branch doesn't change that.  The targets in ant-build-targets don't appear to be being run at all; is the install script perhaps run before them?

The problems with maven-ant-tasks, as used by Cassandra, appear to be deeper.  This build system is deprecated and unmaintained upstream, so digging through it to work out what's going on is challenging to say the least.  Even after hacking snapcraft a bit to emit a Maven settings file as well as an Ant one, I found one problem with maven-ant-tasks which I would consider fatal: as far as I can see, it only supports picking up a single active proxy from its settings file, so unless you explicitly configure things at the project level (which obviously doesn't work in this sort of environment), you can have an HTTP proxy or an HTTPS proxy but not both.  But despite being a fatal problem I don't think this is even what's going on here!  I've straced the build, and I can see it reading the Maven settings file and then later in the same process going ahead and making unproxied requests anyway.

None of this appears to have anything to do with specifically *authenticated* proxies, and we're not the only ones to have this problem; for instance http://grokbase.com/t/ant/user/166fc12046/maven-ant-tasks-proxy-issue is about the same kind of thing and doesn't come to a useful conclusion.  Given the notice on https://maven.apache.org/ant-tasks/, Cassandra would probably be well-advised to switch to something else that's considered well-maintained in the Java world.

Anyway, I have a cunning plan which should help even with this terrible situation, if I can get it to work.  redsocks is a daemon that can tunnel arbitrary TCP connections from uncooperative programs via a proxy: it listens on a local port and you can use iptables to redirect selected connections to it, which it then wraps up in a CONNECT stream and sends to the configured proxy.  This seems like exactly the kind of thing we need here, and would let us avoid having to debug our way through thousands of lines of unfamiliar and tangled build systems.  At the moment this is at the level of thought experiment, but I'll play around and see if I can get something like this to work.