Build proxy doesn't play nicely with maven-ant-tasks

[Evan]

Colin, Kit, and myself tried to debug through this ages ago, but I couldn't find a bug for it.

Ant blows up on snap builds:
https://build.snapcraft.io/user/snapcrafters/cassandra/52418
…
maven-ant-tasks-download:
     [echo] Downloading Maven ANT Tasks...
      [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
      [get] To: /build/c06aa5fe0b4650ca21947706c2dcb57a-xenial/parts/cassandra/build/build/maven-ant-tasks-2.1.3.jar
      [get] Error opening connection java.io.IOException: Server returned HTTP response code: 407 for URL: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
      [get] Error opening connection java.io.IOException: Server returned HTTP response code: 407 for URL: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
      [get] Error opening connection java.io.IOException: Server returned HTTP response code: 407 for URL: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
      [get] Can't get http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar to /build/c06aa5fe0b4650ca21947706c2dcb57a-xenial/parts/cassandra/build/build/maven-ant-tasks-2.1.3.jar

BUILD FAILED
/build/c06aa5fe0b4650ca21947706c2dcb57a-xenial/parts/cassandra/build/build.xml:277: Can't get http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar to /build/c06aa5fe0b4650ca21947706c2dcb57a-xenial/parts/cassandra/build/build/maven-ant-tasks-2.1.3.jar

Total time: 1 second
ant artifacts -Ddist.dir=/build/c06aa5fe0b4650ca21947706c2dcb57a-xenial/parts/cassandra/install
Command '['/bin/sh', '/tmp/tmpg2dgzsm_', 'ant', 'artifacts', '-Ddist.dir=/build/c06aa5fe0b4650ca21947706c2dcb57a-xenial/parts/cassandra/install']' returned non-zero exit status 1
Traceback (most recent call last):
  File "/usr/share/launchpad-buildd/slavebin/buildsnap", line 244, in main
    builder.build()
  File "/usr/share/launchpad-buildd/slavebin/buildsnap", line 191, in build
    ["snapcraft"], path=os.path.join("/build", self.name), env=env)
  File "/usr/share/launchpad-buildd/slavebin/buildsnap", line 100, in run_build_command
    ["/bin/sh", "-c", command], echo=echo, get_output=get_output)
  File "/usr/share/launchpad-buildd/slavebin/buildsnap", line 71, in chroot
    subprocess.check_call(cmd)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/bin/sudo', '/usr/sbin/chroot', '/home/buildd/build-SNAPBUILD-52418/chroot-autobuild', 'linux64', '/bin/sh', '-c', 'cd /build/c06aa5fe0b4650ca21947706c2dcb57a-xenial && env LANG=C.UTF-8 https_proxy=http://snap-proxy.launchpad.net:3128 http_proxy=http://snap-proxy.launchpad.net:3128 GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy snapcraft']' returned non-zero exit status 1
Revoking proxy token...

[Colin Watson (cjwatson)]

As I recall I got quite a way through debugging this and https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545 was a big help, but the build logs in my local Launchpad instance suggest that that wasn't enough. With local-snap-proxy applied, it looks like I ended up with this:

maven-ant-tasks-download:
     [echo] Downloading Maven ANT Tasks...
      [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
      [get] To: /build/cassandra/parts/cassandra/build/build/maven-ant-tasks-2.1.3.jar
      [get] Error getting http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar to /build/cassandra/parts/cassandra/build/build/maven-ant-tasks-2.1.3.jar

BUILD FAILED
/build/cassandra/parts/cassandra/build/build.xml:277: java.net.ConnectException: Connection refused (Connection refused)
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:589)
 at java.net.Socket.connect(Socket.java:538)
 at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
 at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
 at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
 at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
 at sun.net.www.http.HttpClient.New(HttpClient.java:308)
 at sun.net.www.http.HttpClient.New(HttpClient.java:326)
 at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
 at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
 at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
 at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
 at org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:728)
 at org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:641)
 at org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:631)

Total time: 6 seconds
Command '['/bin/sh', '/tmp/tmpqg9m6cnl', 'ant', '-autoproxy', 'artifacts', '-Ddist.dir=/build/cassandra/parts/cassandra/install']' returned non-zero exit status 1

The good news is that my fixes were enough to get your Jenkins snap working, although you need to apply https://github.com/evandandrea/jenkins-snap/pull/3.

[Colin Watson (cjwatson)]

Actually, it looks like I filed https://github.com/snapcore/snapcraft/pull/1256 after those local attempts and got it landed, so it may well be that local-snap-proxy is all that this needs now. I'll give it another try locally at some point and see what happens.

[Evan (ev)]

Ah interesting. I'm using the ant plugin directly (cassandra, not snapcraft):
https://github.com/snapcrafters/cassandra/blob/master/snapcraft.yaml

ant and maven are slightly out of sync, with ant missing authenticated proxy support:

+ if parsed.username is not None:
+ yield '-D{}.proxyUser={}'.format(scheme, parsed.username)
+ if parsed.password is not None:
+ yield '-D{}.proxyPassword={}'.format(scheme, parsed.password)

But that shouldn't break the unauthenticated builds on Launchpad.

[Evan (ev)]

Ah, I skimmed past local-snap-proxy on my first read. Any idea when that might land?

[Adam Collard (adam-collard)]

Ant and gradle fixes landed in https://github.com/snapcore/snapcraft/pull/1490 - still need to see if that fixes the issue

[Ondrej Kubik (ondrak)]

I tested again, and I actually pasted wrong build log, anyway here is correct one:
https://launchpadlibrarian.net/340422375/buildlog_snap_ubuntu_xenial_amd64_opengrok-snap_BUILDING.txt.gz
using snap craft.yaml
https://git.launchpad.net/~ondrak/ondras-snaps/+git/opengrok-snap/tree/snapcraft.yaml

result is still same, failing once it's trying to download

[Evan (ev)]

Also affecting jabref (https://build.snapcraft.io/user/JabRef/jabref/97443):

Building jabref
Downloading https://services.gradle.org/distributions/gradle-4.2.1-bin.zip

Exception in thread "main" java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
 at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2124)
 at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
 at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
 at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
 at org.gradle.wrapper.Download.downloadInternal(Download.java:66)
 at org.gradle.wrapper.Download.download(Download.java:51)
 at org.gradle.wrapper.Install$1.call(Install.java:62)
 at org.gradle.wrapper.Install$1.call(Install.java:48)
 at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)
 at org.gradle.wrapper.Install.createDist(Install.java:48)
 at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
 at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
./gradlew -Dhttp.proxyHost=snap-proxy.launchpad.net -Dhttp.proxyPort=3128 -Dhttp.proxyUser=SNAPBUILD-97443-1508933192 -Dhttp.proxyPassword=6ee4d054add7418ab1751cf7540bbed7 -Dhttps.proxyHost=snap-proxy.launchpad.net -Dhttps.proxyPort=3128 -Dhttps.proxyUser=SNAPBUILD-97443-1508933192 -Dhttps.proxyPassword=6ee4d054add7418ab1751cf7540bbed7 snapJar jar

[Evan (ev)]

My previous comment may be a snapcraft bug. Adam fixed some issues with gradle and proxies in https://github.com/snapcore/snapcraft/pull/1490, but you can see http.proxyUser in these logs so there's evidently still something going on.

[Colin Watson (cjwatson)]

I've just merged https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545. In local testing, with launchpad-buildd running in a suitably-firewalled VM and a local deployment of snap-proxy, this fixes a number of problems in this area: for example, the cases reported in bug 1690834 and bug 1753340 now build cleanly, as does jabref. In short, anything whose problem is just that it doesn't pass through proxy authentication details now works, which deals with quite a few disparate build systems.

Ondra, I don't think your opengrok problem really has much to do with this bug; I suspect that the downloading part of it was dealt with by improvements to snapcraft's ant plugin. In https://launchpadlibrarian.net/340504790/buildlog_snap_ubuntu_xenial_amd64_opengrok-snap_BUILDING.txt.gz I see that it's failing because dist/opengrok-*.tar.gz doesn't exist, and my local-snap-proxy branch doesn't change that. The targets in ant-build-targets don't appear to be being run at all; is the install script perhaps run before them?

The problems with maven-ant-tasks, as used by Cassandra, appear to be deeper. This build system is deprecated and unmaintained upstream, so digging through it to work out what's going on is challenging to say the least. Even after hacking snapcraft a bit to emit a Maven settings file as well as an Ant one, I found one problem with maven-ant-tasks which I would consider fatal: as far as I can see, it only supports picking up a single active proxy from its settings file, so unless you explicitly configure things at the project level (which obviously doesn't work in this sort of environment), you can have an HTTP proxy or an HTTPS proxy but not both. But despite being a fatal problem I don't think this is even what's going on here! I've straced the build, and I can see it reading the Maven settings file and then later in the same process going ahead and making unproxied requests anyway.

None of this appears to have anything to do with specifically *authenticated* proxies, and we're not the only ones to have this problem; for instance http://grokbase.com/t/ant/user/166fc12046/maven-ant-tasks-proxy-issue is about the same kind of thing and doesn't come to a useful conclusion. Given the notice on https://maven.apache.org/ant-tasks/, Cassandra would probably be well-advised to switch to something else that's considered well-maintained in the Java world.

Anyway, I have a cunning plan which should help even with this terrible situation, if I can get it to work. redsocks is a daemon that can tunnel arbitrary TCP connections from uncooperative programs via a proxy: it listens on a local port and you can use iptables to redirect selected connections to it, which it then wraps up in a CONNECT stream and sends to the configured proxy. This seems like exactly the kind of thing we need here, and would let us avoid having to debug our way through thousands of lines of unfamiliar and tangled build systems. At the moment this is at the level of thought experiment, but I'll play around and see if I can get something like this to work.

[Ondrej Kubik (ondrak)]

Thanks Colin, this has been working great for me!
I can confirm that my previously failing builds with ant plugin are now working just fine, including opengrok snap.

As of now I have only one snap which fails to build, and that's using maven-ant build, which as you described is bad idea from the beginning and hard to tackle

[Athos Ribeiro (athos-ribeiro)]

Hi,

I have been working on reviving the Cassandra snap [1]. I thought this would be a good time for that, given [2], which relates to stop using the deprecated "maven-ant-tasks", was resolved. Cassandra upstream no uses "resolver-ant-tasks".

The new resolver doesn't seem to be able to set/go through the proxy as well, as per [3].

Am I missing any details on how to use the proxy for triggering this build?

[1] https://forum.snapcraft.io/t/pointers-to-cassandra-state-history/24216
[2] https://issues.apache.org/jira/browse/CASSANDRA-16391
[3] https://launchpad.net/~athos-ribeiro/+snap/cassandra-test/+build/1404500

[Athos Ribeiro (athos-ribeiro)]

Just a follow up on this one:

I was able to successfully build the cassandra snap by injecting a maven settings.xml file containing the proxy definitions in build time. This was indeed the only way I could get the resolver-ant-tasks to use any proxy settings available.

As mentioned by cjwatson in the past, this may not be a general solution for this issue (see https://bugs.launchpad.net/launchpad-buildd/+bug/1702130/comments/9), but it does get our cassandra build to succeed.

Finally, and unrelated, the s390x builds are failing due to OOM errors, as in https://launchpadlibrarian.net/542224424/buildlog_snap_ubuntu_bionic_s390x_cassandra-test_BUILDING.txt.gz

A build sample is available at https://launchpad.net/~athos-ribeiro/+snap/cassandra-test/+build/1428615