build failing on download by ant build

[Brett Sutton]

I have a snap part which builds fine under lxd however it fails to build on the build servers.

The error appears to indicate that it is unable to download a file which stops the build proceeding.

There also appears to be an secondary error as there is an error regarding a python library however the snap in question doesn't use python for the build. The build uses ant so I'm guessing the python error might be caused by the python plugin for ant.

The problem seems a little similar to a number of other reports related to ant in that there are some problems with ant downloading files.

The difference with this issue is that it reports an error saying it can't find a file rather than that it can't download a file.

For reference:
https://bugs.launchpad.net/launchpad-buildd/+bug/1690834
https://bugs.launchpad.net/launchpad-buildd/+bug/1702130
https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545

The build in question is here:

https://build.snapcraft.io/user/bsutton/tomcat-with-ssl-snap/158974

The primary error is:

testexist:
     [echo] Testing for /root/tomcat-build-libs/commons-daemon-1.1.0/commons-daemon-1.1.0.jar
downloadgz-2:
    [mkdir] Created dir: /root/tomcat-build-libs
setproxy:
trydownload.check:
trydownload:
setproxy:
trydownload.check:
trydownload:
BUILD FAILED
/build/e4039a0b32e4bca67d290294684c86d6-xenial/parts/tomcat-with-ssl/build/build.xml:2623: The following error occurred while executing this line:
/build/e4039a0b32e4bca67d290294684c86d6-xenial/parts/tomcat-with-ssl/build/build.xml:2773: the archive download-604074008.tar.gz doesn't exist

Secondary error:
Total time: 0 seconds
ant
/usr/lib/python3/dist-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled. Import error is: No module named 'pyasn1'
  warnings.warn(import_error_msg)
/usr/lib/python3/dist-packages/ndg/httpsclient/ssl_peer_verification.py:25: UserWarning: SubjectAltName support is disabled - check pyasn1 package installation to enable
  warnings.warn(SUBJ_ALT_NAME_SUPPORT_MSG)
/usr/lib/python3/dist-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled. Import error is: No module named 'pyasn1'

[Brett Sutton (bsutton)]

I've narrowed this problem down and now believe it is a proxy issue.

Ant attempts to download a number of gz files as part of its build process.

Its my understanding that the build servers restrict direct access and all downloads must go via a proxy.
I believe that Ant is ignoring the proxy settings so the download fails.

So two questions:

1) how do I discover the proxy settings on the build servers
2) how do I pass those proxy settings to the Ant build process

Help would be greatly appreciated.

[Adam Collard (adam-collard)]

I'm unable to view the build on build.snapcraft.io - could you share the snapcraft.yaml that failed?

The various Java build tools are very tricky to configure w.r.t. proxies, any insights you can share on how to get Ant to play nicely with an authenticated proxy would be welcome

[Brett Sutton (bsutton)]

I'm no Ant expert but it looks to me like the ant plugin is able to pick
up proxy settings from environment variables.

https://github.com/snapcore/snapcraft/blob/master/snapcraft/plugins/ant.py

I don't do python but :

   def get_proxy_options(self, scheme):
         proxy = os.environ.get('{}_proxy'.format(scheme))
         if proxy:
             parsed = urlsplit(proxy)
             if parsed.hostname is not None:
                 yield '-D{}.proxyHost={}'.format(scheme, parsed.hostname)
             if parsed.port is not None:
                 yield '-D{}.proxyPort={}'.format(scheme, parsed.port)
             if parsed.username is not None:
                 yield '-D{}.proxyUser={}'.format(scheme, parsed.username)
             if parsed.password is not None:
                 yield '-D{}.proxyPassword={}'.format(scheme,
parsed.password)

Which I read as looking for environment vars such as:

http(s)_proxyHost

http(s)_proxyPort

http(s)_proxyUser

http(s)_proxyPassword

Ant directly supports an 'autoproxy' option which I've tried (in the
IrrigationForPi app mentioned below)

https://ant.apache.org/manual/proxy.html

Ant also allows you to configure a properties file that contains args
and I think these can be configured to also pickup environment vars.

I believe you can also pass these type of args from the snapcraft ant
plugins ant-options key.

You will see in the IrrigationForPi snapcraft that I've tried this as well.

So I think I just need to know if the build servers provide environment
vars and their names  or what the settings should be.

My script is attached.

I have a second app which is public which has the same problem and which
is on the build servers so you can look at that one.

https://build.snapcraft.io/user/bsutton/IrrigationForPi-Snap

The code is available here:

https://github.com/bsutton/IrrigationForPi

Snapcraft here:

https://github.com/bsutton/IrrigationForPi/tree/master/snap-projects/installer/snap

its the 'tomcat' part that is causing the problem.

Brett

On 10/05/18 23:18, Adam Collard wrote:
> I'm unable to view the build on build.snapcraft.io - could you share the
> snapcraft.yaml that failed?
>
> The various Java build tools are very tricky to configure w.r.t.
> proxies, any insights you can share on how to get Ant to play nicely
> with an authenticated proxy would be welcome
>
> ** Changed in: launchpad-buildd
> Status: New => Incomplete
>

[Colin Watson (cjwatson)]

launchpad-buildd sets http_proxy and https_proxy environment variables; in the current configuration they will contain a username and password, although this may not be true in future. Note that there's a unique proxy token for each build; you cannot expect to be able to hardcode it.

My experience here is that the hard bit (as in, I tried for several days and eventually gave up) is getting ant to cope with the username and password bits properly. That's why I proposed https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545 to make that no longer necessary, but that's yet to have a full review.

[Brett Sutton (bsutton)]

I have to say that I don't understand why the build servers aren't using transparent proxies.

What ever was the logic behind making them require auth?

Given the builds are meant to use the proxy and I assume the builds are running in a closed environment and hence there is no need for the proxies to be public why on earth would they require authentication.

The other issue here is that I shouldn't be required to customize my snapcraft.yaml just to make it work on the build servers.

The plugins should make running on a build server transparent.
In this case it means the plugins should know about the build server proxies settings and pick them up and run with them without requiring any special config.

Of course if we go back to my earlier point (transparent proxies) this would be a moot point.

[Brett Sutton (bsutton)]

Where can I find the proxy environment variable names for the build servers?

[Colin Watson (cjwatson)]

We use authenticated proxies because only *some* of the builds on our build farm are allowed to talk to the internet at large, and there's nothing to distinguish between them in terms of their network environment; they have to be provided with credentials.

I agree that you should not be required to customise your snapcraft.yaml. I think it's fairly probable that my local-snap-proxy branch will make this much simpler, and possibly you won't need to do anything else at all after that.

I already told you the proxy environment variable names; see comment #4.

[Colin Watson (cjwatson)]

Incidentally, snapcraft's ant plugin already does pick up these environment variables and *attempt* to pass them on to ant, in the bit of code you quoted earlier (you just misidentified what environment variable names it's looking at). The problem is not to do with snapcraft using the wrong environment variable names; the problem is in getting ant to actually use everything properly.

It's been a while since I dug into this, but my memory is that ant's -autoproxy sets a system property and then trusts the JRE to do the right thing, but the JRE doesn't actually honour authenticated proxies and requires a java.net.Authenticator object to be instantiated manually. ant can do the latter, but only with explicit configuration that it's hard to stuff in from the snapcraft level.

The point of my local-snap-proxy branch was to arrange that individual build tools don't need to be aware of the authenticated nature of the proxy, but can instead rely on a secondary proxy running on the builder itself which has the necessary credentials and passes them on to an upstream proxy. That allows us to preserve our build farm security arrangements while keeping things simple for build tools, since in practice too many things don't cope very well with authenticated proxies. I'm slightly confused that you already referenced my merge proposal in your initial bug description but then we've been going through in the course of this bug rediscovering the lines of thought that led to me doing that work in the first place; but maybe I didn't explain its purpose very clearly before, in which case hopefully this will help. And I know that we need to get it actually reviewed and landed.

[Brett Sutton (bsutton)]

Colin,
Thanks for the detailed response and yes I agree that your proposal is the
way forward.

My detailed questions was around getting my build workng until your
proposal got released.

I had seen the environment variables in comment 4 but that didn't appear to
give me enough detail to use them

You mention two variable names but I assume I need four for each protocol.

Url
Port
Username
Password

And I couldn't see how to get these from the details you provided.

On Sun., 13 May 2018, 2:15 am Colin Watson, <email address hidden> wrote:

> Incidentally, snapcraft's ant plugin already does pick up these
> environment variables and *attempt* to pass them on to ant, in the bit
> of code you quoted earlier (you just misidentified what environment
> variable names it's looking at). The problem is not to do with
> snapcraft using the wrong environment variable names; the problem is in
> getting ant to actually use everything properly.
>
> It's been a while since I dug into this, but my memory is that ant's
> -autoproxy sets a system property and then trusts the JRE to do the
> right thing, but the JRE doesn't actually honour authenticated proxies
> and requires a java.net.Authenticator object to be instantiated
> manually. ant can do the latter, but only with explicit configuration
> that it's hard to stuff in from the snapcraft level.
>
> The point of my local-snap-proxy branch was to arrange that individual
> build tools don't need to be aware of the authenticated nature of the
> proxy, but can instead rely on a secondary proxy running on the builder
> itself which has the necessary credentials and passes them on to an
> upstream proxy. That allows us to preserve our build farm security
> arrangements while keeping things simple for build tools, since in
> practice too many things don't cope very well with authenticated
> proxies. I'm slightly confused that you already referenced my merge
> proposal in your initial bug description but then we've been going
> through in the course of this bug rediscovering the lines of thought
> that led to me doing that work in the first place; but maybe I didn't
> explain its purpose very clearly before, in which case hopefully this
> will help. And I know that we need to get it actually reviewed and
> landed.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1753340
>
> Title:
> build failing on download by ant build
>
> Status in launchpad-buildd:
> Incomplete
>
> Bug description:
> I have a snap part which builds fine under lxd however it fails to
> build on the build servers.
>
> The error appears to indicate that it is unable to download a file
> which stops the build proceeding.
>
> There also appears to be an secondary error as there is an error
> regarding a python library however the snap in question doesn't use
> python for the build. The build uses ant so I'm guessing the python
> error might be caused by the python plugin for ant.
>
> The problem seems a little similar to a number of other reports
> related to ant in that there are some problems with ant downloading
> files.
>
> The diffe...

[Colin Watson (cjwatson)]

I'm not sure there is necessarily any way to get your build working
before my proposal is landed. I mean, if we knew how to stuff the
necessary information into ant in a way that causes it to work then we
could just do it in snapcraft, right?

Your assumption about needing four environment variables for each
protocol is incorrect, perhaps because you misunderstood the Python
code. http_proxy and https_proxy are each in this kind of form:

  http://USERNAME:PASSWORD@HOST:PORT/

The snapcraft code that you pointed to in comment #3 unpacks this into
host/port/username/password components and attempts to pass those on to
ant in separate -D options. It's just that it doesn't appear to
actually work ...

[Brett Sutton (bsutton)]

It's there any ETA on the proposal landing?

On Sun., 13 May 2018, 7:01 pm Colin Watson, <email address hidden> wrote:

> I'm not sure there is necessarily any way to get your build working
> before my proposal is landed. I mean, if we knew how to stuff the
> necessary information into ant in a way that causes it to work then we
> could just do it in snapcraft, right?
>
> Your assumption about needing four environment variables for each
> protocol is incorrect, perhaps because you misunderstood the Python
> code. http_proxy and https_proxy are each in this kind of form:
>
> http://USERNAME:PASSWORD@HOST:PORT/
>
> The snapcraft code that you pointed to in comment #3 unpacks this into
> host/port/username/password components and attempts to pass those on to
> ant in separate -D options. It's just that it doesn't appear to
> actually work ...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1753340
>
> Title:
> build failing on download by ant build
>
> Status in launchpad-buildd:
> Incomplete
>
> Bug description:
> I have a snap part which builds fine under lxd however it fails to
> build on the build servers.
>
> The error appears to indicate that it is unable to download a file
> which stops the build proceeding.
>
> There also appears to be an secondary error as there is an error
> regarding a python library however the snap in question doesn't use
> python for the build. The build uses ant so I'm guessing the python
> error might be caused by the python plugin for ant.
>
> The problem seems a little similar to a number of other reports
> related to ant in that there are some problems with ant downloading
> files.
>
> The difference with this issue is that it reports an error saying it
> can't find a file rather than that it can't download a file.
>
> For reference:
> https://bugs.launchpad.net/launchpad-buildd/+bug/1690834
> https://bugs.launchpad.net/launchpad-buildd/+bug/1702130
>
> https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545
>
>
> The build in question is here:
>
> https://build.snapcraft.io/user/bsutton/tomcat-with-ssl-snap/158974
>
> The primary error is:
>
> testexist:
> [echo] Testing for
> /root/tomcat-build-libs/commons-daemon-1.1.0/commons-daemon-1.1.0.jar
> downloadgz-2:
> [mkdir] Created dir: /root/tomcat-build-libs
> setproxy:
> trydownload.check:
> trydownload:
> setproxy:
> trydownload.check:
> trydownload:
> BUILD FAILED
>
> /build/e4039a0b32e4bca67d290294684c86d6-xenial/parts/tomcat-with-ssl/build/build.xml:2623:
> The following error occurred while executing this line:
>
> /build/e4039a0b32e4bca67d290294684c86d6-xenial/parts/tomcat-with-ssl/build/build.xml:2773:
> the archive download-604074008.tar.gz doesn't exist
>
>
> Secondary error:
> Total time: 0 seconds
> ant
> /usr/lib/python3/dist-packages/ndg/httpsclient/subj_alt_name.py:22:
> UserWarning: Error importing pyasn1, subjectAltName check for SSL peer
> verification will be disabled. Import error is: No module named 'pyasn1'
> warnings.warn(import_error_msg)
...

[Colin Watson (cjwatson)]

I don't have an ETA, but I've reminded my reviewer of its existence.

[Brett Sutton (bsutton)]

Now we have a fix (yeah) do we have an ETA on the fix being deployed?

And thanks for the fix!

[Colin Watson (cjwatson)]

I'm working on it. Likely this week.

[Colin Watson (cjwatson)]

Fixed in launchpad-buildd 162, now on production.

[Brett Sutton (bsutton)]

Thanks!

On Thu., 14 Jun. 2018, 8:36 pm Colin Watson, <email address hidden> wrote:

> Fixed in launchpad-buildd 162, now on production.
>
> ** Changed in: launchpad-buildd
> Status: Fix Committed => Fix Released
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1753340
>
> Title:
> build failing on download by ant build
>
> Status in launchpad-buildd:
> Fix Released
>
> Bug description:
> I have a snap part which builds fine under lxd however it fails to
> build on the build servers.
>
> The error appears to indicate that it is unable to download a file
> which stops the build proceeding.
>
> There also appears to be an secondary error as there is an error
> regarding a python library however the snap in question doesn't use
> python for the build. The build uses ant so I'm guessing the python
> error might be caused by the python plugin for ant.
>
> The problem seems a little similar to a number of other reports
> related to ant in that there are some problems with ant downloading
> files.
>
> The difference with this issue is that it reports an error saying it
> can't find a file rather than that it can't download a file.
>
> For reference:
> https://bugs.launchpad.net/launchpad-buildd/+bug/1690834
> https://bugs.launchpad.net/launchpad-buildd/+bug/1702130
>
> https://code.launchpad.net/~cjwatson/launchpad-buildd/local-snap-proxy/+merge/322545
>
>
> The build in question is here:
>
> https://build.snapcraft.io/user/bsutton/tomcat-with-ssl-snap/158974
>
> The primary error is:
>
> testexist:
> [echo] Testing for
> /root/tomcat-build-libs/commons-daemon-1.1.0/commons-daemon-1.1.0.jar
> downloadgz-2:
> [mkdir] Created dir: /root/tomcat-build-libs
> setproxy:
> trydownload.check:
> trydownload:
> setproxy:
> trydownload.check:
> trydownload:
> BUILD FAILED
>
> /build/e4039a0b32e4bca67d290294684c86d6-xenial/parts/tomcat-with-ssl/build/build.xml:2623:
> The following error occurred while executing this line:
>
> /build/e4039a0b32e4bca67d290294684c86d6-xenial/parts/tomcat-with-ssl/build/build.xml:2773:
> the archive download-604074008.tar.gz doesn't exist
>
>
> Secondary error:
> Total time: 0 seconds
> ant
> /usr/lib/python3/dist-packages/ndg/httpsclient/subj_alt_name.py:22:
> UserWarning: Error importing pyasn1, subjectAltName check for SSL peer
> verification will be disabled. Import error is: No module named 'pyasn1'
> warnings.warn(import_error_msg)
>
> /usr/lib/python3/dist-packages/ndg/httpsclient/ssl_peer_verification.py:25:
> UserWarning: SubjectAltName support is disabled - check pyasn1 package
> installation to enable
> warnings.warn(SUBJ_ALT_NAME_SUPPORT_MSG)
> /usr/lib/python3/dist-packages/ndg/httpsclient/subj_alt_name.py:22:
> UserWarning: Error importing pyasn1, subjectAltName check for SSL peer
> verification will be disabled. Import error is: No module named 'pyasn1'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/launchpad-buildd/+bug/1753340/+subscriptions
>

[Brett Sutton (bsutton)]

And my snap now builds !!!!

Thanks for the effort.