trx build dependencies cause build failures

[Steve Varnau]

This build error is happening more frequently, including two of last night's test jobs.

Downloading: https://repository.apache.org/content/repositories/snapshots/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

[INFO] ------------------------------------------------------------------------ ##(SQF)
[INFO] BUILD FAILURE ##(SQF)
[INFO] ------------------------------------------------------------------------ ##(SQF)
[INFO] Total time: 1:26.086s ##(SQF)
[INFO] Finished at: Tue Apr 07 08:38:11 UTC 2015 ##(SQF)
[INFO] Final Memory: 15M/215M ##(SQF)
[INFO] ------------------------------------------------------------------------ ##(SQF)
[ERROR] Failed to execute goal on project hbase-trx-hdp2_2: Could not resolve dependencies for project org.apache:hbase-trx-hdp2_2:jar:1.1.0: Failed to collect dependencies for [org.apache.hbase:hbase-common:jar:0.98.4.2.2.0.0-2041-hadoop2 (compile), org.apache.hbase:hbase-protocol:jar:0.98.4.2.2.0.0-2041-hadoop2 (compile), org.apache.hbase:hbase-client:jar:0.98.4.2.2.0.0-2041-hadoop2 (compile), org.apache.hbase:hbase-server:jar:0.98.4.2.2.0.0-2041-hadoop2 (compile), org.apache.hbase:hbase-thrift:jar:0.98.4.2.2.0.0-2041-hadoop2 (compile), org.apache.hbase:hbase-testing-util:jar:0.98.4.2.2.0.0-2041-hadoop2 (test), org.apache.thrift:libthrift:jar:0.9.1 (compile), commons-logging:commons-logging:jar:1.1.3 (compile), org.apache.zookeeper:zookeeper:jar:3.4.5 (compile), com.google.protobuf:protobuf-java:jar:2.5.0 (compile), org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.5.0 (compile), org.apache.hadoop:hadoop-common:jar:2.5.0 (compile)]: Failed to read artifact descriptor for org.mortbay.jetty:jetty-util:jar:6.1.26.hwx: Could not transfer artifact org.mortbay.jetty:jetty-util:pom:6.1.26.hwx from/to ghelmling.testing (http://people.apache.org/~garyh/mvn/): Connection to http://people.apache.org refused: Connection timed out -> [Help 1] ##(SQF)
[ERROR] ##(SQF)

Note that the build indicates it is downloading pom file. When I try that URL, I get an error:
404 - Retrieval of /org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom from M2Repository(id=snapshots) is forbidden by repository policy SNAPSHOT.

Retrieval of /org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom from M2Repository(id=snapshots) is forbidden by repository policy SNAPSHOT.

Earlier in the build, there is this warning:
[WARNING] The POM for org.mortbay.jetty:jetty:jar:6.1.26.hwx is missing, no dependency information available ##(SQF)

Which makes me think that is why maven is downloading it every single time, because there is no version info to confirm it already has the correct version.

[Chris Sheedy (chris-sheedy)]

The problem still occurs frequently, for example, in https://jenkins02.trafodion.org/job/core-regress-seabase-ahw2.2/446/console which stored the make log in http://logs.trafodion.org/daily/core-regress-seabase-ahw2.2/06ed942/Make.log . The initial failure is:

    Failed to read artifact descriptor for org.mortbay.jetty:jetty-util:jar:6.1.26.hwx: Could not transfer artifact org.mortbay.jetty:jetty-util:pom:6.1.26.hwx from/to ghelmling.testing (http://people.apache.org/~garyh/mvn/): Connection to http://people.apache.org refused: Connection timed out ->

Something is making this download occur every time, because the file is not stored in the local maven repository. Since the file is not local, it is downloaded every time and make is more susceptible to problems accessing apache.org.

A longer extract from the makelog is:

[INFO] ------------------------------------------------------------------------ ##(SQF)
[INFO] Building HBase - Trx 1.1.0 ##(SQF)
[INFO] ------------------------------------------------------------------------ ##(SQF)
Downloading: http://repo.hortonworks.com/content/repositories/releases/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom ##(SQF)

Downloading: http://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom ##(SQF)

Downloading: https://repository.apache.org/content/repositories/releases/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom ##(SQF)

Downloading: http://download.java.net/maven/2/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom ##(SQF)

Downloading: http://repository.codehaus.org/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom ##(SQF)
. . .
Downloading: http://people.apache.org/~garyh/mvn/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom ##(SQF)

Downloading: https://repository.apache.org/content/repositories/snapshots/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom ##(SQF)

[WARNING] The POM for org.mortbay.jetty:jetty:jar:6.1.26.hwx is missing, no dependency information available ##(SQF)
Downloading: http://repo.hortonworks.com/content/repositories/releases/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

Downloading: http://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

Downloading: https://repository.apache.org/content/repositories/releases/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

Downloading: http://download.java.net/maven/2/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

Downloading: http://repository.codehaus.org/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

Downloading: http://repository.jboss.org/nexus/content/groups/public-jboss/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

Downloading: http://people.apache.org/~garyh/mvn/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

Downloading: https://repository.apache.org/content/repositories/snapshots/org/mortbay/jetty/jetty-util/6.1.26.hwx/jetty-util-6.1.26.hwx.pom ##(SQF)

[INFO] ------------------------------------------------------------------------ #...

[Chris Sheedy (chris-sheedy)]

This is the cause of several overnight build job failures

[Atanu Mishra (atanu-mishra)]

Chris -- can you please share the logs when the build is successful, in order to contrast against the failure case?

Assigned to Oliver, but Joanie has done some research previously that might be useful.

I don't expect we will be able to fix this in time for r1.1, hence targeting for the follow-on release.

[Alice Chen (alchen)]

You can get dependency tree information from Maven to see where this dependency is coming from by running :
mvn dependency:tree -Dverbose -Dincludes=org.mortbay.jetty

You will get something like the following :

[INFO] +- org.apache.hbase:hbase-server:jar:0.98.4.2.2.0.0-2041-hadoop2:test
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:test
[INFO] | | \- (org.mortbay.jetty:jetty-util:jar:6.1.26:test - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:test
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:test
[INFO] | | \- (org.mortbay.jetty:jetty:jar:6.1.26:test - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:test
[INFO] | | \- (org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:test - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:test
[INFO] | | \- (org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:test - omitted for duplicate)
[INFO] | \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:test
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.6.0.2.2.0.0-2041:test
[INFO] | +- (org.mortbay.jetty:jetty:jar:6.1.26.hwx:test - omitted for conflict with 6.1.26)
[INFO] | +- (org.mortbay.jetty:jetty-util:jar:6.1.26.hwx:test - omitted for conflict with 6.1.26)
[INFO] | \- org.htrace:htrace-core:jar:3.0.4:test
[INFO] | \- (org.mortbay.jetty:jetty-util:jar:6.1.26:test - omitted for duplicate)
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.6.0.2.2.0.0-2041:test
[INFO] | +- (org.mortbay.jetty:jetty:jar:6.1.26.hwx:test - omitted for conflict with 6.1.26)
[INFO] | \- (org.mortbay.jetty:jetty-util:jar:6.1.26.hwx:test - omitted for conflict with 6.1.26)
[INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0.2.2.0.0-2041:test
[INFO] | \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0.2.2.0.0-2041:test
[INFO] | \- (org.mortbay.jetty:jetty-util:jar:6.1.26.hwx:test - omitted for conflict with 6.1.26)
[INFO] \- org.cloudera.htrace:htrace-core:jar:2.04:test
[INFO] \- (org.mortbay.jetty:jetty-util:jar:6.1.26:test - omitted for duplicate)

[Oliver Bucaojit (oliver-bucaojit)]

I have a change that I think will fix this issue.

From the logs, during the HDP Maven compile, maven is not finding the Jetty artifacts in the repository that we have specified in the POM file:
http://repo.hortonworks.com/content/repositories/releases/

On a successful build I see that Maven goes through various other repositories and finally finds the artifacts in:
 http://repo.hortonworks.com/content/groups/public/

On the failure build, Maven cycles through quite a few repositories and fails on people.apache.org as the connection is timing out. That is before reaching the correct repository.

I find that when I explicitly specify the ' http://repo.hortonworks.com/content/groups/public/' repository in the HDP POM file, Maven will only cycle through the two repositories that we specify when it tries to download Jetty and does not attempt to download from people.apache.org.

Example log:
Downloading: http://repo.hortonworks.com/content/repositories/releases/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom
           ^MDownloading: http://repo.hortonworks.com/content/groups/public/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom3/7 KB ^M7/7 KB ^M ^M
Downloaded: http://repo.hortonworks.com/content/groups/public/org/mortbay/jetty/jetty/6.1.26.hwx/jetty-6.1.26.hwx.pom (7 KB at 54.6 KB/sec)

[Oliver Bucaojit (oliver-bucaojit)]

Fix was merged in May 11:
https://review.trafodion.org/#/c/1639/

[Oliver Bucaojit (oliver-bucaojit)]

Fix merged May 11.
Have not seen any new occurrences of issues downloading Jetty artifacts.