recent trove master CI failures in publish-checkbuild

Bug #1412949 reported by Brian Hunter
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Brian Hunter
Icehouse
Fix Released
High
Ihar Hrachyshka
Juno
Fix Released
High
Ihar Hrachyshka

Bug Description

For about 1 week, all trove/master reviews have been failing with Zuul failing the gate-trove-tox-doc-publish-checkbuild job.

Some facts that are known:
- this fails on both HP and rackspace builder nodes.
- this works on a developer's machine, but fails in the CI environment
- The logs suggest 301 redirect.
       - the logging, and the file system, show a bogus .jar file of some 184 bytes. This results in "not a zipfile" messages
         - Using wget --verbose, we can see a redirect being followed.
              - when we look at the redirect, we see it's redirecting http->https.

Denis Makogon took this issue to #openstack-infra last week, and credit goes to him for digging into the issue.

The following comments helped me close the loop on this.
...
fungi> denis_makogon: looking on logstash.o.o for message:"Caused by: java.io.IOException: Negative seek offset" in the past 7 days, it's only impacting gate-trove-tox-doc-publish-checkbuild but it's definitely happening on both rax and hpcloud workers. first occurrence i see is on change 144787,5 at 2015-01-10T23:00:29.000 utc
...
<clarkb> fungi: denis_makogon looking at that log again and running wget locally I think maven is hitting a 301 and not following it
<clarkb> fungi: denis_makogon: so I think our options are point at the actual destination or make maven smarter

The proposed fix is to "point to the actual destination" which are https urls.

Brian Hunter (bhunter)
Changed in trove:
assignee: nobody → Brian Hunter (bhunter)
Denis M. (dmakogon)
Changed in trove:
status: New → Confirmed
importance: Undecided → High
Changed in trove:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/148728
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=386ddc22680e8e9be3a34241067b0e34a192ebea
Submitter: Jenkins
Branch: master

commit 386ddc22680e8e9be3a34241067b0e34a192ebea
Author: Brian Hunter <email address hidden>
Date: Tue Jan 20 16:29:42 2015 -0500

    Fix trove-tox-doc-publish-checkbuild failures

    The owners of maven.research.rackspacecloud.com/ seem to have introduced
    a 301 redirect permanent from http -> https URLs on around Jan 10th.
    This has broken the master/trove check builds since that time.

    Maven, when running in the infrastructure CI environment, fails to
    follow these redirects.

    This code change simply refers to the maven software repositories using
    https URLs.

    Local testing of this change against Tesora's downstream CI found that
    it may be necessary to purge the broken/cached file from the bare-trusty
    CI build machines ala:
    rm /home/jenkins/.m2/repository/com/rackspace/cloud/api/wadl-tools/1.0.9
    (continued..) /wadl-tools-1.0.9.jar

    Closes-Bug: #1412949

    Change-Id: I2dece05c74ad18a96fa05791e368ff7283a5e4e5

Changed in trove:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/148885

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/148893

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (stable/juno)

Reviewed: https://review.openstack.org/148885
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=37a2ca668cc6a3954dda5c9f98badbcb067d275b
Submitter: Jenkins
Branch: stable/juno

commit 37a2ca668cc6a3954dda5c9f98badbcb067d275b
Author: Brian Hunter <email address hidden>
Date: Tue Jan 20 16:29:42 2015 -0500

    Fix trove-tox-doc-publish-checkbuild failures

    The owners of maven.research.rackspacecloud.com/ seem to have introduced
    a 301 redirect permanent from http -> https URLs on around Jan 10th.
    This has broken the master/trove check builds since that time.

    Maven, when running in the infrastructure CI environment, fails to
    follow these redirects.

    This code change simply refers to the maven software repositories using
    https URLs.

    Local testing of this change against Tesora's downstream CI found that
    it may be necessary to purge the broken/cached file from the bare-trusty
    CI build machines ala:
    rm /home/jenkins/.m2/repository/com/rackspace/cloud/api/wadl-tools/1.0.9
    (continued..) /wadl-tools-1.0.9.jar

    Closes-Bug: #1412949

    Change-Id: I2dece05c74ad18a96fa05791e368ff7283a5e4e5
    (cherry picked from commit 386ddc22680e8e9be3a34241067b0e34a192ebea)

tags: added: in-stable-juno
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (stable/icehouse)

Reviewed: https://review.openstack.org/148893
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=04f0ce4a93ea06e1af29cfe10390f16a30de9fa3
Submitter: Jenkins
Branch: stable/icehouse

commit 04f0ce4a93ea06e1af29cfe10390f16a30de9fa3
Author: Brian Hunter <email address hidden>
Date: Tue Jan 20 16:29:42 2015 -0500

    Fix trove-tox-doc-publish-checkbuild failures

    The owners of maven.research.rackspacecloud.com/ seem to have introduced
    a 301 redirect permanent from http -> https URLs on around Jan 10th.
    This has broken the master/trove check builds since that time.

    Maven, when running in the infrastructure CI environment, fails to
    follow these redirects.

    This code change simply refers to the maven software repositories using
    https URLs.

    Local testing of this change against Tesora's downstream CI found that
    it may be necessary to purge the broken/cached file from the bare-trusty
    CI build machines ala:
    rm /home/jenkins/.m2/repository/com/rackspace/cloud/api/wadl-tools/1.0.9
    (continued..) /wadl-tools-1.0.9.jar

    Closes-Bug: #1412949

    Conflicts:
     apidocs/pom.xml

    Change-Id: I2dece05c74ad18a96fa05791e368ff7283a5e4e5
    (cherry picked from commit 386ddc22680e8e9be3a34241067b0e34a192ebea)

tags: added: in-stable-icehouse
Changed in trove:
milestone: none → kilo-2
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Alan Pevec (apevec)
tags: removed: in-stable-icehouse in-stable-juno
Thierry Carrez (ttx)
Changed in trove:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.