Job failed due to no devstack directory

Bug #1365046 reported by Matthew Booth
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Confirmed
Undecided
Unassigned
OpenStack-Gate
Confirmed
Undecided
Unassigned

Bug Description

Saw this in the logs:

2014-09-03 08:43:37.035 | Setting up the host
2014-09-03 08:43:37.035 | ... this takes a few seconds (logs at logs/devstack-gate-setup-host.txt.gz)
2014-09-03 08:44:23.739 | Setting up the workspace
2014-09-03 08:44:23.739 | ... this takes 3 - 5 minutes (logs at logs/devstack-gate-setup-workspace-new.txt.gz)
2014-09-03 08:53:33.691 | Running gate_hook
2014-09-03 08:53:33.718 | Job timeout set to: 106 minutes
2014-09-03 08:53:33.719 | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 315: cd: /opt/stack/new/devstack: No such file or directory
2014-09-03 08:53:33.719 | ERROR: the main setup script run by this job failed - exit code: 1
2014-09-03 08:53:33.719 | please look at the relevant log files to determine the root cause

Couldn't find anything more specific. Full logs here: http://logs.openstack.org/54/109754/13/check/check-tempest-dsvm-full/10d7fe8/

Tags: gate-failure
Revision history for this message
Clark Boylan (cboylan) wrote :

A bit more triage shows http://logs.openstack.org/54/109754/13/check/check-tempest-dsvm-full/10d7fe8/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-09-03_08_50_25_414. Looks like git hit a TLS error fetching nova refs from the git mirrors. This made workspace setup short circuit and fail without cloning and updating devstack.

tags: added: gate-failure
Jeremy Stanley (fungi)
Changed in openstack-ci:
status: New → Incomplete
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :
Changed in openstack-ci:
status: Incomplete → Confirmed
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Copying error message into comments for completeness:

fatal: unable to access 'https://git.openstack.org/openstack/nova/': GnuTLS recv error (-9): A TLS packet with unexpected length was received.

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

Querying on this isn't good enough since there is a retry:

message:"error\: RPC failed; result=56, HTTP code = 0" AND tags:"console"

So it's not 100% fail. We could add a better error message in that log when the workspace setup fails and we've hit the total number of retries.

Until then we probably need to query on the failure in the console log:

http://logs.openstack.org/47/133947/3/gate/gate-tempest-dsvm-neutron-heat-slow/d39bd4c/console.html#_2014-11-18_12_44_21_376

message:"/opt/stack/new/devstack-gate/devstack-vm-gate.sh: line" AND message:"cd: /opt/stack/new/devstack: No such file or directory" AND tags:"console"

There are 161 hits for that:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiL29wdC9zdGFjay9uZXcvZGV2c3RhY2stZ2F0ZS9kZXZzdGFjay12bS1nYXRlLnNoOiBsaW5lXCIgQU5EIG1lc3NhZ2U6XCJjZDogL29wdC9zdGFjay9uZXcvZGV2c3RhY2s6IE5vIHN1Y2ggZmlsZSBvciBkaXJlY3RvcnlcIiBBTkQgdGFnczpcImNvbnNvbGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQxNjQxNTI4NjE5Nn0=

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/135654

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to devstack-gate (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/135656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/135654
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=5415de703547c0828bee6b89a2534c9f89a9370f
Submitter: Jenkins
Branch: master

commit 5415de703547c0828bee6b89a2534c9f89a9370f
Author: Matt Riedemann <email address hidden>
Date: Wed Nov 19 08:53:13 2014 -0800

    Add query for infra bug 1365046

    161 hits in 7 days, check and gate, all failures.

    There is a better error message to be had in
    devstack-gate so I'm going to work a separate change
    for that and then we can update this query, but for
    now this is the only 100% fail since there is a retry
    loop involved in devstack-gate when doing the git
    remote update.

    Change-Id: I2ac631ee42b7bdf6f4bcc3d61c4dadea321a23d9
    Related-Bug: #1365046

Jeremy Stanley (fungi)
Changed in openstack-gate:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to devstack-gate (master)

Reviewed: https://review.openstack.org/135656
Committed: https://git.openstack.org/cgit/openstack-infra/devstack-gate/commit/?id=aed2c8b87cafd43a53775622f19312b6dcb47fe5
Submitter: Jenkins
Branch: master

commit aed2c8b87cafd43a53775622f19312b6dcb47fe5
Author: Matt Riedemann <email address hidden>
Date: Wed Nov 19 08:58:45 2014 -0800

    Give a more specific error message when giving up in git_remote_update

    git_remote_update is intermittently failing in the gate but we don't
    have a good error message in the devstack logs for this specific case
    due to the retry loop (we don't get 100% failure on a query for the
    error messages in there today).

    This change adds a specific error message when we fail the retry
    attempts and exit so we can fingerprint on this with an elastic recheck
    query.

    Related-Bug: #1365046

    Change-Id: Ibb3269a534d5b28b2d7d8a531e30a77ba6011f2d

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/142169

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/142169
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=7783356a20ae1a151f86d60599ccc19304e5da10
Submitter: Jenkins
Branch: master

commit 7783356a20ae1a151f86d60599ccc19304e5da10
Author: Mauro S. M. Rodrigues <email address hidden>
Date: Tue Dec 16 17:43:17 2014 +0000

    Update query for Bug #1365046

    The Change Id Ibb3269a534d5b28b2d7d8a531e30a77ba6011f2d added a more
    specific message which help us to cover this bug more accurately.

    Change-Id: I942c2884d75eb138e9646198266fad61d080c8f7
    Related-Bug: #1365046

Revision history for this message
Matt Riedemann (mriedem) wrote :

Might be inadvertently fixed.

Changed in openstack-gate:
status: Confirmed → Incomplete
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

I've seen this bug in review https://review.openstack.org/#/c/187571/6

Elastic Recheck recognized it as bug 1455123

Changed in openstack-gate:
status: Incomplete → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.