git clone fails with "fatal: Not a git repository", "git remote update failed."

Bug #1282876 reported by Isaku Yamahata on 2014-02-21
46
This bug affects 10 people
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Fix Released
Critical
Jeremy Stanley
OpenStack-Gate
Critical
Jeremy Stanley

Bug Description

From the log
http://logs.openstack.org/20/73520/6/check/check-tempest-dsvm-neutron-isolated/e48eea7/logs/devstack-gate-setup-workspace-new.txt

The relevant error is

...
+ echo ' Need to clone nova'
  Need to clone nova
+ git clone https://git.openstack.org/openstack/nova
Cloning into 'nova'...
error: RPC failed; result=56, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
+ cd nova
/home/jenkins/workspace/check-tempest-dsvm-neutron-isolated/devstack-gate/functions.sh: line 98: cd: nova: No such file or directory
+ git_remote_set_url origin https://git.openstack.org/openstack/nova
+ git remote set-url origin https://git.openstack.org/openstack/nova
fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Dolph Mathews (dolph) wrote :

I accidentally filed a dupe of this in bug https://bugs.launchpad.net/openstack-ci/+bug/1283038 ... copying my description here--

Attempting to `git clone` nova in logs/devstack-gate-setup-workspace-new.txt [1] resulted in:

+ git clone https://git.openstack.org/openstack/nova
Cloning into 'nova'...
error: RPC failed; result=56, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
+ cd nova
/home/jenkins/workspace/gate-tempest-dsvm-large-ops/devstack-gate/functions.sh: line 98: cd: nova: No such file or directory
+ git_remote_set_url origin https://git.openstack.org/openstack/nova
+ git remote set-url origin https://git.openstack.org/openstack/nova
fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Which ultimately led to:

+ echo 'git remote update failed.'
git remote update failed.

[1]: http://logs.openstack.org/99/73699/11/gate/gate-tempest-dsvm-neutron/5d7df22/logs/devstack-gate-setup-workspace-new.txt

summary: - git fatal: Couldn't find remote ref
+ git clone fails with "fatal: Not a git repository", "git remote update
+ failed."
Dolph Mathews (dolph) wrote :

After talking to fungi, it appears that the "git fatal: Couldn't find remote ref" is actually just a red herring (and is unfortunately normal behavior in the logs).

Anita Kuno (anteaya) wrote :

The salient error is

fatal: Not a git repository (or any parent up to mount parent )

which comes from using https:// to clone

this was lately showing up trying to clone nova and neutron repositories.

Anita Kuno (anteaya) wrote :
Download full text (5.6 KiB)

Another manifestation of this bug: http://logs.openstack.org/88/75188/1/check/check-tempest-dsvm-full/cafeca7/console.html

2014-02-21 13:04:57.782 | Started by user anonymous
2014-02-21 13:04:57.787 | [EnvInject] - Loading node environment variables.
2014-02-21 13:05:01.068 | Building remotely on devstack-precise-rax-iad-1496017 in workspace /home/jenkins/workspace/check-tempest-dsvm-full
2014-02-21 13:05:12.170 | [check-tempest-dsvm-full] $ /bin/bash -xe /tmp/hudson1116277386280346131.sh
2014-02-21 13:05:13.410 | + [[ ! -e devstack-gate ]]
2014-02-21 13:05:13.413 | + git clone git://git.openstack.org/openstack-infra/devstack-gate
2014-02-21 13:05:13.762 | Cloning into 'devstack-gate'...
2014-02-21 13:05:14.637 | [check-tempest-dsvm-full] $ /bin/bash -xe /tmp/hudson8798132986219882664.sh
2014-02-21 13:05:14.682 | + export PYTHONUNBUFFERED=true
2014-02-21 13:05:14.683 | + PYTHONUNBUFFERED=true
2014-02-21 13:05:14.683 | + export DEVSTACK_GATE_TIMEOUT=180
2014-02-21 13:05:14.683 | + DEVSTACK_GATE_TIMEOUT=180
2014-02-21 13:05:14.683 | + export DEVSTACK_GATE_TEMPEST=1
2014-02-21 13:05:14.683 | + DEVSTACK_GATE_TEMPEST=1
2014-02-21 13:05:14.683 | + export DEVSTACK_GATE_TEMPEST_FULL=1
2014-02-21 13:05:14.683 | + DEVSTACK_GATE_TEMPEST_FULL=1
2014-02-21 13:05:14.684 | + export BRANCH_OVERRIDE=default
2014-02-21 13:05:14.684 | + BRANCH_OVERRIDE=default
2014-02-21 13:05:14.684 | + '[' default '!=' default ']'
2014-02-21 13:05:14.684 | + cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
2014-02-21 13:05:14.685 | + ./safe-devstack-vm-gate-wrap.sh
2014-02-21 13:05:14.685 | Triggered by: https://review.openstack.org/75188 patchset 1
2014-02-21 13:05:14.685 | Pipeline: check
2014-02-21 13:05:14.686 | IP configuration of this host:
2014-02-21 13:05:14.686 | 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
2014-02-21 13:05:14.686 | link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2014-02-21 13:05:14.687 | inet 127.0.0.1/8 scope host lo
2014-02-21 13:05:14.687 | inet6 ::1/128 scope host
2014-02-21 13:05:14.687 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.687 | 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
2014-02-21 13:05:14.687 | link/ether bc:76:4e:20:40:a5 brd ff:ff:ff:ff:ff:ff
2014-02-21 13:05:14.687 | inet 162.242.239.57/24 brd 162.242.239.255 scope global eth0
2014-02-21 13:05:14.688 | inet6 2001:4802:7802:102:d43e:b81a:ff20:40a5/64 scope global
2014-02-21 13:05:14.688 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.688 | inet6 fe80::be76:4eff:fe20:40a5/64 scope link
2014-02-21 13:05:14.688 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.688 | 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
2014-02-21 13:05:14.688 | link/ether bc:76:4e:20:4e:69 brd ff:ff:ff:ff:ff:ff
2014-02-21 13:05:14.689 | inet 10.208.166.253/19 brd 10.208.191.255 scope global eth1
2014-02-21 13:05:14.689 | inet6 fe80::be76:4eff:fe20:4e69/64 scope link
2014-02-21 13:05:14.689 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.689 | ...

Read more...

Jeremy Stanley (fungi) on 2014-02-21
description: updated
Changed in openstack-ci:
status: New → Fix Released
importance: Undecided → Critical
assignee: nobody → Jeremy Stanley (fungi)
milestone: none → icehouse
Jeremy Stanley (fungi) wrote :

The underlying error is that some nodepool server images in some provider regions failed to pre-cache some repositories when they were rebuilt overnight (it's possible they attempted to clone them while the git servers were running a repack, which shouldn't cause a problem, but maybe it does). Because servers launched from these images were missing some repositories, DevStack attempted to reclone them when running in jobs leading to a fairly substantial denial of service against the git servers and causing many of those clone attempts to fail, breaking the jobs performing them.

The images in question have all been deleted and replaced at this point along with the nodes built from them. The scripts which perform git repository caching on nodepool images have also been improved so that they'll now retry once on a failure, perform some basic tests within each clone, and abort the image build if issues are encountered.

Reviewed: https://review.openstack.org/75542
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=0f89211d45c9eb1fbcd7ffafa79549753b7d38c0
Submitter: Jenkins
Branch: master

commit 0f89211d45c9eb1fbcd7ffafa79549753b7d38c0
Author: Joe Gordon <email address hidden>
Date: Fri Feb 21 15:53:11 2014 -0800

    Add fingerprint for bug 1282876

    This is an infra bug that can manifest itself in several ways, so this
    fingerprint won't catch all cases. But a fingerprint that only catches a
    subset is better then no fingerprint at all.

    Change-Id: I80d62f500407582b7bd5825943775d4cb86209df
    Related-Bug: #1282876

Jeremy Stanley (fungi) on 2014-12-04
Changed in openstack-gate:
status: New → Fix Released
importance: Undecided → Critical
assignee: nobody → Jeremy Stanley (fungi)

Reviewed: https://review.openstack.org/178220
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=a4485a992c630d300cd010b436b5fb9dc80db5fd
Submitter: Jenkins
Branch: master

commit a4485a992c630d300cd010b436b5fb9dc80db5fd
Author: Sean Dague <email address hidden>
Date: Tue Apr 28 10:45:01 2015 -0400

    Remove tripleo from signature

    The tripleo jobs fail a lot because they are talking to github,
    exclude that queue from this signature which is about our servers
    failing.

    Long term we should exclude everything from the tripleo queue.

    Change-Id: I45c4f86147a04522b026f2f677cab7eb72a867d5
    Related-Bug: #1282876

Reviewed: https://review.openstack.org/178608
Committed: https://git.openstack.org/cgit/openstack-infra/project-config/commit/?id=8039e854aae833f5e62c3f63d1fa5be508c8fcaf
Submitter: Jenkins
Branch: master

commit 8039e854aae833f5e62c3f63d1fa5be508c8fcaf
Author: Sean Dague <email address hidden>
Date: Wed Apr 29 06:56:55 2015 -0400

    fix git_timed retry logic

    git_timed retry logic actually only retried *if* we timed out, not if
    the command itself failed, as it would explicitly exit if the return
    code was anything except 124. This loops in all failure cases, which
    was the intent of the original fix.

    Related-Bug: #1282876

    Change-Id: I576df4832665beb21992ffa873da28a198018530

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers