fatal: Could not read from remote repository

[Joe Gordon]

Query: message:"fatal: Could not read from remote repository." AND tags:"console.html"

http://logs.openstack.org/22/117922/10/gate/gate-tempest-dsvm-neutron-heat-slow/541580e/console.html#_2014-10-22_11_31_05_639

2014-10-22 11:31:05.639 | fatal: Could not read from remote repository.
2014-10-22 11:31:05.639 |
2014-10-22 11:31:05.639 | Please make sure you have the correct access rights
2014-10-22 11:31:05.639 | and the repository exists.

[Clark Boylan (cboylan)]

This appears to be a load issue related to having stale nodepool images. As the nodepool images get older and older the test slave git updates have a larger impact on the git mirrors essentially DDoSing the git.openstack.org cluster. We are working around this by fixing some problems in our image updates to have fresh images, but we should work on making the git cluster handle load more gracefully (more nodes, and maybe enable swap).

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/130305

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130305
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=d9e68570756e5bb1edc1da0ae84898830bd83e73
Submitter: Jenkins
Branch: master

commit d9e68570756e5bb1edc1da0ae84898830bd83e73
Author: Joe Gordon <email address hidden>
Date: Wed Oct 22 11:29:23 2014 -0700

    Add fingerprint for bug 1384373

    Bug 1384373 appears to be related to nodepool images.

    Change-Id: I7583fbd210ef48ff125a4c7d39859eb2e50148e4
    Related-Bug: #1384373

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608303
Committed: https://git.openstack.org/cgit/openstack-infra/zuul-jobs/commit/?id=b5ceee2004b58df989d3802268265b110a9b068b
Submitter: Zuul
Branch: master

commit b5ceee2004b58df989d3802268265b110a9b068b
Author: Clark Boylan <email address hidden>
Date: Fri Oct 5 08:52:51 2018 -0700

    Retry failed git pushses on workspace setup

    We've seen network/ssh failures to push one of the many git repos into
    the workspace. Followup pushes to other repos succeed. Considering that
    subsequent pushes do succeed allow for 3 retries to get these workspaces
    set up.

    Logs below show this happening. The first push of the repo under test
    fails (openstack/tripleo-heat-templates) then the push for openstack
    requirements succeeds.

      LOOP [mirror-workspace-git-repos : Synchronize src repos to workspace directory]
      Timeout exception waiting for the logger. Please check connectivity to [158.69.64.232:19885]
      ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host
      ubuntu-xenial -> localhost | fatal: Could not read from remote repository.
      ubuntu-xenial -> localhost |
      ubuntu-xenial -> localhost | Please make sure you have the correct access rights
      ubuntu-xenial -> localhost | and the repository exists.
      ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host
      ubuntu-xenial -> localhost | fatal: Could not read from remote repository.
      ubuntu-xenial -> localhost |
      ubuntu-xenial -> localhost | Please make sure you have the correct access rights
      ubuntu-xenial -> localhost | and the repository exists.
      ubuntu-xenial -> localhost | To git+ssh://zuul@158.69.64.232//home/zuul/src/git.openstack.org/openstack/requirements
      ubuntu-xenial -> localhost | 66b2988..740ae1a master -> master
      ubuntu-xenial -> localhost | * [new branch] stable/newton -> stable/newton
      ubuntu-xenial -> localhost | * [new branch] stable/ocata -> stable/ocata
      ubuntu-xenial -> localhost | * [new branch] stable/pike -> stable/pike
      ubuntu-xenial -> localhost | * [new branch] stable/queens -> stable/queens
      ubuntu-xenial -> localhost | * [new branch] stable/rocky -> stable/rocky
      ubuntu-xenial -> localhost | * [new branch] origin/HEAD -> origin/HEAD
      ubuntu-xenial -> localhost | * [new branch] origin/master -> origin/master
      ubuntu-xenial -> localhost | * [new branch] origin/stable/newton -> origin/stable/newton
      ubuntu-xenial -> localhost | * [new branch] origin/stable/ocata -> origin/stable/ocata
      ubuntu-xenial -> localhost | * [new branch] origin/stable/pike -> origin/stable/pike
      ubuntu-xenial -> localhost | * [new branch] origin/stable/queens -> origin/stable/queens
      ubuntu-xenial -> localhost | * [new branch] origin/stable/rocky -> origin/stable/rocky
      ubuntu-xenial -> localhost | ok: Item: Runtime: 0:00:01.828724
      ubuntu-xenial -> localhost | changed: All items complete

      ubuntu-xenial -> localhost | ssh: connect to ho...