Frequent connection timeouts for git.openstack.org

[Ryan Hsu]

VMware Minesweeper has been frequently hitting connection time outs when trying to clone from git.openstack.org today. We are also seeing the problem on our own workstations suggesting this may be a problem with git.openstack.org.

The error message can be seen when trying to stack Devstack:

+ timeout -s SIGINT 0 git clone git://git.openstack.org/openstack/requirements.git /opt/stack/requirements
fatal: unable to connect to git.openstack.org:
git.openstack.org[0: 192.237.223.224]: errno=Connection timed out
git.openstack.org[1: 2001:4800:7813:516:3bc3:d7f6:ff04:aacb]:

[Clark Boylan (cboylan)]

How recently have you seen this issue? we added another node to the git.openstack.org load balancer pool late yesterday UTC time and that seemed to have helped mitigate this problem. We are also in the process of replacing the older existing nodes in the pool with newer more performant machines.

Getting concrete numbers for when you last saw this will help us track the problem.

[Clark Boylan (cboylan)]

Also you can see CPU use over time for the pool at http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=2. Note git05 is the new more performant node and it is currently receiving an equal share of the requests.

[Ryan Hsu (rhsu)]

We saw multiple nodes hit the problem around 1PM PST and things were intermittent for a good 2 hours. It seems to be okay at the moment but we are only running 2 builds currently. I'm continuing to monitor.

[James E. Blair (corvus)]

I'm assuming this hasn't been a significant problem since the git pool upgrade. If it is, please let us know.

(Separately, if you do a lot of testing, you may want to cache repositories locally like we do on the devstack-gate nodes.)

[Sebastien Dupont (dupont-sebastien)]

I've been hitting timeouts or awfully slow git clone over the past 2 months while launching devstack.

The git clones done by stack.sh are slow when using git.openstack.org (~30 minutes to clone neutron e.g.). Note that if I manually launch the git clone at the same time, it works fine (~30s for neutron).

Adjusting GIT_BASE to github in local.conf as suggested here helps: http://stackoverflow.com/questions/20390267/installing-openstack-errors

I'll try using a local cache for the repositories as mentioned in the parent post, but having stack.sh not working out of the box is a pity.