Network problems in CI environment

[Ben Nemec]

We seem to be having intermittent networking glitches in the CI environment. This is manifesting as undercloud installs that fail cloning the puppet modules with messages like:

fatal: unable to access 'https://github.com/puppetlabs/puppetlabs-concat.git/': Could not resolve host: github.com; Name or service not known

or

error: RPC failed; result=6, HTTP code = 0
fatal: The remote end hung up unexpectedly

I can't logstash on this right now (although I pushed a CI change that would allow us to: https://review.openstack.org/302999), but it appears to be happening frequently. If you see a job that failed in between 15 and 25 minutes chances are it's this. I've also seen image builds fail in similar ways, so there are probably some longer failed jobs that are also being hit by this.

Just eyeballing the status page, I'd say half of our jobs are failing on something related to this right now, so I'm calling it critical.

[Ben Nemec (bnemec)]

I should note that I don't think this is just github flakiness. The DNS lookup failure happens before we even try to talk to github.

[Ben Nemec (bnemec)]

Can probably add

2016-04-07 17:55:42.960 | Slave went offline during the build
2016-04-07 17:55:42.960 | ERROR: Connection was broken: java.io.IOException: Unexpected termination of the channel

to the list of symptoms.

[John Trowbridge (trown)]

Setting alert on this, status page is almost all red.

[Derek Higgins (derekh)]

I noticed last night that nodepool seems to have leaked all of its floating IP's, after this it started constantly starting new instances and deleting them again (after they fail to get a floating ip), I deleted all the free and unused floating IP's from nodepools account that that seems to have settled things down(as they are now available to new instances).

We also currently have a problem with a commit to the mongodb puppet modules, once that is fixed we'll know for sure if the network problem is now solved.

[Emilien Macchi (emilienm)]

Looking at http://tripleo.org/cistatus.html today, a lot of green.

* mongodb patch has been reverted, we don't have the bug anymore.
* nodepool issue seems fixed by Derek.

Dropping the tag for now.

[Sagi (Sergey) Shnaidman (sshnaidm)]

A lot of jobs failed because of delorean build fail:

12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras: (28, 'Connection timed out after 30000 milliseconds')

https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-ha/1068/
http://logs.openstack.org/39/305039/7/check-tripleo/gate-tripleo-ci-f22-ha/c547c90

[Derek Higgins (derekh)]

Closing this, as we havn't had seen these errors in some time.

Closing this bug, please feel free to reopen it if you disagree.