Timeouts when requesting a glance image created with a remote image from git.o.o

[Matthew Treinish]

After bringing the bluebox cloud back online recently we're starting to see a spike in failures on tempest tests that ask glance to pull an image over http. In the gate we default to using the git.o.o for this since it needs to be rock solid anyway for our jobs. However, there are connection issues now as seen by:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22The%20HTTP%20URL%20is%20invalid%3A%20HTTPConnectionPool(host%3D'git.openstack.org'%2C%20port%3D80)%3A%20Max%20retries%20exceeded%20with%20url%3A%20%2Fstatic%2Fopenstack.png%20(Caused%20by%20NewConnectionError('%3A%20Failed%20to%20establish%20a%20new%20connection%3A%20%5BErrno%20-2%5D%20Name%20or%20service%20not%20known'%2C))%5C%22

http://status.openstack.org/openstack-health/#/test/tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image?end=2016-08-16T15:00:00.000Z

[Matthew Treinish (treinish)]

This also failing on osic nodes with a slightly different error message:

http://logs.openstack.org/35/348535/5/check/gate-grenade-dsvm-designate-powerdns/e7746e5/console.html#_2016-08-15_22_29_38_525987

[Matthew Treinish (treinish)]

There is a working theory that this is not network related but instead node speed related and a recent keystone patch which resulted in way to much logging is eating too much of the available resources and causing weird failures on the slower nodes that osic and bluebox provide. There is a keystone revert in progress:

https://review.openstack.org/#/c/356010/

If we see a drop off on failures after that lands, then it's the likely culprit and I'll retarget this against keystone and mark it as closed

[Ken Giusti (kgiusti)]

Same error as Matthew for oslo.messaging:

http://logs.openstack.org/90/349290/3/check/gate-oslo.messaging-src-dsvm-full-zmq/dd1de25/console.html#_2016-08-16_11_23_11_817870

Been failing since the 15th:

http://status.openstack.org/openstack-health/#/job/gate-oslo.messaging-src-dsvm-full-zmq

Note the test appears to run fine on non-osic nodes.

[Matthew Treinish (treinish)]

So after looking some more this is isolated to osic and only on trusty nodes. The bluebox failure was just a temporary network issue and doesn't look to be a systemic thing. It's likely related to the osic move to ipv6

[Matt Riedemann (mriedem)]

This doesn't appear to be an issue anymore so closing it as invalid.