Spurious ECONNREFUSED errors during Glance unit tests

[Thierry Carrez]

Got weird errors in unit testing:

ERROR: glance.tests.functional.v2.test_images.TestImageDirectURLVisibility.test_image_direct_url_visible
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1
ConnectionError: [Errno 111] ECONNREFUSED
(See https://jenkins.openstack.org/job/gate-glance-python26/2868/consoleFull)

and

ERROR: glance.tests.functional.v1.test_api.TestApi.test_mismatched_size
error: [Errno 111] ECONNREFUSED
(https://jenkins.openstack.org/job/gate-glance-python27/2881/consoleFull)

[Clark Boylan (cboylan)]

This appears to potentially be happening due to two different races.

The first is in TCP port selection for the port given to the various test servers that are started for tested. These ports are determined by opening a socket on port 0, remembering the port it used, then closing the socket. This value is then passed to the server for use. It is possible that something else on the test system opens a socket on that port first.

The second potential race is that the functional tests appear to be exec'ing the test servers. Is it possible that the start_servers method is not waiting for the test servers to come up to the point that they are listening on the ports given to them?

The relavent code appears to begin at: https://github.com/openstack/glance/blob/master/glance/tests/functional/__init__.py#L628

[Jay Pipes (jaypipes)]

Those aren't unit tests, FYI... that's why they are in glance/tests/functional...

[Clark Boylan (cboylan)]

This appears to be affecting tempest-devstack-gate-vm-* jobs too. See http://logs.openstack.org/21690/6/gate/gate-tempest-devstack-vm-postgres-full/2585/

That leads me to think the potential port selection race I talked about above is not at fault here (as we shouldn't be attempting to use random ports when running glance in devstack).

[Clark Boylan (cboylan)]

http://paste.openstack.org/show/31354/ contains what appears to be the relavent pieces of syslog for the devstack failure related to glance client connection errors.

[Clark Boylan (cboylan)]

This appears to be related to https://bugs.launchpad.net/glance/+bug/1046612 and https://bugs.launchpad.net/glance/+bug/1052622. I didn't want to mark them all as duplicates as Brian appears to be tracking them separately.

[Clark Boylan (cboylan)]

Another devstack-gate job failed as a result of this bug: http://logs.openstack.org/22011/1/gate/gate-tempest-devstack-vm-full/2948/.

[Robert Collins (lifeless)]

I think this should be triaged to critical, as its breaking merges.

[Mark Washenberger (markwash)]

This was likely fixed by the following commit:

Reviewed: https://review.openstack.org/24866
Committed: http://github.com/openstack/glance/commit/6335fdb1d3efbbb6c3add7a2c31e279fc6e40ff3

[John Bresnahan (jbresnah)]

I have not seen this problem since the test suite re-factors. I believe it is no longer valid.