requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",)) in functional tests

[Matt Riedemann]

We see this about 11 times in 7 days in the check and gate queues on the nova functional jobs:

http://logs.openstack.org/20/459420/1/check/gate-nova-tox-functional-ubuntu-xenial/c0bdfa6/console.html#_2017-04-25_13_55_17_705409

2017-04-25 13:55:17.704694 | Captured traceback:
2017-04-25 13:55:17.704705 | ~~~~~~~~~~~~~~~~~~~
2017-04-25 13:55:17.704720 | Traceback (most recent call last):
2017-04-25 13:55:17.704749 | File "nova/tests/functional/api_sample_tests/test_fixed_ips.py", line 100, in test_get_fixed_ip
2017-04-25 13:55:17.704763 | self._test_get_fixed_ip()
2017-04-25 13:55:17.704791 | File "nova/tests/functional/api_sample_tests/test_fixed_ips.py", line 91, in _test_get_fixed_ip
2017-04-25 13:55:17.704817 | response = self._do_get('os-fixed-ips/192.168.1.1')
2017-04-25 13:55:17.704844 | File "nova/tests/functional/api_samples_test_base.py", line 488, in _do_get
2017-04-25 13:55:17.704857 | headers=headers)
2017-04-25 13:55:17.704883 | File "nova/tests/functional/api_samples_test_base.py", line 479, in _get_response
2017-04-25 13:55:17.704901 | headers=headers, strip_version=strip_version)
2017-04-25 13:55:17.704923 | File "nova/tests/functional/api/client.py", line 164, in api_request
2017-04-25 13:55:17.704938 | auth_result = self._authenticate()
2017-04-25 13:55:17.704961 | File "nova/tests/functional/api/client.py", line 150, in _authenticate
2017-04-25 13:55:17.704972 | headers=headers)
2017-04-25 13:55:17.704994 | File "nova/tests/functional/api/client.py", line 138, in request
2017-04-25 13:55:17.705016 | response = requests.request(method, url, data=body, headers=_headers)
2017-04-25 13:55:17.705059 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
2017-04-25 13:55:17.705083 | return session.request(method=method, url=url, **kwargs)
2017-04-25 13:55:17.705163 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
2017-04-25 13:55:17.705202 | resp = self.send(prep, **send_kwargs)
2017-04-25 13:55:17.705280 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
2017-04-25 13:55:17.705321 | r = adapter.send(request, **kwargs)
2017-04-25 13:55:17.705367 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
2017-04-25 13:55:17.705384 | raise ConnectionError(err, request=request)
2017-04-25 13:55:17.705409 | requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Which API it fails on is completely random I think.

It looks like this is maybe a result of the OSAPIFixture spawning a greenthread to run the nova-api service:

(9:19:08 AM) cdent: if for some reason the new greenthread doesn't get yielded to when it should, the client will try to read off the socket which nothing paying proper attention on the other side

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22requests.exceptions.ConnectionError%3A%20('Connection%20aborted.'%2C%20BadStatusLine(%5C%22''%5C%22%2C))%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/459733

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/459735

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/459735
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4542650b239a35c4e21b0d7801670c3ebafc986f
Submitter: Jenkins
Branch: master

commit 4542650b239a35c4e21b0d7801670c3ebafc986f
Author: Sean Dague <email address hidden>
Date: Tue Apr 25 10:45:05 2017 -0400

    disable keepalive for functional tests

    We believe that some of the current race failures in the gate are
    because of eventlet vs. requests for our full stack in a process
    testing. One of the ways to make this better is turn off keepalive on
    eventlet.

    Related-Bug: #1686109

    Change-Id: Ibdb29274cdf26de74f4ca0cc3bdab622299a91bf

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Dent (<email address hidden>) on branch: master
Review: https://review.openstack.org/459733
Reason: This has proved its point (that eventlet in the tests is fraught)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/483972

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483972
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d812c7eb3c7f0ce80c3548e9aec10b9fdc650a67
Submitter: Jenkins
Branch: master

commit d812c7eb3c7f0ce80c3548e9aec10b9fdc650a67
Author: Matt Riedemann <email address hidden>
Date: Fri Jul 14 11:40:13 2017 -0400

    Set wsgi.keep_alive=False globally for tests

    This was previously set for all functional API samples
    tests to try and fix a weird issue where requests and responses
    seemed to get mixed up between the compute API wsgi service
    fixture used in the API samples functional tests and the
    Placement API functional tests.

    However, we're also seeing these weird mixup issues in
    non-API samples tests, and there are other wsgi server fixtures
    that should be setting this, like the OSMetadataServer
    fixture. And there are functional tests that don't inherit from
    _IntegratedTestBase so we should just set this globally to be
    safe.

    Related-Bug: #1686109
    Related-Bug: #1696834

    Change-Id: Ie307e4d43be4b3e40dd75384ea6bb77598811713