unit test jobs sometimes time out

[Colleen Murphy]

Both the py27 and py35 unit test jobs sometimes time out, with no obvious reason. For example:

http://logs.openstack.org/33/530133/13/check/openstack-tox-py27/d6560ad/
http://logs.openstack.org/23/524423/44/gate/openstack-tox-py35/0f8083f/
http://logs.openstack.org/23/524423/44/check/openstack-tox-py35/ea6fcfe/

There's an empty space of time between the output of a test and the message that the job has timed out, so it's unclear if it's hanging on a particular unknown test or if an infrastructure issue has caused the test runner to stop for some reason.

[Colleen Murphy (krinkle)]

Another timeout, but with different symptoms, not sure if related: http://logs.openstack.org/44/533744/18/check/openstack-tox-py27/f5ec5ac/job-output.txt.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/539351

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539351
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=45e8b6e0ece1a1d4be5ded84438d772d5369e515
Submitter: Zuul
Branch: master

commit 45e8b6e0ece1a1d4be5ded84438d772d5369e515
Author: Colleen Murphy <email address hidden>
Date: Tue Jan 30 23:45:55 2018 +0100

    Increase MySQL max_connections for unit tests

    We've seen that sometimes our unit tests create a lot of mysql
    connections simultaneously[1][2] and cause the tests to fail. Increase
    the max_connections limit from 151 to 1024, which is what devstack
    uses[3].

    [1] http://logs.openstack.org/43/536543/9/check/openstack-tox-py35/0729ea7/job-output.txt.gz#_2018-01-30_19_36_38_136523
    [2] http://logs.openstack.org/23/524423/44/gate/openstack-tox-py35/0f8083f/job-output.txt.gz#_2018-01-28_21_11_52_855308
    [3] http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/databases/mysql?id=640150683d137a9a10dfe833e7a34b7ea6cdfb1a#n99

    Change-Id: Ic9f9ca696af1876eb5a728356a98d70352ef9bc3
    Related-bug: #1746016

[Colleen Murphy (krinkle)]

I haven't seen this happen in a while, and at the moment I suspect it was related to infrastructure problems discussed here: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-01-31.log.html#t2018-01-31T16:43:42

Marking this invalid for now, we can reopen if necessary.