test_app_using_ipv6_and_ssl fails with requests 2.9.0

[Matt Riedemann]

http://logs.openstack.org/50/257950/1/check/gate-nova-python27/6dfe685/testr_results.html.gz

ft1.13019: nova.tests.unit.test_wsgi.TestWSGIServerWithSSL.test_app_using_ipv6_and_ssl_StringException: Empty attachments:
  stdout

pythonlogging:'': {{{
2015-12-15 15:55:06,381 WARNING [oslo_reports.guru_meditation_report] Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2015-12-15 15:55:06,383 INFO [nova.wsgi] fake_ssl listening on ::1:54837
2015-12-15 15:55:06,390 INFO [nova.fake_ssl.wsgi.server] (6121) wsgi starting up on https://::1:54837/
}}}

stderr: {{{
/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
}}}

Traceback (most recent call last):
  File "nova/tests/unit/test_wsgi.py", line 333, in test_app_using_ipv6_and_ssl
    verify=os.path.join(SSL_CERT_DIR, 'ca.crt'))
  File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: hostname '[::1]' doesn't match either of 'localhost', 'ip6-localhost', '127.0.0.1', '::1'

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22requests.exceptions.SSLError:%20hostname%20'%5B::1%5D'%20doesn't%20match%20either%20of%20'localhost',%20'ip6-localhost',%20'127.0.0.1',%20'::1'%5C%22%20AND%20tags:%5C%22console%5C%22

just started today when requests 2.9.0 was released:

https://pypi.python.org/pypi/requests/2.9.0

[Matt Riedemann (mriedem)]

There is a warning in the logs too:

http://logs.openstack.org/50/257950/1/check/gate-nova-python27/6dfe685/console.html#_2015-12-15_15_55_06_478

2015-12-15 15:55:06.478 | Captured stderr:
2015-12-15 15:55:06.479 | ~~~~~~~~~~~~~~~~
2015-12-15 15:55:06.479 | /home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
2015-12-15 15:55:06.479 | SNIMissingWarning
2015-12-15 15:55:06.479 | /home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
2015-12-15 15:55:06.479 | InsecurePlatformWarning

[Matt Riedemann (mriedem)]

There might be a problem in here:

https://github.com/kennethreitz/requests/commit/0352ec0bd3aabb131b0cbb649f2062d59fc8f4a1

[Cory Benfield (lukasaoz)]

Sorry folks, I believe the issue is this one: https://github.com/shazow/urllib3/issues/760

[Matt Riedemann (mriedem)]

This is the requests issue for tracking:

https://github.com/shazow/urllib3/issues/760

It's the move to urllib3 1.13.

[Cory Benfield (lukasaoz)]

It would be convenient for our sake if you could provide us a copy of the cert you were trying to validate. I'd like to add it to our regression tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/258001

[Cory Benfield (lukasaoz)]

Nevermind, I minted some appropriate certificates myself. A proposed fix has been made on urllib3.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/258001
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ba82e4a875f01bb883deb78517953989b690a36c
Submitter: Jenkins
Branch: master

commit ba82e4a875f01bb883deb78517953989b690a36c
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 15 08:38:17 2015 -0800

    Block requests 2.9.0

    Breaks the wsgi IPv6 unit tests.

    Depends-On: Ifb080c72a16ed4b93040fb7c06bdc072b7247f6e

    Change-Id: I158bd1cba9bc85521ac0e236f66e7780a59e9ac2
    Closes-Bug: #1526413

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/258599

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/258599
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=94d6b692d8d81e68ca7cf9e66e80adb03b8a88ef
Submitter: Jenkins
Branch: stable/liberty

commit 94d6b692d8d81e68ca7cf9e66e80adb03b8a88ef
Author: Mike Bayer <email address hidden>
Date: Sat Dec 5 22:34:17 2015 -0500

    Omnibus stable fix for upstream requirements breaks

    (Both of these patches have to be merged at the same time, because
    there are multiple upstream contract breaks affecting us at the same
    time)

    Item #1:

    Remove SQLite BigInteger/Integer translation logic

    This patch removes code that was causing the
    SQLAlchemy BigInteger type to render as INTEGER
    on the SQLite backend only. The comment
    in create_shadow_table() in nova/db/sqlalchemy/utils.py
    which states that BigInteger isn't supported by SQLite
    is not correct. SQLite has an "affinity"-based,
    dynamic typing model [1] that allows virtually any
    name to be used for a type; datatypes are actually
    determined on a row-by-row basis, but any
    type name that has the substring INT inside of it
    is automatically given integer affinity by default,
    so there is no limitation on this type.

    There is one difference on SQLite between INTEGER
    and BIGINT, and that is that SQLite's "rowid"
    feature only takes effect for a primary key column
    if it is named "INTEGER PRIMARY KEY" [2]. However,
    as all tests pass here it does not seem to be the
    case that we are using BigInteger primary keys
    in tests, and if we were, I'd rather approach SQLite's
    quirk here in a different way.

    Note that this patch relies upon the assumption
    that **SQLite is only used in Nova as a test database,
    not in actual deployments**. If this statement is
    not true, then information on these other use cases
    will be needed.

    Due to Alembic issue #341 [3] which has been resolved
    for version 0.8.4 [4], the comparison of
    INTEGER/BIGINT/SMALLINT is now significant by default,
    so this change (or alternatively, one that changes Nova's
    type comparison rules within migration tests) is needed
    in order for Nova to pass the test_migrations test; the
    error condition that raises with the latest Alembic is
    shown in [5].

    [1] http://sqlite.org/datatype3.html

    [2] http://sqlite.org/lang_createtable.html#rowid

    [3] https://bitbucket.org/zzzeek/alembic/issues/341/integer-vs-biginteger-diffrence-are-not

    [4] http://alembic.readthedocs.org/en/latest/changelog.html#change-f8a974100f54b74d7f90ef765edae1ed

    [5] http://paste.openstack.org/show/480966/

    (cherry picked from commit 793f4b7c711236f50dbebd8739f08604bff0af14)

    Item #2:

    Block requests 2.9.0

    Breaks the wsgi IPv6 unit tests.

    Depends-On: Ifb080c72a16ed4b93040fb7c06bdc072b7247f6e

    Closes-Bug: #1526413
    (cherry picked from commit ba82e4a875f01bb883deb78517953989b690a36c)

    Change-Id: Ia9f2d403e644651b6cd9bbf257ed94d257b2a1db

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.1 release.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

[Victor Morales (electrocucaracha)]

I'm using requests=2.9.1 and I'm getting the following error, I'm wondering if this needs to be changed for excluding all 2.9.x versions

2016-02-18 12:05:47,487 INFO [nova.fake_ssl.wsgi.server] (20672) wsgi starting up on https://[::1]:36814
}}}

Traceback (most recent call last):
  File "nova/tests/unit/test_wsgi.py", line 324, in test_app_using_ipv6_and_ssl
    verify=os.path.join(SSL_CERT_DIR, 'ca.crt'))
  File "/home/vjmorale/src/vagrant-intel-devstack/stack/nova/.tox/cover/lib/python2.7/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/home/vjmorale/src/vagrant-intel-devstack/stack/nova/.tox/cover/lib/python2.7/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/vjmorale/src/vagrant-intel-devstack/stack/nova/.tox/cover/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vjmorale/src/vagrant-intel-devstack/stack/nova/.tox/cover/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/home/vjmorale/src/vagrant-intel-devstack/stack/nova/.tox/cover/lib/python2.7/site-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:590)
Ran 4 tests in 10.466s (+7.304s)

[Cory Benfield (lukasaoz)]

Victor, your issue is unrelated to this one. This issue was caused by errors with validating host names when using IPv6 addresses, which can only manifest itself by host name validation errors. The error you're seeing is caused by the server aborting the TLS handshake, likely because of an incompatibility in client and server settings.

There is no reason to blame this on requests 2.9.1 as far as I can see.