ssl_adapter.py doesn't work on RDO

[Martin Millnert]

A system running RDO Kilo (on EL 7) has the following two relevant packages installed:
 python-requests --> /usr/lib/python2.7/site-packages/requests
 python-urllib3 --> /usr/lib/python2.7/site-packages/urllib3

The SSL Adapter in contrail-controller/src/config/common/ssl_adapter.py gets installed as /usr/lib/python2.7/site-packages/cfgm_common/ssl_adapter.py .
When using SSL against Keystone v3 from the contrail-api, that uses this code, on a system such as the above, there will be a class loader type bug, due to a incorrect order of imports in ssl_adapter.py.

First, the bug, using the attached test_case.py, with some added debugging within /usr/lib/python2.7/site-packages/urllib3/connectionpool.py (also attached in next comment):

[root@dev14 ~]# ./testcase.py
('1: ', <bound method apa.openks of <__main__.apa object at 0x261ba50>>)
('minapa._ks: ', None)
('2: ', <bound method apa.openks of <__main__.apa object at 0x261ba50>>)
('minapa._ks: ', None)
'openks definition'
('retval: ', None)
('minapa._ks: ', <requests.sessions.Session object at 0x261ba90>)
HTTP get
xxx: connectionpool: timeout: Timeout(connect=3, read=3, total=None)
xxx: connectionpool TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
xxx: connectionpool: timeout: Timeout(connect=3, read=3, total=None)
xxx: connectionpool TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
<Response [200]>
HTTPS get
xxx: connectionpool: timeout: Timeout(connect=7, read=7, total=None)
xxx: connectionpool TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=7, read=7, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'urllib3.util.timeout.Timeout'>
Traceback (most recent call last):
  File "./testcase.py", line 46, in <module>
    main()
  File "./testcase.py", line 41, in main
    resp = minapa._ks.get("https://google.com", timeout=7, verify=False)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 478, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 466, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 574, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 542, in urlopen
    timeout_obj = self._get_timeout(timeout)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 304, in _get_timeout
    return Timeout.from_float(timeout)
  File "/usr/lib/python2.7/site-packages/urllib3/util/timeout.py", line 153, in from_float
    return Timeout(read=timeout, connect=timeout)
  File "/usr/lib/python2.7/site-packages/urllib3/util/timeout.py", line 95, in __init__
    self._connect = self._validate_timeout(connect, 'connect')
  File "/usr/lib/python2.7/site-packages/urllib3/util/timeout.py", line 126, in _validate_timeout
    "int or float." % (name, value))
ValueError: Timeout value connect was Timeout(connect=7, read=7, total=None), but it must be an int or float.

From the debugging, we see that the code around https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L297 is evaluating a locally instantiated Timeout object, per the import on https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L47 , to be loaded via the /usr/lib/python2.7/site-packages/urllib3 path rather than through the symlink within python-requests ( /usr/lib/python2.7/site-packages/requests/packages/urllib3 ).

This happens due to incorrect order of import statements in contrail-controller/src/config/common/ssl_adapter.py
This is unfixed in master (thereby, by policy, in all versions?)

A patch will be submitted shortly.

[Martin Millnert (r-martin-5)]

/usr/lib/python2.7/site-packages/urllib3/connectionpool.py with some simple debug lines to demonstrate the error more clearly

[Martin Millnert (r-martin-5)]

The fix is the following ssl_adapter.py, also being submitted in git-review.
Additionally, PEP8 fixed the source code.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15996
Submitter: Martin Millnert (<email address hidden>)

[Martin Millnert (r-martin-5)]

This may be moot considering contrail-controller/src/config/vnc_openstack/vnc_openstack/__init__.py has moved away from using python-requests natively for keystone v3, into using keystone client, which alleviates this issue.

The fix however is applicable for R2.22-dev, any other branches that intends to support SSL but don't use the keystoneclient for the keystone v3 client operations.

In that case it should also be noted, that the code at https://github.com/Juniper/contrail-controller/blob/R2.22-dev/src/config/vnc_openstack/vnc_openstack/__init__.py#L360 is incorrect python code, as per the below error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vnc_openstack/__init__.py", line 398, in _ksv3_process_request
    resp = self._ks.get(uri, headers, verify=False)
TypeError: get() takes exactly 2 arguments (4 given)

The code snippet:
           resp = self._ks.get(uri,headers,verify=False)
becomes a call as:
self._ks.get(self, uri, headers, verify=False)
And this does not match requests.get: http://docs.python-requests.org/en/latest/api/#main-interface
A quick fix is for example:
           resp = self._ks.get(url=uri, params=headers, verify=False)

[Siva Gurumurthy (sgurumurthy)]

Hi Sachin,
   Need your help to nail down this issue.
   A lot of reviews like this are failing at the ‘verify contrail’ stage with the error ‘No BGP configuration for self’.

== Contrail Control ==
supervisor-control: active
supervisor-control: ['contrail-control active pid 1984, uptime 0:11:44', 'contrail-control-nodemgr active pid 1983, uptime 0:11:44', 'contrail-dns active pid 1985, uptime 0:11:44', 'contrail-named active pid 1986, uptime 0:11:44', '']
contrail-control: DEFAULT/S.http_server_port not present
contrail-control initializing (No BGP configuration for self)pid 1984, uptime 0:11:44
contrail-control-nodemgr: DEFAULT/S.http_server_port not present

This is for the smlite kilo job.
Can you please take look at the failure?

Thanks,
Siva.

On 5/17/16, 6:59 AM, "opencontrail-ci-admin" <email address hidden> wrote:

>SMLite provisioning job has completed
>exit_code=-1
>
>Slave VM: ci-oc-slave-ubuntu14-10-84-34-150
>Workspace: /home/jenkins/workspace/ci-contrail-fabric-utils-systest-ubuntu14-kilo
>Review: https://review.opencontrail.org/20277
>Job: https://jenkins.opencontrail.org/job/ci-contrail-fabric-utils-systest-ubuntu14-kilo/12/
>
>

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15996
Committed: http://github.org/Juniper/contrail-controller/commit/cf6c7fa6f0092115bff9dff465a320255ca6b0b5
Submitter: Zuul
Branch: master

commit cf6c7fa6f0092115bff9dff465a320255ca6b0b5
Author: Martin Millnert <email address hidden>
Date: Wed Dec 23 14:19:48 2015 +0100

ssl_adapter.py: Properly fix order of imports

A system running RDO Kilo (on EL 7) has the following two relevant
packages installed:
python-requests --> /usr/lib/python2.7/site-packages/requests
python-urllib3 --> /usr/lib/python2.7/site-packages/urllib3

The SSL Adapter in contrail-controller/src/config/common/ssl_adapter.py
gets installed as
/usr/lib/python2.7/site-packages/cfgm_common/ssl_adapter.py .

When using SSL against Keystone v3 from the contrail-api, that uses this
code, on a system such as the above, there will be a class loader type
bug, due to a incorrect order of imports in ssl_adapter.py.

The intended ImportError doesn't happen, since the native urllib3
actually exists, which leads to a mixup of name space between
the Pool Manager and the requests code, when executing.

The proper fix is to reverse the order of the imports, and have the
fallback become the base case.
Any non-RDO system will trigger the ImportError, and load the module
the regular way. And an RDO-based system works, as well.

Change-Id: Iba275c3521b1cdde5595fa443b449fdd962286fb
Closes-Bug: 1528851

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21384
Submitter: Senthilnathan Murugappan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21384
Committed: http://github.org/Juniper/contrail-controller/commit/d7898b6033030be3f6b206ac6875f1cd6e60a9a8
Submitter: Zuul
Branch: R3.0

commit d7898b6033030be3f6b206ac6875f1cd6e60a9a8
Author: Martin Millnert <email address hidden>
Date: Wed Dec 23 14:19:48 2015 +0100

ssl_adapter.py: Properly fix order of imports

A system running RDO Kilo (on EL 7) has the following two relevant
packages installed:
python-requests --> /usr/lib/python2.7/site-packages/requests
python-urllib3 --> /usr/lib/python2.7/site-packages/urllib3

The SSL Adapter in contrail-controller/src/config/common/ssl_adapter.py
gets installed as
/usr/lib/python2.7/site-packages/cfgm_common/ssl_adapter.py .

When using SSL against Keystone v3 from the contrail-api, that uses this
code, on a system such as the above, there will be a class loader type
bug, due to a incorrect order of imports in ssl_adapter.py.

The intended ImportError doesn't happen, since the native urllib3
actually exists, which leads to a mixup of name space between
the Pool Manager and the requests code, when executing.

The proper fix is to reverse the order of the imports, and have the
fallback become the base case.
Any non-RDO system will trigger the ImportError, and load the module
the regular way. And an RDO-based system works, as well.

Change-Id: Iba275c3521b1cdde5595fa443b449fdd962286fb
Closes-Bug: 1528851