list operations fail using Python binding with object of type 'NoneType' has no len()

[Gavin B]

A recent change in python-neutronclient has led to simple operations such as a neutron net-list failing when using the python binding (the CLI operation works OK).

(test_venv2)cetest@cer405n0046:~/gavin$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from os import environ
>>> from neutronclient.v2_0.client import Client as NeutronClient
>>> nc = NeutronClient(username=environ['OS_USERNAME'], password=environ['OS_PASSWORD'], tenant_name=environ['OS_TENANT_NAME'], auth_url=environ['OS_AUTH_URL'], ca_cert=environ.get('OS_CACERT', None))
>>> nc.list_networks()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 568, in list_networks
    **_params)
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 315, in list
    for r in self._pagination(collection, path, **params):
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 328, in _pagination
    res = self.get(path, params=params)
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 301, in get
    headers=headers, params=params)
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 278, in retry_request
    headers=headers, params=params)
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 201, in do_request
    self._check_uri_length(action)
  File "/home/cetest/gavin/test_venv2/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 188, in _check_uri_length
    uri_len = len(self.httpclient.endpoint_url) + len(action)
TypeError: object of type 'NoneType' has no len()
>>>
(test_venv2)cetest@cer405n0046:~/gavin$ pip freeze | grep neutron
python-neutronclient==2.3.11.post21

We have seen the problem with python-neutronclient==2.3.9.46.gde7bb1e

But not with python-neutronclient==2.3.9.31.g64b2d8a

Other similar list operations fail the same way.

[Gavin B (gavin-brebner-orange)]

Virtual environment where the only thing I change is the python-neutronclient :

(toto)cetest@cer405n0046:~/gavin$ python repro_neutron.py
Traceback (most recent call last):
  File "repro_neutron.py", line 13, in <module>
    nc.list_networks()
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 568, in list_networks
    **_params)
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 315, in list
    for r in self._pagination(collection, path, **params):
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 328, in _pagination
    res = self.get(path, params=params)
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 301, in get
    headers=headers, params=params)
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 278, in retry_request
    headers=headers, params=params)
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 201, in do_request
    self._check_uri_length(action)
  File "/home/cetest/gavin/toto/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 188, in _check_uri_length
    uri_len = len(self.httpclient.endpoint_url) + len(action)
TypeError: object of type 'NoneType' has no len()
(toto)cetest@cer405n0046:~/gavin$ pip freeze | grep neutron
python-neutronclient==2.3.11.post24
(toto)cetest@cer405n0046:~/gavin$ pip uninstall python-neutronclient
Uninstalling python-neutronclient:
  /home/cetest/gavin/toto/bin/neutron
  /home/cetest/gavin/toto/lib/python2.7/site-packages/neutronclient
  /home/cetest/gavin/toto/lib/python2.7/site-packages/python_neutronclient-2.3.11.post24-py2.7.egg-info
Proceed (y/n)? y
  Successfully uninstalled python-neutronclient
(toto)cetest@cer405n0046:~/gavin$ pip install python-neutronclient
Downloading/unpacking python-neutronclient
  http://10.22.157.17/pypi/latest/python-neutronclient/ uses an insecure transport scheme (http). Consider using https if 10.22.157.17 has it available
  http://pypi.python.org/simple/python-neutronclient/ uses an insecure transport scheme (http). Consider using https if pypi.python.org has it available
  Downloading python_neutronclient-2.3.11-py2.py3-none-any.whl (199kB): 199kB downloaded
Requirement already satisfied (use --upgrade to upgrade): iso8601>=0.1.9 in ./toto/lib/python2.7/site-packages (from python-neutronclient)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.2.0,!=2.4.0 in ./toto/lib/python2.7/site-packages (from python-neutronclient)
Requirement already satisfied (use --upgrade to upgrade): cliff>=1.7.0 in ./toto/lib/python2.7/site-packages (from python-neutronclient)
Requirement already satisfied (use --upgrade to upgrade): oslo.serialization>=1.2.0 in ./toto/lib/python2.7/site-packages (from python-neutronclient)
Requirement already satisfied (use --upgrade t...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/164456

[Akihiro Motoki (amotoki)]

This is a critical regression in commit 4b6ed760d4303744907feefd81e60f38ae3750ef.
It prevents all library usage of neutronclient without passing endpoint_url and token.
(Note that Horizon is not affected as Horizon passes endpoint_url and token when initializing neutronclient object)
IMHO we should revert the change first and then explore a better way to check max_uri_length again.

[Akihiro Motoki (amotoki)]

After more investigation, the situation is no so simple.

- Before SessionClient support, check_max_uri() worked as expected.
- After SessionClient support was introduced, it seems that check_max_uri() did not work (bug 1422736)
- Then the fix for bug 1422736 ( https://review.openstack.org/#/c/160923/ ) introduced this bug.

As you can see, just reverting some patch does not solve the problem.

Apart from the solution, our test framework for the Client class is not sufficient. This failure comes from the fact most neutron client tests mock out "Client" class and we don't have good unit tests for Client class. Most tests are done as a part of CLI tests unfortunately.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/164456
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=5f0f2802a83646dc4cad29078e2c8e82c69dc041
Submitter: Jenkins
Branch: master

commit 5f0f2802a83646dc4cad29078e2c8e82c69dc041
Author: Anand Shanmugam <email address hidden>
Date: Sat Mar 14 23:47:13 2015 +0530

    Fix failures when calling list operations using Python binding

    This bug is caused because of the regression caused by
    I1b719bed406b83c5f2deac06e127798a91f51ad7 . The original bug was
    raised because check_max_uri was not working after the introduction
    of sessionclient. The change created a regression in which the
    python bindings of the neutron client was not usable and causing
    traces.

    The fix is to revert the change id and add a new check_max_uri to
    the sessionclient. Now uri length will be checked in sessionclient
    and httpclient as well. please see bug for further info.

    Closes-Bug: #1431449

    Change-Id: Ief2352a90bb75a76e8c671d51beb0fb7a53a22f9