JSONDecodeError happened in senlin-api when running a senlin command

[Haiwei Xu]

When running a senlin command like 'node-list' will got a 500 exception like below:

2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist
-packages/openstack/resource2.py", line 713, in list
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk params=query_params)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist
-packages/keystoneauth1/session.py", line 710, in get
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return self.request(url, 'GET', *
*kwargs)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist
-packages/openstack/session.py", line 64, in map_exceptions_wrapper
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return func(*args, **kwargs)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 367, in request
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return super(Session, self).request(*args, **kwargs)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return wrapped(*args, **kwargs)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 489, in request
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk **endpoint_filter)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 353, in get_endpoint
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk endpoint = self._get_endpoint_ver
sions(service_type, sc_endpoint)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 234, in _get_endpoint_versions
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk response = self._parse_versions_response(just_root + path)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 168, in _parse_versions_response
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk response = self.get(uri)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 710, in get
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return self.request(url, 'GET', **kwargs)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 77, in map_exceptions_wrapper
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk body = e.response.json()
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 826, in json
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return complexjson.loads(self.text, **kwargs)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 516, in loads
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return _default_decoder.decode(s)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 374, in decode
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk obj, end = self.raw_decode(s)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 404, in raw_decode
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk return self.scan_once(s, idx=_w(s, idx).end())
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2016-11-08 03:48:57.088 TRACE senlin.drivers.openstack.sdk

Not sure this is senlin's problem

[XueFeng Liu (jonnary-liu)]

This problem can be easily reproduced.
Step:
1.reference install guide:
http://docs.openstack.org/developer/senlin/install.html
2.Download DevStack:
$ git clone https://git.openstack.org/openstack-dev/devstack
$ cd devstack

3.Add following repo as external repositories into local.conf file
[[local|localrc]]
#Enable senlin
enable_plugin senlin https://git.openstack.org/openstack/senlin
#Enable senlin-dashboard
enable_plugin senlin-dashboard https://git.openstack.org/openstack/senlin-dashboard

4.Run ./stack.sh:
$ ./stack.sh

5.Wait install result
If install success.Then run senlin-client command:

6.But all command will raise HttpException
openstack cluster build info
HttpException: Expecting value: line 1 column 1 (char 0)

[XueFeng Liu (jonnary-liu)]

And some lib's version info:
root@tecs:/home/openstack/senlin# pip search openstacksdk
...
openstacksdk-hubic (0.2) - HubiC authentication interface compatible with the Python OpenStack SDK
openstacksdk-plugin (0.3.0) - OpenStack SDK example plugin
python-openstacksdk (0.5.2) - Deprecated SDK for OpenStack. Use openstacksdk.
openstacksdk (0.9.9) - An SDK for building applications to work with OpenStack
  INSTALLED: 0.9.9 (latest)
rackcli (0.1.1)
root@tecs:/home/openstack/senlin# pip search senlinclient
...
python-senlinclient (1.0.0) - OpenStack Clustering API Client Library
  INSTALLED: 1.0.0 (latest)

[XueFeng Liu (jonnary-liu)]

update python-openstacksdk and install python-openstacksdk newest version.
Issue is still in.

root@tecs:/home/cp /usr/local/lib/python2.7/dist-packages/openstack /usr/local/lib/python2.7/dist-packages/openstack_bak
root@tecs:/home/cd /home/openstack/python-openstacksdk
root@tecs:/home/openstack/python-openstacksdk#git remote update
root@tecs:/home/openstack/python-openstacksdk#git checkout master
root@tecs:/home/openstack/python-openstacksdk#git pull --ff-only origin master
root@tecs:/home/openstack/python-openstacksdk#python-openstacksdk# python setup.py install

Maybe we need research this to maker users install senlin unblocked:)

[XueFeng Liu (jonnary-liu)]

A temporary solution steps:
1.Find senlin requirements.txt
root@promote:/home/openstack/senlin# cat requirements.txt
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.

pbr>=1.6 # Apache-2.0
...
openstacksdk>=0.9.7 # Apache-2.0
...

2.pip search openstacksdk
openstacksdk (0.9.9) - An SDK for building applications to work with OpenStack
  INSTALLED: 0.9.10.dev2
  LATEST: 0.9.9

3.pip install openstacksdk==0.9.7
4.Restart senlin-api
5.Then ok
root@promote:/home/openstack/senlin# openstack cluster build info
+--------+---------------------+
| Field | Value |
+--------+---------------------+
| api | { |
| | "revision": "1.0" |
| | } |
| engine | { |
| | "revision": "1.0" |
| | } |
+--------+---------------------+

6.pip install openstacksdk==0.9.9

7.Restart senlin-api

8.Aslo ok.(It's )
root@promote:/home/openstack/senlin# openstack cluster build info
+--------+---------------------+
| Field | Value |
+--------+---------------------+
| api | { |
| | "revision": "1.0" |
| | } |
| engine | { |
| | "revision": "1.0" |
| | } |
+--------+---------------------+

9.More info see attachment log.txt

[Qiming Teng (tengqim)]

This is a sdk bug introduced when handling exceptions. It is not a senlin bug.

[Qiming Teng (tengqim)]

Solved in 0.9.10 release. Should have added a backref of bug report when fixing this.