Invalid operation nodes/?op=list returns 500 error

[Ali Al-Shabibi]

When using maasclient to issue API requests the server suffers a RuntimeError.

##################################
# Relevant /var/log/maas entries #
##################################

2016-09-07 07:59:26 [maasserver] ERROR: ################################ Exception: dictionary changed size during iteration ################################
2016-09-07 07:59:26 [maasserver] ERROR: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 180, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/maasserver/utils/async.py", line 176, in savepoint
    yield
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 180, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 55, in __call__
    response = upcall(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 21, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 185, in __call__
    request = self.cleanup_request(request)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 269, in cleanup_request
    for k in sanitized.keys():
RuntimeError: dictionary changed size during iteration

2016-09-07 07:59:26 [-] 127.0.0.1 - - [07/Sep/2016:07:59:26 +0000] "GET /MAAS/api/2.0/nodes/?op=list&oauth_nonce=113840182036746051781473235166&oauth_timestamp=1473235166&oauth_version=1.0&oauth_signature_method=PLAINTEXT&oauth_consumer_key=XXXXXXXXXXXXXXXXX&oauth_token=XXXXXXXXXXXXXXXXX&oauth_signature=XXXXXXXXXXXXXXXXXXXXXXXXX HTTP/1.1" 500 40 "-" "python-requests/2.11.1"

########################
# dpkg -l '*maas*'|cat #
########################

ubuntu@xenial-maas:/usr/lib$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-==============================-============-=================================================
ii maas 2.0.0+bzr5189-0ubuntu1~16.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS server common files
ii maas-dhcp 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS DHCP server
ii maas-dns 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS DNS server
ii maas-proxy 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0+bzr5189-0ubuntu1~16.04.1 all Rack Controller for MAAS
ii maas-region-api 2.0.0+bzr5189-0ubuntu1~16.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0+bzr5189-0ubuntu1~16.04.1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0+bzr5189-0ubuntu1~16.04.1 all MAAS server provisioning libraries (Python 3)

#######################
# Steps to reproduce: #
#######################

Python 3.5.2 (default, Sep 7 2016, 00:46:12)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from maasclient import MaasClient
>>> from maasclient.auth import MaasAuth
>>> auth = MaasAuth("http://10.90.0.2/MAAS/api/2.0", "gFH8GA9FLp3bWpesZX:rVyJz3ThB93FrrNyPw:Fu9KZEwBenXRLUu9TMXVkMjLJsuXNXDb")
>>> client = MaasClient(auth)
>>> ret = client.get("/nodes/", dict(op='list'))
>>> ret.raise_for_status()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: http://10.90.0.2/MAAS/api/2.0/nodes/?op=list&oauth_nonce=78916911522743293441473235562&oauth_timestamp=1473235562&oauth_version=1.0&oauth_signature_method=PLAINTEXT&oauth_consumer_key=gFH8GA9FLp3bWpesZX&oauth_token=rVyJz3ThB93FrrNyPw&oauth_signature=%26Fu9KZEwBenXRLUu9TMXVkMjLJsuXNXDb

It's unclear to me if this is a maas bug or a django piston error. But given that I don't know django or piston, I'll post this here until told otherwise.

[Brendan Donegan (brendan-donegan)]

nodes/?op=list is not a valid endpoint in MAAS 2.0 - see the API docs for Nodes here: https://maas.ubuntu.com/docs2.0/api.html#nodes, you want to just list the nodes by doing a GET on http://10.90.0.2/MAAS/api/2.0/nodes/ (with appropriate headers)

The list operation is a MAAS 1.9 concept - the API has actually changed major version between MAAS 1.9 and 2.0 , notice the 1.0 changed to 2.0 in the URL.

If you have code that uses the API, you will likely have to rewrite large parts of it.

However there is still probably a bug here, in that this shouldn't really cause a server error.

[Brendan Donegan (brendan-donegan)]

Also nodes may not really be the right endpoint as this will return all hardware entities in MAAS, including controllers (the MAAS region controller and the rack controllers) and devices. You probably want to use machines if your intent is to find systems under MAAS control

[Sam Lee (samlee)]

I had the same problem, and found the solution.

The problem was adding the oauth parameters to the query strings like the example above:

"GET /MAAS/api/2.0/nodes/?op=list&oauth_nonce=113840182036746051781473235166&oauth_timestamp=1473235166&oauth_version=1.0&oauth_signature_method=PLAINTEXT&oauth_consumer_key=XXXXXXXXXXXXXXXXX&oauth_token=XXXXXXXXXXXXXXXXX&oauth_signature=XXXXXXXXXXXXXXXXXXXXXXXXX"

The solution was to move the oauth parameters to the http header. So the url would look like instead:

"GET /MAAS/api/2.0/nodes/" (As mentioned, maas api 2.0 doesn't need the 'op=list' anymore.)

[Adam Collard (adam-collard)]

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team