Got 500 http error when using MAAS Api

[Aymen Frikha]

We got random HTTP 500 errors when we use MAAS API essentially with POST and PUT requests:

2018-06-14 13:12:32 regiond: [info] 192.0.2.5 PUT /MAAS/api/2.0/machines/ahxeyd/ HTTP/1.1 --> 200 OK (referrer: -; agent: python-requests/2.19.0)
2018-06-14 13:12:33 maasserver: [error] ################################ Exception: 'Machine' object is not iterable ################################
2018-06-14 13:12:33 maasserver: [error] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 277, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 70, in __call__
    response = upcall(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 21, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 229, in __call__
    else: stream = srl.render(request)
  File "/usr/lib/python3/dist-packages/piston3/emitters.py", line 464, in render
    response = serializers.serialize(format, self.data, indent=True)
  File "/usr/lib/python3/dist-packages/django/core/serializers/__init__.py", line 129, in serialize
    s.serialize(queryset, **options)
  File "/usr/lib/python3/dist-packages/django/core/serializers/base.py", line 80, in serialize
    for count, obj in enumerate(queryset, start=1):
TypeError: 'Machine' object is not iterable

2018-06-14 13:12:33 regiond: [info] 192.0.2.5 POST /MAAS/api/2.0/machines/ahxeyd/?op=commission HTTP/1.1 --> 500 INTERNAL_SERVER_ERROR (referrer: -; agent: python-requests/2.19.0)

We use MAAS 2.4 on ubuntu 18.04.

This is the script we use to send requests: https://paste.ubuntu.com/p/33WmbvVY2N/

[Andres Rodriguez (andreserl)]

Hi Aymen,

Could you please provide more context on when and how this happens? looking at the logs and having a script doesn't give us enough context to determine your issue. As such, can you explain what it does, how it does, it, etc?

Thanks!

[Andres Rodriguez (andreserl)]

Also, please attach all logs, that is:

regiond.log, maas.log and rackd.log, and if you could get me the hostname of the machine in question, it would be good too.

thanks.

[J-F (jf.cloutier)]

Hi Andres,

I worked with Aymen and he reported the bug for me.

The script is a test script we are using as part of our CI/CD pipeline, validating some functionality we are using on MAAS.

This MAAS instance is deployed in openstack and the test will spin up a VM which get discovered, then we commission it and then deploy OS on it. In most case all of this goes just fine and the test pass.

From time to time, while we send POST/PUT API commands, we get HTTP/500 errors. The error reported is always following this pattern: XXXX object is not iterable where XXXX is sometime MACHINE, VLAN or SUBNET depending on what API call we are doing.

I haven't been able to identify pattern which cause API call to fail. When it fails, if we retry the same operation sometime it will works just fine and sometime it will fails with HTTP/500 (once again, I haven't been able to identify pattern).

I will trigger the issue again and provide the other logs you are looking for.

[J-F (jf.cloutier)]

Here are the logs:
Regiond.log: https://paste.ubuntu.com/p/RMW6tjWqQn/
rackd.log: https://paste.ubuntu.com/p/rrhbvD9zJv/
maas.log is empty

those errors got generated by an API Call from an ansible module doing API call to move subnet. There is a retry on the task and it looks like first 2 errors was when the module is updating the subnet which worked after retry and then it failed on enabling DHCP.

Here is a snippet of the module we are using which caused issue in this specific case:
https://paste.ubuntu.com/p/gsQZYjCDbp/

Thanks!

[Gabriel Ramirez (gabriel1109)]

Hi,

The problem arises only on 2.4.0 from the MaaS stable PPA and is the result of using any for the accept header. The below is an exact call I used on the REST api directly to reproduce this:

POST /MAAS/api/2.0/machines/tkfp3q/?op=commission HTTP/1.1
Host: maas:5240
Authorization: OAuth oauth_consumer_key="2CrZ4D9p2YnkztkVCA",oauth_token="7zrgp8RJYK2LEtzBHV",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1530651774",oauth_nonce="CDDBKf",oauth_version="1.0",oauth_signature="PgiTcn6xKv8uZ0T4diYGoi%2BLIyg%3D"
Accept: */*
Cache-Control: no-cache
Postman-Token: 1933664a-bf8c-49e5-59bf-21e5b9d7b961

If the Accept header changes to something such as 'application/json' then the request works.

Note: pre 2.4.0 it worked with 'any' but the response from the REST API would alternate (i.e. application/json first, then application/python-pickle). Now it just 500s with a 'machine object not iterable error'

[Adam Collard (adam-collard)]

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team