[2.2, rev5774] Error updating block devices after upgrade

[Andres Rodriguez]

After upgrade from rev5773 to rev5774, there's an error updating block devices for the rack controller:

2017-03-02 08:07:23 maasserver: [error] ################################ Exception: 'JSONDecodeError' object has no attribute 'message' ################################
2017-03-02 08:07:23 maasserver: [error] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/metadataserver/builtin_scripts/hooks.py", line 306, in update_node_physical_block_devices
    blockdevs = json.loads(output.decode("ascii"))
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 177, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 59, in __call__
    response = upcall(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 21, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 190, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 188, in __call__
    result = meth(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 298, in dispatch
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 587, in signal
    target_status = process(node, request, status)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 450, in _process_commissioning
    node, node.current_commissioning_script_set, request, status)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 415, in _store_results
    script_result.store_result(**args)
  File "/usr/lib/python3/dist-packages/metadataserver/models/scriptresult.py", line 153, in store_result
    exit_status=self.exit_status)
  File "/usr/lib/python3/dist-packages/metadataserver/builtin_scripts/hooks.py", line 308, in update_node_physical_block_devices
    raise ValueError(e.message + ': ' + output)
AttributeError: 'JSONDecodeError' object has no attribute 'message'

[Andres Rodriguez (andreserl)]

For machines no error was surfaced but they got stuck after running the block devices commissioning script

[Andres Rodriguez (andreserl)]

fwiw, there is no error surfaced when this failed in the machines, so this also needs to be fixed.

[Blake Rouse (blake-rouse)]

This has always been a bug in MAAS and now is the perfect time to fix it. The processing of the result of the script output from a commissioning script should affect the pass/fail status of that script.

Say we get lshw information, but the hook that processes that information raises an exception. We should catch that error and mark the script as failed, since MAAS failed to process that hook. The only issue is really where to put the error and stacktrace from that failure. I suggest placing it in the stderr of the script like this:

==== MAAS Hook Processing Failed ====
...stacktrace...

[Andres Rodriguez (andreserl)]

I've restarted maas-regiond and maas-rackd and noticed:

==> regiond.log <==
2017-03-02 10:19:05 maasserver: [error] ################################ Exception: 'JSONDecodeError' object has no attribute 'message' ################################
2017-03-02 10:19:05 maasserver: [error] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/metadataserver/builtin_scripts/hooks.py", line 306, in update_node_physical_block_devices
    blockdevs = json.loads(output.decode("ascii"))
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 177, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 59, in __call__
    response = upcall(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 21, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 190, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 188, in __call__
    result = meth(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 298, in dispatch
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 587, in signal
    target_status = process(node, request, status)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 450, in _process_commissioning
    node, node.current_commissioning_script_set, request, status)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 415, in _store_results
    script_result.store_result(**args)
  File "/usr/lib/python3/dist-packages/metadataserver/models/scriptresult.py", line 153, in store_result
    exit_status=self.exit_status)
  File "/usr/lib/python3/dist-packages/metadataserver/builtin_scripts/hooks.py", line 308, in update_node_physical_block_devices
    raise ValueError(e.message + ': ' + output)
AttributeError: 'JSONDecodeError' object has no attribute 'message'