NodeStateViolation when commissioning

[Gavin Panella]

I started commissioning of a node, but it wasn't able to power on because amttool was not available, so I assume it tried to mark the node as broken/failed. I observed the following in maas-django.log

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 143, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/utils/async.py", line 152, in call_within_transaction
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/rpc/nodes.py", line 44, in mark_node_failed
    node.mark_failed(error_description)
  File "/usr/lib/python2.7/dist-packages/maasserver/models/node.py", line 1309, in mark_failed
    self.status)
maasserver.exceptions.NodeStateViolation: The status of the node is 2; this status cannot be transitioned to a corresponding failed status.

[Raphaël Badin (rvb)]

Looks like 2 things were trying to mark the node as "failed". The first one succeeded and the node was transitioned to 'FAILED_COMMISSIONING'. The second thing came along but the node couldn't be as "failed" because it was already marked "failed". Now, the stacktrace is not detailed enough to tell us precisely what this second thing was. AFAIK the only thing that can mark a node failed right now is the power code (when it fails to power up the node).

[Julian Edwards (julian-edwards)]

This is possibly the same as bug 1368398.

[Julian Edwards (julian-edwards)]

Oh actually it can't be, looking at the code.