MAAS Incorrectly compares AMT versions (as strings)

[Mark Shuttleworth]

On Xenial I seem to be unable to query power status on AMT nodes.

AMT_PASSWORD=***** /usr/bin/amttool 192.168.9.3 powercycle
404 Not Found at /usr/bin/amttool line 297.

I see a version of this error in the MAAS logs.

2016-02-03 16:02:02+0000 [ClusterClient,client] Failed to refresh power state.
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback
     self._startRunCallbacks(fail)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
     self._runCallbacks()
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
     _inlineCallbacks(r, g, deferred)
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
     result = result.throwExceptionIntoGenerator(g)
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python3/dist-packages/provisioningserver/power/query.py", line 146, in get_power_state
     raise exc_type(exc_value).with_traceback(exc_trace)
   File "/usr/lib/python3/dist-packages/provisioningserver/power/query.py", line 117, in get_power_state
     system_id, hostname, power_type, context)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
     result = result.throwExceptionIntoGenerator(g)
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python3/dist-packages/provisioningserver/drivers/power/__init__.py", line 240, in query
     self.power_query, system_id, context)
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
     result = inContext.theWork()
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/provisioningserver/drivers/power/amt.py", line 366, in power_query
     return self.amttool_query_state(ip_address, power_pass)
   File "/usr/lib/python3/dist-packages/provisioningserver/drivers/power/amt.py", line 155, in amttool_query_state
     'info', ip_address, power_pass)
   File "/usr/lib/python3/dist-packages/provisioningserver/drivers/power/amt.py", line 115, in _issue_amttool_command
     return self._run(command, power_pass, stdin=stdin)
   File "/usr/lib/python3/dist-packages/provisioningserver/drivers/power/amt.py", line 105, in _run
     "Failed to run command: %s with error: %s" % (command, stderr))
 provisioningserver.drivers.power.PowerFatalError: Failed to run command: ('amttool', '192.168.9.3', 'info') with error: b'404 Not Found at /usr/bin/amttool line 242.\n'

[Mike Pontillo (mpontillo)]

Possibly related: #1513198. I noticed that we see the same "404 Not Found" error when when querying AMT versions > 8. Do you know which version of AMT is running on the node being queried?

In MAAS, it seems that we are relying on an initial run of "amttool <ip-address> info" in order to report the AMT version before choosing a tool to use in order to do the power control (or query).

Can you add the full output of the following command (stdout and stderr) to this bug:

AMT_PASSWORD=your-password /usr/bin/amttool 192.168.9.3 info

Reading the code, I'm concerned that if the AMT version is 10 or higher, MAAS may incorrectly attempt to use amttool rather than wsman. This is because in amt.py, we have the following check:

        version = stdout.split(
            'AMT version:')[1].split()[0].split('.')[0]
        if version > '8':
             return 'wsman'
        else:
            return 'amttool'

If the version here is '10', I suspect we'll incorrectly select 'amttool':

    >>> '10' > '8'
    False

[Mark Shuttleworth (sabdfl)]

On 03/02/16 16:42, Mike Pontillo wrote:
> Reading the code, I'm concerned that if the AMT version is 10 or higher,
> MAAS may incorrectly attempt to use amttool rather than wsman. This is
> because in amt.py, we have the following check:
>
> version = stdout.split(
> 'AMT version:')[1].split()[0].split('.')[0]
> if version > '8':
> return 'wsman'
> else:
> return 'amttool'
>
> If the version here is '10', I suspect we'll incorrectly select
> 'amttool':
>
> >>> '10' > '8'
> False

ZOMG. That's likely what's going on here. Reading that code makes me
want to find another AMT implementation.

### AMT info on machine '192.168.9.3' ###
AMT version: 10.0.47
404 Not Found at /usr/bin/amttool line 242.

[Andres Rodriguez (andreserl)]

@Mark,

I just want to confirm. Were you running 1.9 against this AMT machine and it would still fail to power manage? I wonder if this is a python3-ism.

Thanks.

[Andres Rodriguez (andreserl)]

(Note that his was initially tested by the Certification team to ensure wsman was used for AMT versions > 8)

[Andres Rodriguez (andreserl)]

I've looked into this, and this doesn't affect 1.9, because at the time AMT still was a shell script template. In 1.10 this was changed to python code and wasn't ported correctly.

[Mike Pontillo (mpontillo)]

@Andres, that was my incorrect assumption. Thanks for checking 1.9.