VMWare power fails to get IP from hostname

[Matthew Cooper]

MAAS version: 2.6.1 (7832-g17912cdc9-0ubuntu1~18.04.1)

On the configuration of a machine which VMware 6.7 virtual machine power configuration setting the VMware hostname returns an error of "Error: failed to detect a valid IP address from 'm-node-03.whitecrane.io'" (log of error below). This need to be the hostname, not the IP address as it returns an error of "Error: (SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)".

****** LOG *******

2019-09-25 22:18:07 maasserver.websockets.protocol: [critical] Error on request (21) machine.update: failed to detect a valid IP address from 'example.test.com'
 Traceback (most recent call last):
   File "/usr/lib/python3.6/threading.py", line 864, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 852, in worker
     return target()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
     task()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
     task()
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
     result = inContext.theWork()
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper
     result = func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 756, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 563, in retrier
     return func(*args, **kwargs)
   File "/usr/lib/python3.6/contextlib.py", line 52, in inner
     return func(*args, **kwds)
   File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 402, in prep_user_execute
     method_name, method, params)
   File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 421, in _call_method_track_queries
     result = method(params)
   File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 436, in update
     data = super(NodeHandler, self).update(params)
   File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 537, in update
     obj = form.save()
   File "/usr/lib/python3/dist-packages/maasserver/forms/__init__.py", line 1328, in save
     node = super(WithMACAddressesMixin, self).save()
   File "/usr/lib/python3/dist-packages/maasserver/forms/__init__.py", line 1113, in save
     WithPowerTypeMixin.set_values(self, machine)
   File "/usr/lib/python3/dist-packages/maasserver/forms/__init__.py", line 381, in set_values
     params_field_name)
   File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 123, in __setattr__
     prop_obj.fset(self, value)
   File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 1285, in power_parameters
     self.bmc.save()
   File "/usr/lib/python3/dist-packages/maasserver/models/bmc.py", line 299, in save
     super(BMC, self).save(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 204, in save
     validate_unique=False)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1235, in full_clean
     self.clean()
   File "/usr/lib/python3/dist-packages/maasserver/models/bmc.py", line 352, in clean
     raise error
   File "/usr/lib/python3/dist-packages/maasserver/models/bmc.py", line 338, in clean
     subnet = Subnet.objects.get_best_subnet_for_ip(new_ip)
   File "/usr/lib/python3/dist-packages/maasserver/models/subnet.py", line 164, in get_best_subnet_for_ip
     ip = IPAddress(ip)
   File "/usr/lib/python3/dist-packages/netaddr/ip/__init__.py", line 306, in __init__
     'address from %r' % addr)
 netaddr.core.AddrFormatError: failed to detect a valid IP address from 'example.test.com'

[Nick Niehoff (nniehoff)]

This still exists in 2.6.2. A workaround is to use the IP which then causes the SSL error. For that you can set VMware API protocol to https+unverified

[Nick Niehoff (nniehoff)]

The CLI results in the same error:

maas admin machines create architecture=amd64/generic mac_addresses=xxx hostname=xxx power_type=vmware power_parameters_power_pass=xxx power_parameters_power_user=xxx power_parameters_power_address=example.test.com power_parameters_power_protocol=https+unverified power_parameters_power_uuid=xxx --debug
500 Internal Server Error

   Content-Length: 55
     Content-Type: text/plain; charset=utf-8
             Date: Fri, 10 Jan 2020 15:10:11 GMT
           Server: TwistedWeb/18.9.0
           Status: 500
             Vary: Cookie
  X-Frame-Options: SAMEORIGIN

failed to detect a valid IP address from 'vcenter.maas'

[Victor Tapia (vtapia)]

The attached patch enables the use of hostnames for BMC, but it's just a proof of concept (still a place to start from):
- It does not include hostname validation
- It takes only the first entry from the DNS resolved list
- Adds that entry to the staticipaddresses table, but does not update it if it changes.

[Eric Desrochers (slashd)]

I've been told by the MAAS team that the reason why MAAS support IP addresses for BMCs and not hostnames is that MAAS needs to figure out which rackcontroller(s) can reach a given BMC based on which subnets it has access to.

For the moment, safe assumption is that MAAS will remain IP (ipv4|ipv6) only for now.

Which bring me to a second question for the MAAS team, that I didn't get an answer yet.

If we aren't going to accept a hostname in that field could we at least update the UI to reflect "VMware IP" instead of "VMware hostname"?

- Eric

[Chris Privitere (cprivite)]

Can the MaaS team explain why first doing a DNS lookup of the hostname and then doing the logic about rackcontrollers is not an acceptable workflow?

[Alberto Donato (ack)]

@Chris, it's not unacceptable, it's just not as simple as allowing to store the hostname instead of an IP address.

The solution intially poroposed from the attached patch is not viable as the lookup is done when configuring the node power, but would still store the IP. This is subject to change and would not solve the issue with SSL certificates using the hostname, which would fail validation.

Storing and using the hostname is a more involved change, as it requires to extend the logic for finding which rackcontroller can talk to a host (which could possible have different IP addresses and being reachable by different rackcontrollers) and use the hostname when talking to the node so that SSL certs are accepted.

[Chris Privitere (cprivite)]

So don't use the attached patch and write a solution that would work.

[Chris Privitere (cprivite)]

My goal isn't to push this patch through asap or anything, it's to make sure we're on a course where this will be resolved eventually.