[2.0] VMware power type error: __init__() takes 1 positional argument but 2 were given

[jim0112]

[Impact]
python-pyvmomi in xenial is outdated and does not allow connecting to vsphere with TLS validation. Updating to 6.5 fixes this and probably a few other issues.

[Test case]
For example,

>>> from pyVim.connect import SmartConnect
>>> import ssl
>>> context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
>>> context.verify_mode = ssl.CERT_NONE
>>> si = SmartConnect(host=host, user=user, pwd=password, port=port, sslContext=context)

should work.

This is basically a HWE type scenario.

[Regression potential]
Compatibility with vSphere 4.1 is not guaranteed anymore. Some bug fixes; and otherwise it's mostly just new data types and methods, so low likelihood of regressions on that side. As a bonus point, we won't have any changes to bionic.

[Original bug report]
When adding VMware power type in 2.0.0 beta7:

Failed to query node's BMC - __init__() takes 1 positional argument but 2 were given

The same VMware details (UUID, vCenter IP address, username/password, etc.) worked in an earlier version of the maas server, which was Ubuntu 14.04 / maas 1.9.

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================================-============-=================================================
ii maas 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS server common files
ii maas-dhcp 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS DHCP server
ii maas-dns 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS DNS server
ii maas-proxy 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all Rack Controller for MAAS
ii maas-region-api 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS server provisioning libraries (Python 3)

[Mike Pontillo (mpontillo)]

Thanks for bringing this bug to our attention.

Unfortunately, I don't see the exception reported in the error message anywhere in the log, so it's difficult to know what the exact issue is here.

Can you check to ensure that the python3-pyvmomi package is installed on the rack controller, and let me know what its version is?

Also, which VMware product are you connecting to and what version is it?

I have heard reports that the new version of the pyvmomi API is strict regarding validation of SSL certificates. You may need to download the certificate from the VMware server and ensure the rack controller considers it to be trusted. (I think placing it in /etc/ssl/certs and running "update-ca-certificates" would do it.)

It may help us get a better error message if you try the power query from a Python shell on the rack controller, such as:

python3
from provisioningserver.drivers.hardware import vmware as vmware
vmware.power_query_vmware("vmware-host", "username", "password", "vm_name", "uuid", port=None, protocol=None)

You could try port=443 and protocol="https" as well. Replace the other quoted strings the your power information for the node.

[jim0112 (james-leavers)]

Thanks for the tip. Looks like you are correct about the SSL certificate - I will try importing it.

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==================================================================-======================================-======================================-=========================================================================================================================================
ii python-pyvmomi 5.5.0-2014.1.1-3 all VMware vSphere Python SDK - Python 2.x

/usr/lib/python3/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pyVim/connect.py", line 307, in __Login
    content = si.RetrieveContent()
  File "/usr/lib/python3/dist-packages/pyVmomi/VmomiSupport.py", line 566, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python3/dist-packages/pyVmomi/VmomiSupport.py", line 375, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/python3/dist-packages/pyVmomi/SoapAdapter.py", line 1254, in InvokeMethod
    conn.request('POST', self.path, req, headers)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 1260, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 376, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 748, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 984, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 629, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/provisioningserver/drivers/hardware/vmware.py", line 465, in power_query_vmware
    if api.connect():
  File "/usr/lib/python3/dist-packages/provisioningserver/drivers/hardware/vmware.py", line 173, in connect
    **extra_args)
  Fi...

[Mike Pontillo (mpontillo)]

Thanks. Please let me know if it works after the certificate is imported.

Can I assume that your region controller is on the same machine as your rack controller? (you didn't upgrade the region and not the rack?)

I see that you have python-pyvmomi version 5.5.0-2014.1.1-3 installed - can you ensure "python3-pyvmomi" is installed as well? (MAAS 2.0 is based on Python 3.5, so we require the "python3" version of all our dependencies now.)

[jim0112 (james-leavers)]

Yes, now working after the following:

Copy certificates to /usr/share/ca-certificates/internal
Add internal/certificate.crt to /etc/ca-certificates.conf
update-ca-certificates

Yes, this is one machine that is a region & rack controller.

python3-pyvmomi is already installed, so perhaps this happened automatically when python-pyvmomi was installed.

Thanks!

[Dave Hindle (da6e)]

Hi Guys

I am having the same difficulty as James but his fix has not worked for me.

I get the certificate by following this http://unix.stackexchange.com/questions/299628/ubuntu-maas-ssl-error-vmware-vcenter-6-connection.

I put the file in ca-certificates/internal as suggested by James.

Following update-ca-certificates, I can see a matching certificate in the /etc/ssl/certs/ca-certificates.crt file so I'm sure the certificate I downloaded is installed but I still get the CERTIFICATE_VERIFY_FAILED message (the traceback is identical to the one James posted).

Can you suggest anything else I should look at?

Thanks

[Mike Pontillo (mpontillo)]

Dave, which version of MAAS are you running? If you have the latest from the PPA, there is a workaround for this: use 'https+unverified' as the protocol when specifying your power parameters.

See also:

https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1608639/comments/1

[Dave Hindle (da6e)]

Hi Mike

Appreciate the help!

We're on maas 2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 (with python3-pyvmomi 5.5.0-2014.1.1-3)

Using unverified, I get

provisioningserver.drivers.hardware.vmware.VMwareAPIException: Failed to set up unverified SSL context. Please update to a more recent version of the python3-pyvmomi package.

I picked up and installed the certificate as described in the linked post but have not modified the power parameters as described (I expect this would not work yet because of the above exception).

I'll take a look at the ppa install.

Thanks

[Dave Hindle (da6e)]

Oh great!

sudo add-apt-repository ppa:maas/stable
sudo apt-get update

W: The repository 'http://ppa.launchpad.net/maas/stable/ubuntu xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch http://ppa.launchpad.net/maas/stable/ubuntu/dists/xenial/main/binary-amd64/Packages 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

[Dave Hindle (da6e)]

Hi All

I have ignored the ppc for maas (error above) and installed pyvmomi directly from github (since I couldn't figure out how to install the python3 version using pip).

I can now connect and get the power status.

Next job will be persuading maas to use the unverified protocol option.

[Dave Hindle (da6e)]

So near and yet so far! Once I'd configured my MAAS machines to use the https+unverified VMWare option I was able to see whether they were powered on or off. But I was still not able to Power them On or Off through MAAS (or more importantly Commission them). It turned out that this option is not available in the free ESXi - the rackd.log told all:

'Current license or ESXi version prohibits execution of the requested operation.'

[Christian Reis (kiko)]

Thanks for the diagnosis -- the bug report suggests that there's a problem in the exception handling code, but it's unclear whether it's in MAAS or deeper in the stack?

[Mike Pontillo (mpontillo)]

For the record, I've used this as a workaround in the past to get the latest version of pyvmomi:

   sudo pip3 install --system --upgrade pyvmomi

I was not aware that this restriction had been placed on the free version of ESXi. Thanks for the heads up. I wonder if this is a new restriction? I had used ESXi 5.5 to test before, but that may have been with a trial license.

[Mike Pontillo (mpontillo)]

@kiko, the bug mentioned in the original report was fixed in a MAAS release candidate, but fixing it uncovered issues deeper in the stack due to the way pyvmomi now verifies certificates. (In the first release that was in the PPA for MAAS 1.9, it did not verify certificates at all, which was actually much more convenient for those using the VMware power driver.)

So with the latest version of the pyvmomi library, there are still two commonly-seen issues (well, maybe 3, now that we know the free version of ESXi doesn't support power control):

(1) The TLS certificate chain (served by the VMware HTTPS server) is not trusted by the system.
(2) The TLS certificate chain is trusted and the signatures are valid and non-expired, but the certificate common name or subject alternate name does not match the hostname or IP address of the connection string specified by the power driver. (For example, if your VMware server has two IP addresses, but the certificate only lists one. Or if the VMware server is configured with a certificate that is signed with a CN and/or subjectAltName of "localhost" or 127.0.0.1.)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Aymen Frikha (aym-frikha)]

I had same issue, but exporting the certificate only didn't resolve it. There is a new version of pyvmomi library. So I resolved the problem by downgrading the library version:

sudo pip3 install pyvmomi==6.0.0.2016.4

[Dustin Kirkland  (kirkland)]

Looks like Ubuntu/MAAS needs an updated python3-pyvmomi.

[Kurt (burnbrighter)]

I am running in to this issue. I see no new activity for over a year. Is there an update or documented workaround that fixes everything? Just trying to make power work properly on VMWare workstation. Thanks

[Julian Andres Klode (juliank)]

I think that bug is fixed with the new upstream release in bionic:

python-pyvmomi (6.5.0.2017.5-0ubuntu1) bionic; urgency=medium

  * New upstream release
  * Remove patch, applied upstream.

 -- Dimitri John Ledkov <email address hidden> Thu, 22 Mar 2018 01:20:53 +0000

[Brian Murray (brian-murray)]

Ubuntu 17.10 is still supported for a couple of more months, should this also be fixed there?

[Julian Andres Klode (juliank)]

I don't think it's worth fixing it in 17.10 too, as I fear nobody will be able to verify it, so I only did xenial.

[Łukasz Zemczak (sil2100)]

Ok, the package looks good in overall, I'm a bit worried about dropping support for vSphere 4.1 (since only the last 4 versions are guaranteed to be supported). I wouldn't want to introduce regressions into xenial for people relying on vSphere 4.1 - can we somehow check if the compatibility is actually broken? Did anyone do an analysis of that?

From how things look, if the newer version is breaking backward compatibility, this looks more like a candidate for backports instead of an SRU.

[Julian Andres Klode (juliank)]

Support for vsphere 4.1 ended in 2014/05 (end of security, bug fixes, etc); with technical guidance (provide of existing bugfixes) ending in 2016/05. So I think we're probably fine.

Checking the compatibility seems hard, I certainly have not checked it.

[Łukasz Zemczak (sil2100)]

Ok, I think I'm willing to give it a shot but with a longer testing period to see if anyone would be actually broken by this. I suppose it's not like it's no-longer compatible but just that compatibility is not guaranteed anymore. Too bad we don't have anyone to check that. Maybe we could find someone?
Will be annoying if it does cause breakages though, since we'll have to work-around the version number in case of any fixes. I'd normally just push for backports but I know we don't have much manpower there so I'm worried it'll stall and besides, as mentioned, this does fit the HWE-type scenario in a way.

[Łukasz Zemczak (sil2100)]

Hello jim0112, or anyone else affected,

Accepted python-pyvmomi into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pyvmomi/6.5.0.2017.5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Brown (adambr2)]

Cisco CCP validated that this proposed package works for our use case.

[Charles Crouch (ccrouch)]

FWIW Adam tested http://launchpadlibrarian.net/370641826/python-pyvmomi_6.5.0.2017.5-0ubuntu1~16.04.1_all.deb

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pyvmomi - 6.5.0.2017.5-0ubuntu1~16.04.1

---------------
python-pyvmomi (6.5.0.2017.5-0ubuntu1~16.04.1) xenial; urgency=medium

  * Upload to xenial (LP: #1593469)

 -- Julian Andres Klode <email address hidden> Thu, 05 Apr 2018 15:13:56 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for python-pyvmomi has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Björn Tillenius (bjornt)]

As far as I can see, this issue should now be fixed .If it's not, please reopen the bug with log output from the latest version.