Power State could not be queried (vmware)

[Julian Zhou]

Steps to reproduce:
1) Install MaaS from stable repo (with dist-upgrade) && VMware workstation 11 on Ubuntu 14.04 LTS
2) Install python-pyvmomi from pip since it appears to be missing from trusty repos
2) Power on a vmware vm so it auto-registers with MaaS
3) Change power setup of said node to http://vgy.me/TTCm6w
4) Attempt to commission node.
When I attempt to power on, I get "Failed to power on node - Node could not be powered on: vmware failed with return code 1: No connection adapters were found for '://http://layne.tcn://sdk/vimServiceVersions.xml' "

MaaS.log: https://gist.github.com/jztech101/5dd1a30c6a13ba3e4b0c
clusterd.log: https://gist.github.com/jztech101/af5759d5207c33454861
regiond.log: https://gist.github.com/jztech101/d2c3b6f3d06f57dd281a

dpkg:
jztech101@layne:/var/log/maas$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===========================================-====================================-============-===============================================================================
ii maas 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server all-in-one metapackage
ii maas-cli 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS command line API tool
ii maas-cluster-controller 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server cluster controller
ii maas-common 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server common files
ii maas-dhcp 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS DHCP server
ii maas-dns 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS DNS server
ii maas-proxy 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS Caching Proxy
ii maas-region-controller 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server complete region controller
ii maas-region-controller-min 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS Server minimum region controller
ii python-django-maas 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server Django web framework
ii python-maas-client 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS python API client
ii python-maas-provisioningserver 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server provisioning libraries

[Andres Rodriguez (andreserl)]

Hi Julian,

This might be an issue provided you are testing with python-pyvmomi using pip. The python-pyvmomi package is in the MAAS PPA;s. https://launchpad.net/~maas/+archive/ubuntu/stable .

Also, you need to set the credentials if they haven't been set.

For the time being, I'm marking this as invalid.

Thanks

[Mike Pontillo (mpontillo)]

FYI, python-pyvmomi should be in our PPA (even for Trusty). I would expect the module from pip to work correctly, though it's possible that this is a regression.

I think it's possible that this is a bug with adding VMware virtual machines manually, as opposed to using "Add Hardware > Chassis". Late in the cycle, I didn't do much testing with manual VMware enlistment, since "Add Hardware" works so much better.

Can you try adding your VMs in bulk using "Add Hardware > Chassis"? (If you only want to add one VM, you can use the prefix filter and just type in the name of the VM you want to enlist.)

[Julian Zhou (jzhou2oo6)]

Hm. I'll try the chassis thing tomorrow. But for now, where do I get the vmware credentials? I have vmware workstation 11 on localhost ;_;

[Julian Zhou (jzhou2oo6)]

ok. Chassis worked and the machine successfully powered on. However it fails to detect power state, and then says it failed commissioning in the control panels:
Failed to query node's BMC - Power state could not be queried: /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) off

By the way. Manual Added VMware virtual machines still do not work properly

[Julian Zhou (jzhou2oo6)]

It states this by the way, even though the node successfully powered on. Failed to power on node - Timeout after 7 tries

[Julian Zhou (jzhou2oo6)]

oh I forgot. Here is clusterd.log: https://gist.github.com/jztech101/f33e80f675bef439e3aa

[Mike Pontillo (mpontillo)]

Try removing the version of pyvmomi installed from pip and use the one in our PPA. This may be a bug with the most recent version of python-pyvmomi. Thanks for bringing this to our attention.

[Julian Zhou (jzhou2oo6)]

Failed to query node's BMC - Power state could not be queried: vmware failed with return code 1: [Errno 2] No such file or directory: '://layne.tcn://sdk/vimServiceVersions.xml' <-- now I am getting this. Perhaps now would be the time to mention, I was wrong. Turns out i am running workstation 12

[daniel (danielbascunan)]

i have de same problem...
i have vmware workstation 11 and install MAAS on Ubuntu 15.10.
everinth is ok but when i tray to do some acction on the nodes i have this error

Failed to query node's BMC - Power state could not be queried: vmware failed with return code 1: No connection adapters were found for '://30.30.30.1://sdk/vimServiceVersions.xml'

anyone knows how can we resovle this?

[Denis Cavalcante (denis-cavalcante)]

I also have the same problem... workstation 12 and MAAS on Ubuntu 14.04 (LTS)

[Frank (frankaritchie)]

Has there been any progress on this issue?

[Mike Pontillo (mpontillo)]

Unfortunately I don't have the time to triage this fully at the moment (and it would require me to rebuild a VMware setup; I'm developing on Xenial and my copy of VMware Workstation seems to not support that yet).

My suspicion is that there is a bug with defining VMware nodes on a one-by-one basis. A potential workaround would be to use the "Add Hardware > Chassis" command to add the VMware node, optionally using the prefix filter if you wish to target a particular virtual machine to add to MAAS.

[Frank (frankaritchie)]

I am experiencing the issue with the "OpenStack Autopilot for Vsphere"
trial, not Workstation

On Fri, Feb 19, 2016 at 4:00 PM, Mike Pontillo <email address hidden>
wrote:

> Unfortunately I don't have the time to triage this fully at the moment
> (and it would require me to rebuild a VMware setup; I'm developing on
> Xenial and my copy of VMware Workstation seems to not support that yet).
>
> My suspicion is that there is a bug with defining VMware nodes on a one-
> by-one basis. A potential workaround would be to use the "Add Hardware >
> Chassis" command to add the VMware node, optionally using the prefix
> filter if you wish to target a particular virtual machine to add to
> MAAS.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1499934
>
> Title:
> Power State could not be queried (vmware)
>
> Status in MAAS:
> Triaged
>
> Bug description:
> Steps to reproduce:
> 1) Install MaaS from stable repo (with dist-upgrade) && VMware
> workstation 11 on Ubuntu 14.04 LTS
> 2) Install python-pyvmomi from pip since it appears to be missing from
> trusty repos
> 2) Power on a vmware vm so it auto-registers with MaaS
> 3) Change power setup of said node to http://vgy.me/TTCm6w
> 4) Attempt to commission node.
> When I attempt to power on, I get "Failed to power on node - Node could
> not be powered on: vmware failed with return code 1: No connection adapters
> were found for '://http://layne.tcn://sdk/vimServiceVersions.xml' "
>
> MaaS.log: https://gist.github.com/jztech101/5dd1a30c6a13ba3e4b0c
> clusterd.log: https://gist.github.com/jztech101/af5759d5207c33454861
> regiond.log: https://gist.github.com/jztech101/d2c3b6f3d06f57dd281a
>
> dpkg:
> jztech101@layne:/var/log/maas$ dpkg -l '*maas*'|cat
> Desired=Unknown/Install/Remove/Purge/Hold
> |
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name Version
> Architecture Description
>
> +++-===========================================-====================================-============-===============================================================================
> ii maas
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server all-in-one
> metapackage
> ii maas-cli
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS command line API tool
> ii maas-cluster-controller
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server cluster
> controller
> ii maas-common
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server common files
> ii maas-dhcp
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS DHCP server
> ii maas-dns
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS DNS server
> ii maas-proxy
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS Caching Proxy
> ii maas-region-controller
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS server complete
> region controller
> ii maas-region-controller-min
> 1.8.2+bzr4041-0ubuntu1~trusty1 all MAAS Serv...

[Chris (chris-minnoy)]

Package python-pyvmomi in PPA is version 5.5.
To support vSphere 6.0 (ESXi 6.0) you need the latest vmomi version 6.0.
Pip installs the latest, but MAAS (I use version 1.8) keeps asking to install the python-pyvmomi package.
In the next release of Ubuntu (Xenial) python-pyvmomi should be upgraded to version 6.

Also I noticed that for the ESXi version not connected to a vCenter (for example in trial mode), you need to accept the self signed cert without verification (see example getallvms.py in vmomi samples). Not sure if MAAS can handle that.
File vmware.py (/usr/lib/python2.7/dist-packages/provisioningserver/drivers/hardware) seems not to be able to handle
the no-verify scenario.

I do wonder however why MAAS can't find the pyvmomi installed by PIP, and needs to have the ubuntu package installed?
Does anyone know how to configure MAAS to refer to the PIP install?

[Eric Desrochers (slashd)]

It has been brought to my attention that someone from the community is also experiencing this problem on Trusty (Ubuntu 14.04.4 LTS)

# dpkg:
ii python-pyvmomi 5.5.0-1ubuntu1 all VMware vSphere Python SDK - Python 2.x
ii maas 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS server all-in-one metapackage
ii maas-cli 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS command line API tool
ii maas-cluster-controller 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS server cluster controller
ii maas-common 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS server common files
ii maas-dhcp 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS DHCP server
ii maas-dns 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS DNS server
ii maas-proxy 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS Caching Proxy
ii maas-region-controller 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS server complete region controller
ii maas-region-controller-min 1.9.0+bzr4533-0ubuntu1~trusty1 all MAAS Server minimum region controller

# MAAS logs:
maas.power: [ERROR] maastest: Failed to refresh power state: [Errno 2] No such file or directory: '://<ESX_HOST>://sdk/vimServiceVersions.xml'
maas.power: [ERROR] Power state could not be queried: [Errno 2] No such file or directory: '://<ESX_HOST>://sdk/vimServiceVersions.xml'
maas.power: [ERROR] maastest: Failed to refresh power state: [Errno 2] No such file or directory: '://<ESX_HOST>://sdk/vimServiceVersions.xml'
maas.power: [ERROR] Power state could not be queried: [Errno 2] No such file or directory: '://<ESX_HOST>://sdk/vimServiceVersions.xml'

Eric

[Mike Pontillo (mpontillo)]

I have proposed a fix for at least a portion of this issue. I tested VMware ESXi 5.5 with the version of python-pyvmomi we have in the MAAS PPA. The MAAS server was running on Trusty. (other combinations have not been tested, such as MAAS running on any Ubuntu release more recent than Trusty, nor the verison of pyvmomi available in pip.)

The fix comes down to this:

https://paste.ubuntu.com/15277153/

If a hotfix is required, the text of that pastebin could be downloaded into (for example) /tmp/vmware.patch, and then the following command could be executed:

sudo patch -d /usr/lib/python2.7/dist-packages -p1 -i /tmp/vmware.patch

If you don't want to apply the patch, there are several workarounds:
 * Make sure to add VMware systems using "Add Hardware > Chassis" (existing systems will be ignored)
 * Do not edit the power parameters; the bug occurs because we see an empty string in the "port" or "protocol" field and try to use it to construct the URL
 * If you have already edited the power parameters, (or have added the node manually) try specifying "https" for the protocol and "443" for the port, rather than leaving the fields blank. (which should be okay, since they are optional fields - but that's what the above patch addresses.)

I'm not sure about the issue regarding TLS certificate trust. I think this must be an issue that occurs due to a newer version of python-pyvmomi than we have in the PPA. This will most likely need to be tracked under a separate bug (if one does not exist already) and fixed for MAAS 1.10 and/or 2.0.

Also, since we are not using features specific to VMware 6.0, it seems to me that the existing version of python-pyvmomi should work. (you should not use pip to upgrade to 6.0, as we have not tested with yet, and additional bugs may be present.)

[Mike Pontillo (mpontillo)]

@k0vic, this fix has been committed on the MAAS 1.9 branch, but a point release has not been created that contains the fix yet.