[1.9] VMware power management fails when VMs are organized in nested subfolders

[Ante Karamatić]

Unfortunately I don't have logs handy right now, but I could add some tomorrow.

If vm name is used in VmWare power management configuration for a machine, power calls result in exceptions. From the memory, API is complaining that there's no 'summary' property for 'Folder' object (detailed logs to be provided tomorrow).

Once vm name is removed and vm uuid is used instead, power management works without issues.

[Ante Karamatić (ivoks)]

This is the type of error:

2015-11-12 16:07:26+0100 [ClusterClient,client] Failed to refresh power state.
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 423, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
            self._runCallbacks()
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/power/query.py", line 126, in get_power_state
            system_id, hostname, power_type, context)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/__init__.py", line 246, in query
            self.power_query, system_id, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/vmware.py", line 70, in power_query
            host, username, password, vm_name, uuid, port, protocol)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/hardware/vmware.py", line 480, in power_query_vmware
            .format(uuid=uuid), traceback.format_exc())
        provisioningserver.drivers.hardware.vmware.VMwareAPIException: (u'Failed to get power state for uuid=', 'Traceback (most recent call last):\n File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/hardware/vmware.py", line 472, in power_query_vmware\n vm = _find_vm_by_uuid_or_name(api, uuid, vm_name)\n File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/hardware/vmware.py", line 428, in _find_vm_by_uuid_or_name\n vm = api.find_vm_by_name(vm_name)\n File "/usr/lib/python2.7/di...

[Ante Karamatić (ivoks)]

When I try importing VMware vCenter, I get:

Nov 12 10:52:13 maas-fo maas.rpc.cluster: [ERROR] Failed to probe and enlist VMware nodes: 'vim.Folder' object has no attribute 'summary'

It's worth noting that VMs in vCenter are organized in Folders. Maybe library is assuming that it will find only VMs, while in fact there might be folders too?

[Gavin Panella (allenap)]

Newell, any idea what's going on here?

[Blake Rouse (blake-rouse)]

Think that is more of a Mike question, since he implemented the VMWare driver.

[Mike Pontillo (mpontillo)]

I remember running into this when developing the power driver. Depending on the VMware API version, getting the VMs by name sometimes works and sometimes doesn't. That's why the option of getting the VM by UUID exists. (The "Add Hardware > Chassis" option tends to do the right thing and fill in the UUID during discovery.)

I think I implemented some code which will attempt to find the VM by name if the UUID is not available, and that's likely where you're seeing the bug. (Which version of VMware are python-pyvmomi are you using, and how exactly are the VMs organized in the system?)

If I can get access to the test environment, I could possibly come up with a fix/workaround. I do not have a license for vCenter to do my own testing. But I would need to know the following in order to reproduce the problem:

(1) Which version of VMware are you using, and which VMware product(s)?
(2) Which Ubuntu release is the cluster running, and which version of python[3]-pyvmomi is installed?
(3) How exactly are the VMs organized within VMware?

[Gavin Panella (allenap)]

Perhaps we should move the VMware power driver out to Universe before 2.0?

[rory schramm (roryschramm)]

I'm running into this problem as well when adding vm's from vcenter via add-hardware -> add-chassis -> power-type = vmware.

log shows:

Apr 19 11:34:59 maas maas.rpc.cluster: [ERROR] Failed to probe and enlist VMware nodes: 'vim.Folder' object has no attribute 'summary'

using vcenter 5.5 appliance build 325642
esxi servers are 5.5 U3 - build 3248547

vm's are in a folder called maas. I also tried with no folders.

maas:

root@maas:/var/log/maas# apt-cache policy maas
maas:
  Installed: 1.9.1+bzr4543-0ubuntu1~trusty1
  Candidate: 1.9.1+bzr4543-0ubuntu1~trusty1
  Version table:
 *** 1.9.1+bzr4543-0ubuntu1~trusty1 0
        500 http://ppa.launchpad.net/maas/stable/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status

root@maas:/var/log/maas# apt-cache policy python-pyvmomi
python-pyvmomi:
  Installed: 5.5.0-1ubuntu1
  Candidate: 5.5.0-1ubuntu1
  Version table:
 *** 5.5.0-1ubuntu1 0
        500 http://ppa.launchpad.net/maas/stable/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status

root@maas:/var/log/maas# cat /etc/*-rel*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"
NAME="Ubuntu"
VERSION="14.04.4 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.4 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"