Can't create nova aggregate on Xen

[Nikita Gubenko]

Hi

I want to use XCP resource pool feature with openstack (2013.1). I have 2 XCP 1.6 hosts with 1 compute vm on each. Scheme was working ok with "separate" XCP hosts. Now I'm trying to add first host to aggregate:

nova aggregate-create TestPool nova
nova aggregate-set-metadata 2 hypervisor_pool=true
nova aggregate-set-metadata 2 operational_state=created
nova aggregate-add-host 2 DevStackOSDomU

After that operational_state becomes always "changing". In the logs I see

2013-03-28 21:19:50.992 ERROR nova.openstack.common.rpc.amqp [req-3867d0c9-97f0-4711-8266-62cddbdb32b7 admin admin] Exception during message handling
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 117, in wrapped
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 94, in wrapped
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3938, in add_aggregate_host
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp slave_info=slave_info)
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/driver.py", line 579, in add_to_aggregate
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp return self._pool.add_to_aggregate(context, aggregate, host, **kwargs)
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/pool.py", line 105, in add_to_aggregate
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp master_compute = aggregate['metadetails']['master_compute']
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp KeyError: 'master_compute'
2013-03-28 21:19:50.992 TRACE nova.openstack.common.rpc.amqp

I'm new to openstack, is it a bug or am I missing something?
Thanks

[Dan Smith (danms)]

Note that I can't reproduce this on KVM (but I'm not sure I should be able to).

[Nikita Gubenko (nikita-gubenko)]

I don't know python and maybe there is something wrong with my installation, but the way I fixed it was

nova/virt/xenapi/pool.py
line 94:
        if len(aggregate['hosts']) == 1:
changed
to
        if len(aggregate['hosts']) == 0:

because aggregate['hosts'] showed me 0 while adding a host

After that I was able to successfully add first host to aggregate

[John Garbutt (johngarbutt)]

Looks like some no db compute changes broke this one.

Taking a closer look the issue, I think the problem is that when nova updates the aggregate to add the host, the old version of the aggregate is sent to the compute node:
https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2711

Not got change to fix this myself right now, but it looks like you could pass the aggregate id down to the compute node instead of sending the out of date aggregate dict.

[Vangelis Tasoulas (cyberang3l)]

@Nikita,

This looks like a reasonable fix to me as right after you created the aggregation pool, the "hosts" list does not contain any entries hence the length of it is 0 and not 1 (it is 1 after you have added the master).

I applied this fix myself but I would like to ask if you have noticed any further problems.

Thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/37660

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/37660
Committed: http://github.com/openstack/nova/commit/75329ada9859c8d9a5c6d453beb5a16d97023492
Submitter: Jenkins
Branch: master

commit 75329ada9859c8d9a5c6d453beb5a16d97023492
Author: John Garbutt <email address hidden>
Date: Thu Jul 18 11:55:48 2013 +0100

    Send updated aggregate to compute on add/rm host

    The XenAPI driver relies on getting an up-to-date version of the
    aggregate during add_host_to_aggregate and remove_host_from_aggregate
    This change ensures that happens.

    Fixes bug 1161619
    Change-Id: I764e1fc3b9ff20b6eda8f6ec00849f2164de0a49