numa cell ids need to be normalized before creating xml

[Przemyslaw Czesnowicz]

When creating instance with numa topology, if that instance is placed on a host numa node different then 0 , invalid libvirt.xml will be generated. it's because instance cell id's are used to store host id assignment. Instance cell id's should be normalized before generating libvirt.xml

2014-11-28 16:35:55.195 ERROR nova.virt.libvirt.driver [-] Error defining a domain with XML: <domain type="kvm">
  <uuid>b90dbaea-ed6f-49fa-b3bf-75137005c621</uuid>
  <name>instance-00000005</name>
  <memory>2097152</memory>
  <numatune>
    <memory mode="strict" nodeset="1"/>
    <memnode cellid="1" mode="strict" nodeset="1"/>
  </numatune>
  <vcpu>4</vcpu>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1.0"/>
      <nova:name>vm1</nova:name>
      <nova:creationTime>2014-11-28 16:35:55</nova:creationTime>
      <nova:flavor name="nano_mini">
        <nova:memory>2048</nova:memory>
        <nova:disk>0</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="04d6fa121cbc4b60b57c96eb32b356b1">admin</nova:user>
        <nova:project uuid="9193517bf6c64d0296d60095c88b2e5a">demo</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="c0fc1ed6-1ee5-47fa-a2e7-f1aee3c6750b"/>
    </nova:instance>
  </metadata>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2015.1.0</entry>
      <entry name="serial">10881668-12bf-49c2-9efb-28bb8d278929</entry>
      <entry name="uuid">b90dbaea-ed6f-49fa-b3bf-75137005c621</entry>
    </system>
  </sysinfo>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cputune>
    <vcpupin vcpu="0" cpuset="10-19,30-39"/>
    <vcpupin vcpu="1" cpuset="10-19,30-39"/>
    <vcpupin vcpu="2" cpuset="10-19,30-39"/>
    <vcpupin vcpu="3" cpuset="10-19,30-39"/>
  </cputune>
  <clock offset="utc">
    <timer name="pit" tickpolicy="delay"/>
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="hpet" present="no"/>
  </clock>
  <cpu match="exact">
    <topology sockets="4" cores="1" threads="1"/>
    <numa>
      <cell id="1" cpus="0-3" memory="2097152"/>
    </numa>
  </cpu>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" cache="none"/>
      <source file="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/disk.config"/>
      <target bus="ide" dev="hdd"/>
    </disk>
    <interface type="hostdev" managed="yes">
      <mac address="fa:16:3e:69:8c:17"/>
      <source>
        <address type="pci" domain="0x0000" bus="0x82" slot="0x10" function="0x2"/>
      </source>
      <vlan>
        <tag id="1000"/>
      </vlan>
    </interface>
    <serial type="file">
      <source path="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/console.log"/>
    </serial>
    <serial type="pty"/>
    <graphics type="vnc" autoport="yes" keymap="en-us" listen="127.0.0.1"/>
    <video>
      <model type="cirrus"/>
    </video>
    <memballoon model="virtio">
      <stats period="10"/>
    </memballoon>
  </devices>
</domain>

2014-11-28 16:35:55.195 ERROR nova.compute.manager [-] [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] Instance failed to spawn
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] Traceback (most recent call last):
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/shared/stack/nova/nova/compute/manager.py", line 2246, in _build_resources
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] yield resources
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/shared/stack/nova/nova/compute/manager.py", line 2116, in _build_and_run_instance
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] instance_type=instance_type)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 2641, in spawn
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] block_device_info, disk_info=disk_info)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4490, in _create_domain_and_network
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] power_on=power_on)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4423, in _create_domain
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] LOG.error(err)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/usr/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] six.reraise(self.type_, self.value, self.tb)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4407, in _create_domain
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] domain = self._conn.defineXML(xml)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] rv = execute(f, *args, **kwargs)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] six.reraise(c, e, tb)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] rv = meth(*args, **kwargs)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3484, in defineXML
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] libvirtError: XML error: Exactly one 'cell' element per guest NUMA cell allowed, non-contiguous ranges or ranges not starting from 0 are not allowed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/138128

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/138128
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cb98f1e2f288de0b3d5f85c84717540b9290f8a8
Submitter: Jenkins
Branch: master

commit cb98f1e2f288de0b3d5f85c84717540b9290f8a8
Author: Przemyslaw Czesnowicz <email address hidden>
Date: Mon Dec 1 11:43:29 2014 +0000

    Libvirt normalize numa cell ids

    This patch normalizes the cell ids before writing xml.
    As cell ids are used to store host cell association,
    they need to be normalized before writing xml, or otherwise
    the xml will be invalid.

    Change-Id: I9b565670f5b0ad24c7a5ff1853648594643b4c7a
    Closes-bug: #1397381

[Nikola Đipanov (ndipanov)]

Actually the proposed fix will break for NUMA topologies where host cell ID that the instance cell is pinned to is not in increasing order. The reason for this is the following loop

https://github.com/openstack/nova/blob/4715dd9b402137ba16c8afa0519aca2eccb9c2cf/nova/virt/libvirt/driver.py#L3509

As it will go through the host cells (always in ascending order, and find instance cells that match (may not be in ascending order depending on how the fitting logic and usage of the host) but we simply append memnodes so they may end up in the wrong order when we hit the normalization loop:

https://github.com/openstack/nova/blob/4715dd9b402137ba16c8afa0519aca2eccb9c2cf/nova/virt/libvirt/driver.py#L3553

Thus assigning CPUs to the correct host numa node, but force all memory to the wrong one. Ouch!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/140357

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140357
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=08290cec7f02fa908dfda89d1749142d2d520f48
Submitter: Jenkins
Branch: master

commit 08290cec7f02fa908dfda89d1749142d2d520f48
Author: Nikola Dipanov <email address hidden>
Date: Tue Dec 9 15:58:48 2014 +0100

    libvirt: Fix NUMA memnode assignments to host cells

    This fix

      https://review.openstack.org/138128

    breaks assignments for NUMA topologies where host cell ID
    that the instance cell is pinned to is not in increasing order. The
    reason for this is that the main loop of the method that will go through
    the host cells (always in ascending order), and find instance cells that
    match (may not be in ascending order depending on how the fitting logic
    and usage of the host) but we simply append memnodes so they may end up
    in the wrong order when we hit the normalization loop.

    We fix this by "pre-allocating" the memnodes list and making sure to add
    the matching memnode to the same position as the guest config cell
    before hitting the normalization loop.

    Change-Id: I91055dc705f2ef54c830e641f9cf9ba0683eb313
    Closes-bug: #1397381