QEMU >= 5.0.0 with -accel tcg uses a tb-size of 1GB causing OOM issues in CI

[Lee Yarwood]

This is a Nova tracker for a set of issues being seen in OpenStack CI jobs using QEMU >= 5.0.0 caused by the following change in defaults witin QEMU:

https://github.com/qemu/qemu/commit/600e17b26

https://gitlab.com/qemu-project/qemu/-/issues/693

At present most of the impacted jobs are being given an increased amount of swap with lower Tempest concurrency settings to avoid the issue, for example for CentOS 8 stream:

https://review.opendev.org/c/openstack/devstack/+/803706

https://review.opendev.org/c/openstack/tempest/+/797614

Longer term a libvirt RFE has been raised to allow Nova to control the size of the cache:

https://gitlab.com/libvirt/libvirt/-/issues/229

[Sylvain Bauza (sylvain-bauza)]

Putting to Confirmed given we already triaged it.

[yatin (yatinkarel)]

Ubuntu jammy is also impacted as it has qemu-6.

Libvirt[1] side is implemented with https://gitlab.com/libvirt/libvirt/-/commit/58bf03f85e295345d0b9750b630406974416bdec

[1] https://libvirt.org/formatdomain.html#hypervisor-features

[yatin (yatinkarel)]

Something like below is required with option to allow configurable tb-cache

# nova/virt/libvirt/config.py
class LibvirtConfigGuestFeatureTCG(LibvirtConfigGuestFeature):

    def __init__(self, **kwargs):
        super(LibvirtConfigGuestFeatureTCG, self).__init__("tcg", **kwargs)

    def format_dom(self):
        root = super(LibvirtConfigGuestFeatureTCG, self).format_dom()
        root.append(self._text_node("tb-cache", "32",
                                    unit="MiB"))

        return root

# nova/virt/libvirt/driver.py
if CONF.libvirt.virt_type in ('qemu'):
    guest.add_feature(vconfig.LibvirtConfigGuestFeatureTCG())

[yatin (yatinkarel)]

Fixed with https://review.opendev.org/c/openstack/nova/+/868419

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 28.0.0.0rc1 release candidate.