Using virtio-interfaces with dpdk leads to assert and crash

[Matthew Ruffell]

[Impact]

If you start a QEMU VM with a dpdk backed virtio-interface, the VM can hit a user triggerable crash during boot.

qemu-system-x86_64: accel/kvm/kvm-all.c:1837: kvm_irqchip_commit_routes: Assertion `ret == 0' failed.
2024-08-24 17:46:20.823+0000: shutting down, reason=crashed

This has been assigned CVE-2024-4693.

There is no workaround, other than to not use dpdk backed virtio-interfaces, and use a generic network device instead.

[Testcase]

Deploy a bare metal host running Noble.

Install a KVM stack:

$ sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils

Set up DPDK with Open vSwitch by following the Ubuntu Server Docs:
https://ubuntu.com/server/docs/how-to-use-dpdk-with-open-vswitch

$ sudo apt-get install openvswitch-switch-dpdk
$ sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
$ ovs-vsctl set Open_vSwitch . "other_config:dpdk-init=true"
# run on core 0 only
$ ovs-vsctl set Open_vSwitch . "other_config:dpdk-lcore-mask=0x1"
# Allocate 2G huge pages (not Numa node aware)
$ ovs-vsctl set Open_vSwitch . "other_config:dpdk-alloc-mem=2048"
# limit to one whitelisted device
$ ovs-vsctl set Open_vSwitch . "other_config:dpdk-extra=--pci-whitelist=0000:04:00.0"
$ sudo service openvswitch-switch restart

Attach DPDK ports to Open vSwitch

$ ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev
$ ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk "options:dpdk-devargs=${OVSDEV_PCIID}"

Open vSwitch DPDK to KVM guests

$ ovs-vsctl add-port ovsdpdkbr0 vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient "options:vhost-server-path=/var/run/vhostuserclient/vhost-user-client-1"

Start a noble instance in KVM, e.g. with uvt-kvm

$ uvt-kvm create --cpu 4 --memory 4096 --disk 10 noble-test release=noble arch=amd64

Edit the virsh to add the dpdk virtio interface

$ virsh edit noble-test

<interface type='vhostuser'>
<source type='unix'
path='/var/run/vhostuserclient/vhost-user-client-1'
mode='server'/>
<model type='virtio'/>
</interface>

Restart the vm, and qemu should hit the assert:

qemu-system-x86_64: accel/kvm/kvm-all.c:1837: kvm_irqchip_commit_routes: Assertion `ret == 0' failed.
2024-08-24 17:46:20.823+0000: shutting down, reason=crashed

Alternatively, you can follow the instructions in the upstream bug:

Run dpdk-testpmd as vhost-user backend in hypervisor:

$ build/app/dpdk-testpmd -a 0000:00:00.0 -l 0-3 -n 4 --vdev 'net_vhost0,iface=/tmp/vfe-net0,queues=4'

Prepare virtio device inside VM:

$ ifconfig eth1 down
$ echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
$ mount -t hugetlbfs nodev /mnt/huge
$ modprobe uio
$ insmod dpdk-kmods/linux/igb_uio/igb_uio.ko
$ dpdk/usertools/dpdk-devbind.py --bind=igb_uio 00:06.0

Run testpmd inside VM:

$ dpdk/build/app/dpdk-testpmd -a 00:06.0 -- --txd=128 --rxd=128 --txq=4 --rxq=4 --nb-cores=1 --forward-mode=txonly --stats-period=1

QEMU will crash.

If you install the test packages from the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf398663-test

The crash should no longer occur.

[Where problems could occur]

This change affects the virtio-pci subsystem in qemu, by fixing up how interrupt vectors are assigned and released.

A regression was already reported in the primary patch upstream, see the below other info section. The followup patch was created to fix this regression, and is necessary.

If a regression were to occur in Ubuntu, it could affect any VM using virtio devices, which is the vast majority of VMs. There is no workaround, other than to remove the virtio-interfaces from the XML files to start the VM. Most users would downgrade instead until a fix is available.

[Other info]

This was fixed upstream in the following commits:

commit 2ce6cff94df2650c460f809e5ad263f1d22507c0
Author: Cindy Lu <email address hidden>
Date: Fri, 12 Apr 2024 14:26:55 +0800
Subject: virtio-pci: fix use of a released vector
Link: https://gitlab.com/qemu-project/qemu/-/commit/2ce6cff94df2650c460f809e5ad263f1d22507c0

commit a8e63ff289d137197ad7a701a587cc432872d798
Author: Cindy Lu <email address hidden>
Date: Tue Aug 6 17:37:12 2024 +0800
Subject: virtio-pci: Fix the use of an uninitialized irqfd
Link: https://gitlab.com/qemu-project/qemu/-/commit/a8e63ff289d137197ad7a701a587cc432872d798

Noble requires both, and Oracular requires the latter only.

Upstream bug:
https://gitlab.com/qemu-project/qemu/-/issues/1928
Upstream bug for regression:
https://gitlab.com/qemu-project/qemu/-/issues/2321
https://gitlab.com/qemu-project/qemu/-/issues/2334

Ubuntu CVE tracker:
https://ubuntu.com/security/CVE-2024-4693

[Matthew Ruffell (mruffell)]

Debdiff for Noble that fixes this issue.

[Matthew Ruffell (mruffell)]

Debdiff for oracular that fixes this issue.

[Matthew Ruffell (mruffell)]

Fix committed in:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa

Noble:
1:8.2.2+ds-0ubuntu1.4
Oracular:
1:9.0.2+ds-4ubuntu5.1

[Matthew Ruffell (mruffell)]

This was fixed in 1:8.2.2+ds-0ubuntu1.4 for Noble and 1:9.0.2+ds-4ubuntu5.1 for Oracular.