[SRU] Cannot create instance with multiqueue image and vif_type=tap (calico)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Rodrigo Barbieri | ||
Stein |
Fix Released
|
Undecided
|
Unassigned | ||
Train |
Fix Released
|
Undecided
|
Unassigned | ||
Ussuri |
Fix Released
|
Undecided
|
Unassigned | ||
Ubuntu Cloud Archive |
Invalid
|
Undecided
|
Unassigned | ||
Stein |
Fix Released
|
Undecided
|
Unassigned | ||
Train |
Fix Released
|
Undecided
|
Unassigned | ||
Ussuri |
Fix Released
|
Undecided
|
Unassigned | ||
nova (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Focal |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
When using calico, the vif_type is tap, therefore when the instance is being created, the method plug_tap() is invoked, which creates the tap device prior to launching the instance.
That tap device is currently always created without multiqueue as per [1]. When libvirt creates the instance, the XML definition "queues=<x>" clashes with the fact that the pre-existing tap interface doesn't have multiqueue enabled, and therefore errors out with the exception below. The code at [2] already handles multiqueue, but it is never invoked with multiqueue=True.
Alternatively, as a current workaround, if the instance is shutdown through virsh, or rebooted through nova, it causes the tap device to be removed, to be created again by libvirt instead, allowing the tap device to be set up with multiqueue appropriately if its XML is manually edited. This begs the question as why the plug_tap() method needs to pre-create the interface at all, if when the VM rebooted libvirt does so regardless of plug_tap().
Steps to reproduce:
1) Ubuntu bionic + devstack master + follow instructions at [3]
2) wget https:/
3) openstack image create bionic-mq --file bionic-
4) openstack image create bionic --file bionic-
5) ssh-keygen
6) openstack keypair create key1 --public-key ~/.ssh/id_rsa.pub
7) openstack flavor create --vcpu 2 --ram 1024 --disk 10 --public --id 10 test_flavor
8) openstack server create --network calico --flavor test_flavor --image bionic --key-name key1 no-mq
instance is created successfully
9) ip a
6: tapcc353751-13: <BROADCAST,
10) sudo virsh edit 1
add "<driver name='vhost' queues='2'/>" to the interface section
11) openstack server reboot no-mq
wait a few secs
12) ip a
7: tapcc353751-13: <BROADCAST,
13) ssh to the instance and run "sudo ethtool -l <interface>"
Combined: 2
14) openstack server delete no-mq
15) openstack server create --network calico --flavor test_flavor --image bionic-mq --key-name key1 mq
instance fails to be created, log shows the below stack trace.
[1] https:/
[2] https:/
[3] https:/
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.
Aug 27 18:58:38 devstack nova-compute[7968]: INFO nova.compute.
=======
[Impact]
Users of calico plugin cannot use multiqueue in Nova. The VM fails to boot. The workaround is to edit the XML manually and reboot it through nova, so the tap interface is recreated by libvirt while the vif.plug() method is not re-run by Nova, allowing multiqueue to be set up properly by libvirt. This workaround does not scale well.
[Test case]
1. Setting up env
1a. Deploy environment
1b. Install calico plugin as per [0]
1c. Setup SSH
ssh-keygen
1d. Create keypair for testing
openstack keypair create key1 --public-key ~/.ssh/id_rsa.pub
1e. Create test flavor
openstack flavor create --vcpu 2 --ram 1024 --disk 10 --public --id 10 test_flavor
1f. Download an example image
wget https:/
1g. Create image in glance with multiqueue metadata
openstack image create bionic-mq --file bionic-
1h. Create same image in glance without multiqueue metadata
openstack image create bionic --file bionic-
1f. Create instance without multiqueue. Make sure instance creation and connectivity succeeds.
openstack server create --network calico --flavor test_flavor --image bionic --key-name key1 no-mq
2. Reproducing the bug
2a. Create instance with multiqueue
openstack server create --network calico --flavor test_flavor --image bionic-mq --key-name key1 mq
Instance creation will fail
2b. Check logs for error
egrep "libvirt.
3. Cleanup
3a. Delete instances "mq" and "no-mq"
4. Install package that contains the fixed code
5. Repeat step 2a. 2a should now succeed.
[Regression Potential]
New Code path is not triggered if image metadata is not used. For all other use cases, the previous behavior is maintained.
[Other Info]
None
[0] https:/
tags: | added: sts |
description: | updated |
summary: |
- Cannot create instance with multiqueue image and vif_type=tap (calico) + [SRU] Cannot create instance with multiqueue image and vif_type=tap + (calico) |
tags: | added: sts-sru-needed |
Changed in nova (Ubuntu): | |
status: | New → Fix Released |
Fix proposed to branch: master /review. opendev. org/748533
Review: https:/