[SRU] Cannot create instance with multiqueue image and vif_type=tap (calico)

[Rodrigo Barbieri]

When using calico, the vif_type is tap, therefore when the instance is being created, the method plug_tap() is invoked, which creates the tap device prior to launching the instance.

That tap device is currently always created without multiqueue as per [1]. When libvirt creates the instance, the XML definition "queues=<x>" clashes with the fact that the pre-existing tap interface doesn't have multiqueue enabled, and therefore errors out with the exception below. The code at [2] already handles multiqueue, but it is never invoked with multiqueue=True.

Alternatively, as a current workaround, if the instance is shutdown through virsh, or rebooted through nova, it causes the tap device to be removed, to be created again by libvirt instead, allowing the tap device to be set up with multiqueue appropriately if its XML is manually edited. This begs the question as why the plug_tap() method needs to pre-create the interface at all, if when the VM rebooted libvirt does so regardless of plug_tap().

Steps to reproduce:

1) Ubuntu bionic + devstack master + follow instructions at [3]
2) wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
3) openstack image create bionic-mq --file bionic-server-cloudimg-amd64.img --property hw_vif_multiqueue_enabled=True
4) openstack image create bionic --file bionic-server-cloudimg-amd64.img
5) ssh-keygen
6) openstack keypair create key1 --public-key ~/.ssh/id_rsa.pub
7) openstack flavor create --vcpu 2 --ram 1024 --disk 10 --public --id 10 test_flavor
8) openstack server create --network calico --flavor test_flavor --image bionic --key-name key1 no-mq

instance is created successfully

9) ip a

6: tapcc353751-13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

10) sudo virsh edit 1

add "<driver name='vhost' queues='2'/>" to the interface section

11) openstack server reboot no-mq

wait a few secs

12) ip a

7: tapcc353751-13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN group default qlen 1000

13) ssh to the instance and run "sudo ethtool -l <interface>"

Combined: 2

14) openstack server delete no-mq

15) openstack server create --network calico --flavor test_flavor --image bionic-mq --key-name key1 mq

instance fails to be created, log shows the below stack trace.

[1] https://github.com/openstack/nova/blob/f521f4dbace0e35bedd089369da6f6969da5ca32/nova/virt/libvirt/vif.py#L701
[2] https://github.com/openstack/nova/blob/f521f4dbace0e35bedd089369da6f6969da5ca32/nova/privsep/linux_net.py#L109
[3] https://docs.projectcalico.org/getting-started/openstack/installation/devstack

Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [None req-71d40776-0fa7-466e-9060-11472b5bce42 admin admin] [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] Instance failed to spawn: libvirt.libvirtError: Unable to create tap device tapb6021dd0-fd: Invalid argument
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] Traceback (most recent call last):
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/compute/manager.py", line 2628, in _build_resources
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] yield resources
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/compute/manager.py", line 2401, in _build_and_run_instance
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] accel_info=accel_info)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3701, in spawn
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] cleanup_instance_disks=created_disks)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6700, in _create_guest_with_network
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] cleanup_instance_disks=cleanup_instance_disks)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] self.force_reraise()
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] six.reraise(self.type_, self.value, self.tb)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] raise value
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6669, in _create_guest_with_network
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] post_xml_callback=post_xml_callback)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6599, in _create_guest
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] guest.launch(pause=pause)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 160, in launch
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] self._encoded_xml, errors='ignore')
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] self.force_reraise()
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] six.reraise(self.type_, self.value, self.tb)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] raise value
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 155, in launch
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] return self._domain.createWithFlags(flags)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] result = proxy_call(self._autowrap, f, *args, **kwargs)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 148, in proxy_call
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] rv = execute(f, *args, **kwargs)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 129, in execute
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] six.reraise(c, e, tb)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] raise value
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 83, in tworker
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] rv = meth(*args, **kwargs)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 1098, in createWithFlags
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] libvirt.libvirtError: Unable to create tap device tapb6021dd0-fd: Invalid argument
Aug 27 18:58:38 devstack nova-compute[7968]: ERROR nova.compute.manager [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4]
Aug 27 18:58:38 devstack nova-compute[7968]: INFO nova.compute.manager [None req-71d40776-0fa7-466e-9060-11472b5bce42 admin admin] [instance: 69a0a527-9c33-432f-8889-c421ae8aebb4] Terminating instance

=======================================================================

[Impact]

Users of calico plugin cannot use multiqueue in Nova. The VM fails to boot. The workaround is to edit the XML manually and reboot it through nova, so the tap interface is recreated by libvirt while the vif.plug() method is not re-run by Nova, allowing multiqueue to be set up properly by libvirt. This workaround does not scale well.

[Test case]

1. Setting up env
1a. Deploy environment
1b. Install calico plugin as per [0]
1c. Setup SSH

ssh-keygen

1d. Create keypair for testing

openstack keypair create key1 --public-key ~/.ssh/id_rsa.pub

1e. Create test flavor

openstack flavor create --vcpu 2 --ram 1024 --disk 10 --public --id 10 test_flavor

1f. Download an example image

wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img

1g. Create image in glance with multiqueue metadata

openstack image create bionic-mq --file bionic-server-cloudimg-amd64.img --property hw_vif_multiqueue_enabled=True

1h. Create same image in glance without multiqueue metadata

openstack image create bionic --file bionic-server-cloudimg-amd64.img

1f. Create instance without multiqueue. Make sure instance creation and connectivity succeeds.

openstack server create --network calico --flavor test_flavor --image bionic --key-name key1 no-mq

2. Reproducing the bug

2a. Create instance with multiqueue

openstack server create --network calico --flavor test_flavor --image bionic-mq --key-name key1 mq

Instance creation will fail

2b. Check logs for error

egrep "libvirt.libvirtError: Unable to create tap device .*: Invalid argument" /var/log/nova/nova-compute.log

3. Cleanup

3a. Delete instances "mq" and "no-mq"

4. Install package that contains the fixed code

5. Repeat step 2a. 2a should now succeed.

[Regression Potential]

New Code path is not triggered if image metadata is not used. For all other use cases, the previous behavior is maintained.

[Other Info]

None

[0] https://docs.projectcalico.org/getting-started/openstack/installation/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/748533

[sean mooney (sean-k-mooney)]

triaging this as medium this affect all release where vnic_type=tap and multiqueue are supported

[sean mooney (sean-k-mooney)]

sts is not a tag we use, what is it for?

[Rodrigo Barbieri (rodrigo-barbieri2010)]

@sean, regarding "sts" tag: I have notification rules associated with this tag for me and my team mates

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748533
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=84cfc8e9ab1396ec17abcfc9646c7d40f1d966ae
Submitter: Zuul
Branch: master

commit 84cfc8e9ab1396ec17abcfc9646c7d40f1d966ae
Author: Rodrigo Barbieri <email address hidden>
Date: Thu Aug 27 17:20:19 2020 -0300

    Allow tap interface with multiqueue

    When vif_type="tap" (such as when using calico),
    attempting to create an instance using an image that has
    the property hw_vif_multiqueue_enabled=True fails, because
    the interface is always being created without multiqueue
    flags.

    This change checks if the property is defined and passes
    the multiqueue parameter to create the tap interface
    accordingly.

    In case the multiqueue parameter is passed but the
    vif_model is not virtio (or unspecified), the old
    behavior is maintained.

    Change-Id: I0307c43dcd0cace1620d2ac75925651d4ee2e96c
    Closes-bug: #1893263

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/752245

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/752245
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a69845f3732843ee1451b2e4ebf547d9801e898d
Submitter: Zuul
Branch: stable/ussuri

commit a69845f3732843ee1451b2e4ebf547d9801e898d
Author: Rodrigo Barbieri <email address hidden>
Date: Thu Aug 27 17:20:19 2020 -0300

    Allow tap interface with multiqueue

    When vif_type="tap" (such as when using calico),
    attempting to create an instance using an image that has
    the property hw_vif_multiqueue_enabled=True fails, because
    the interface is always being created without multiqueue
    flags.

    This change checks if the property is defined and passes
    the multiqueue parameter to create the tap interface
    accordingly.

    In case the multiqueue parameter is passed but the
    vif_model is not virtio (or unspecified), the old
    behavior is maintained.

    Conflicts:
        nova/virt/libvirt/vif.py

    NOTE: The conflict is due to not having patch
    Iefa6009874395cafe3121e44a48d882c38230d16 in stable/ussuri.

    Change-Id: I0307c43dcd0cace1620d2ac75925651d4ee2e96c
    Closes-bug: #1893263
    (cherry picked from commit 84cfc8e9ab1396ec17abcfc9646c7d40f1d966ae)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/757870

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/757886

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757870
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=750655c19daf4e8d0dcf35d3b1cab5e6d8ac69ba
Submitter: Zuul
Branch: stable/train

commit 750655c19daf4e8d0dcf35d3b1cab5e6d8ac69ba
Author: Rodrigo Barbieri <email address hidden>
Date: Thu Aug 27 17:20:19 2020 -0300

    Allow tap interface with multiqueue

    When vif_type="tap" (such as when using calico),
    attempting to create an instance using an image that has
    the property hw_vif_multiqueue_enabled=True fails, because
    the interface is always being created without multiqueue
    flags.

    This change checks if the property is defined and passes
    the multiqueue parameter to create the tap interface
    accordingly.

    In case the multiqueue parameter is passed but the
    vif_model is not virtio (or unspecified), the old
    behavior is maintained.

    Change-Id: I0307c43dcd0cace1620d2ac75925651d4ee2e96c
    Closes-bug: #1893263
    (cherry picked from commit 84cfc8e9ab1396ec17abcfc9646c7d40f1d966ae)
    (cherry picked from commit a69845f3732843ee1451b2e4ebf547d9801e898d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757886
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8699156d86e0a40b25dd4e22092654b8bc14cccf
Submitter: Zuul
Branch: stable/stein

commit 8699156d86e0a40b25dd4e22092654b8bc14cccf
Author: Rodrigo Barbieri <email address hidden>
Date: Thu Aug 27 17:20:19 2020 -0300

    Allow tap interface with multiqueue

    When vif_type="tap" (such as when using calico),
    attempting to create an instance using an image that has
    the property hw_vif_multiqueue_enabled=True fails, because
    the interface is always being created without multiqueue
    flags.

    This change checks if the property is defined and passes
    the multiqueue parameter to create the tap interface
    accordingly.

    In case the multiqueue parameter is passed but the
    vif_model is not virtio (or unspecified), the old
    behavior is maintained.

    Conflicts:
      nova/virt/libvirt/vif.py

    NOTE: The conflict is for not having change
    Iab16a15a5f141a7f7aae4172b092c50ee7964597 in stable/stein.

    Change-Id: I0307c43dcd0cace1620d2ac75925651d4ee2e96c
    Closes-bug: #1893263
    (cherry picked from commit 84cfc8e9ab1396ec17abcfc9646c7d40f1d966ae)
    (cherry picked from commit a69845f3732843ee1451b2e4ebf547d9801e898d)
    (cherry picked from commit 750655c19daf4e8d0dcf35d3b1cab5e6d8ac69ba)

[Rodrigo Barbieri (rodrigo-barbieri2010)]

debdiff for SRU focal-ussuri

[Rodrigo Barbieri (rodrigo-barbieri2010)]

debdiff for SRU bionic-train

[Rodrigo Barbieri (rodrigo-barbieri2010)]

debdiff for SRU bionic-stein

[Chris MacNaughton (chris.macnaughton)]

Removed the Victoria task for this as the fix shipped in Victoria, and was backported from there.

[Chris MacNaughton (chris.macnaughton)]

Hello Rodrigo, or anyone else affected,

Accepted nova into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris MacNaughton (chris.macnaughton)]

Hello Rodrigo, or anyone else affected,

Accepted nova into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Rodrigo, or anyone else affected,

Accepted nova into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:21.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris MacNaughton (chris.macnaughton)]

Hello Rodrigo, or anyone else affected,

Accepted nova into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Rodrigo Barbieri (rodrigo-barbieri2010)]

I was able to fix the issue I was having by forcing the hostname in /etc/felix/felix.cfg using the property:

FelixHostname = <full_hostname>

I've successfully validated bionic-stein and bionic-train, however, I cannot install the calico packages on focal:

root@juju-3294b7-calico-ussuri-6:/home/ubuntu# add-apt-repository ppa:project-calico/calico-3.17
 This PPA contains packages for Calico's 3.17.x release series, mainly to support OpenStack-based systems (although the Felix package can be used stand-alone). It will be updated with patch releases for that series.
 More info: https://launchpad.net/~project-calico/+archive/ubuntu/calico-3.17
Press [ENTER] to continue or Ctrl-c to cancel adding it.

Hit:1 http://nova.clouds.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://nova.clouds.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease
Ign:5 http://ppa.launchpad.net/project-calico/calico-3.17/ubuntu focal InRelease
Err:6 http://ppa.launchpad.net/project-calico/calico-3.17/ubuntu focal Release
  404 Not Found [IP: 91.189.95.83 80]
Reading package lists... Done
E: The repository 'http://ppa.launchpad.net/project-calico/calico-3.17/ubuntu focal Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Slightly more context on my previous comment: When I said " was able to fix the issue I was having by forcing the hostname in /etc/felix/felix.cfg using the property" I was referring to a problem I was having setting up the environment to validate the SRU, not the main issue itself.

[Mauricio Faria de Oliveira (mfo)]

> I've successfully validated bionic-stein and bionic-train, however, I cannot install the calico packages on focal:
> ...
> E: The repository 'http://ppa.launchpad.net/project-calico/calico-3.17/ubuntu focal Release' does not have a Release file.

Apparently the issue is that the PPA does not build/have Focal yet (no '/dists/focal/', only trusty/xenial/bionic) in its URL:

http://ppa.launchpad.net/project-calico/calico-3.17/ubuntu/

[Chris MacNaughton (chris.macnaughton)]

This is going to be resolved via an upstream point release for Nova that is proposed in https://bugs.launchpad.net/cloud-archive/+bug/1906271

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:21.1.1-0ubuntu2

---------------
nova (2:21.1.1-0ubuntu2) focal; urgency=medium

  * d/p/lp1892361.patch: Update pci stat pools based on PCI device changes (LP: #1892361).

 -- Chris MacNaughton <email address hidden> Mon, 18 Jan 2021 15:25:16 +0000

[Chris MacNaughton (chris.macnaughton)]

Marking this fix-released for the cloud archive targets as the related point release has landed!

[Rodrigo Barbieri (rodrigo-barbieri2010)]

[0] has been fixed through PRs [1] and [2], and master packages built for focal are available at [3].

I tested them on a new focal-ussuri deployment, therefore already having the fix from the point release [4], and it works.

Attaching validation for focal-ussuri, and previous validations for stein and train that I had done before.

[0] https://github.com/projectcalico/calico/issues/4216
[1] https://github.com/projectcalico/networking-calico/pull/48
[2] https://github.com/projectcalico/packaging/pull/39
[3] https://launchpad.net/~project-calico/+archive/ubuntu/master
[4] https://bugs.launchpad.net/cloud-archive/+bug/1906271