Activity log for bug #1931244

Date Who What changed Old value New value Message
2021-06-08 11:56:58 Edward Hope-Morley bug added bug
2021-06-08 13:32:37 Brian Haley neutron: importance Undecided High
2021-06-08 13:33:45 Brian Haley neutron: assignee Terry Wilson (otherwiseguy)
2021-06-08 13:34:33 Brian Haley tags ovn
2021-06-08 13:34:53 Brian Haley tags ovn ovn sriov-pci-pt
2021-06-08 13:44:31 Corey Bryant bug task added neutron (Ubuntu)
2021-06-08 13:44:58 Corey Bryant neutron (Ubuntu): status New Triaged
2021-06-08 13:45:04 Corey Bryant neutron (Ubuntu): importance Undecided High
2021-06-08 13:47:17 Corey Bryant bug task added cloud-archive
2021-06-08 13:47:36 Corey Bryant cloud-archive: status New Triaged
2021-06-08 13:47:41 Corey Bryant cloud-archive: importance Undecided High
2021-06-08 14:09:36 Corey Bryant nominated for series cloud-archive/victoria
2021-06-08 14:09:36 Corey Bryant bug task added cloud-archive/victoria
2021-06-08 14:09:36 Corey Bryant nominated for series cloud-archive/ussuri
2021-06-08 14:09:36 Corey Bryant bug task added cloud-archive/ussuri
2021-06-08 14:09:36 Corey Bryant nominated for series cloud-archive/xena
2021-06-08 14:09:36 Corey Bryant bug task added cloud-archive/xena
2021-06-08 14:09:36 Corey Bryant nominated for series cloud-archive/wallaby
2021-06-08 14:09:36 Corey Bryant bug task added cloud-archive/wallaby
2021-06-08 14:09:56 Corey Bryant cloud-archive/wallaby: status New Triaged
2021-06-08 14:10:01 Corey Bryant cloud-archive/ussuri: status New Triaged
2021-06-08 14:10:06 Corey Bryant cloud-archive/victoria: status New Triaged
2021-06-08 14:10:15 Corey Bryant cloud-archive/wallaby: importance Undecided High
2021-06-08 14:10:22 Corey Bryant cloud-archive/ussuri: importance Undecided High
2021-06-08 14:10:27 Corey Bryant cloud-archive/victoria: importance Undecided High
2021-06-08 14:10:53 Corey Bryant nominated for series Ubuntu Groovy
2021-06-08 14:10:53 Corey Bryant bug task added neutron (Ubuntu Groovy)
2021-06-08 14:10:53 Corey Bryant nominated for series Ubuntu Hirsute
2021-06-08 14:10:53 Corey Bryant bug task added neutron (Ubuntu Hirsute)
2021-06-08 14:10:53 Corey Bryant nominated for series Ubuntu Focal
2021-06-08 14:10:53 Corey Bryant bug task added neutron (Ubuntu Focal)
2021-06-08 14:10:53 Corey Bryant nominated for series Ubuntu Impish
2021-06-08 14:10:53 Corey Bryant bug task added neutron (Ubuntu Impish)
2021-06-08 14:11:11 Corey Bryant neutron (Ubuntu Focal): importance Undecided High
2021-06-08 14:11:11 Corey Bryant neutron (Ubuntu Focal): status New Triaged
2021-06-08 14:11:30 Corey Bryant neutron (Ubuntu Groovy): importance Undecided High
2021-06-08 14:11:30 Corey Bryant neutron (Ubuntu Groovy): status New Triaged
2021-06-08 14:12:15 Corey Bryant neutron (Ubuntu Hirsute): importance Undecided High
2021-06-08 14:12:15 Corey Bryant neutron (Ubuntu Hirsute): status New Triaged
2021-06-08 14:29:08 Corey Bryant description I have an Openstack Ussuri 16.3.2 deployment using OVN. When I create a vm with one or more sriov ports it fails with: 2021-06-08 11:38:31.939 526862 WARNING nova.virt.libvirt.driver [req-c4be797e-7d7e-4e73-8406-f74ae51db192 696c98b722a44d229e16b6d6474a27d4 0b9102977dcc4d4ab662b48494bb3110 - 2e0bf6ec95c047d986a61f7570222149 2e0bf6ec95c047d986a61f7570222149] [instance: 7ab9b374-51eb-4e94-8055-c69e8a7d76b3] Timeout waiting for [('network-vif-plugged', 'c2b7c68d-c465-4ca2-869a-59bc73b2b595'), ('network-vif-plugged', 'a50de16a-29ac-4dca-9cb6-0247a932fbf3')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds A bit of analysis shows that nova-compute did its thing and sits there waiting on network-vif-plugged. The sriov-agent then notices new VFs configured and sends a get_devices_details_list() rpc call to neutron and neutron never responds. Reverting to 16.3.1 fixes the issue. Taking a closer look at 16.3.2 by reverting patches lead to [1] as the culprit. [1] https://github.com/openstack/neutron/commit/7cf9597570f288d27768dc5ff7be04824d09f8bc I have an Openstack Ussuri 16.3.2 deployment using OVN. When I create a vm with one or more sriov ports it fails with: 2021-06-08 11:38:31.939 526862 WARNING nova.virt.libvirt.driver [req-c4be797e-7d7e-4e73-8406-f74ae51db192 696c98b722a44d229e16b6d6474a27d4 0b9102977dcc4d4ab662b48494bb3110 - 2e0bf6ec95c047d986a61f7570222149 2e0bf6ec95c047d986a61f7570222149] [instance: 7ab9b374-51eb-4e94-8055-c69e8a7d76b3] Timeout waiting for [('network-vif-plugged', 'c2b7c68d-c465-4ca2-869a-59bc73b2b595'), ('network-vif-plugged', 'a50de16a-29ac-4dca-9cb6-0247a932fbf3')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds A bit of analysis shows that nova-compute did its thing and sits there waiting on network-vif-plugged. The sriov-agent then notices new VFs configured and sends a get_devices_details_list() rpc call to neutron and neutron never responds. Reverting to 16.3.1 fixes the issue. Taking a closer look at 16.3.2 by reverting patches lead to [1] as the culprit. [1] https://github.com/openstack/neutron/commit/7cf9597570f288d27768dc5ff7be04824d09f8bc === Ubuntu SRU details === [Impact] [Test Case] See above. For now we are planning to revert the commit as a stop-gap to prevent further upgrades from being regressed. [Regression Potential] There is regression potential in that the patch being reverted contributes partial fixes to the following related bugs. Considering most openstack users are on ussuri at this point and 16.3.2 has not been available for very long the revert that we are proposing would seem to have the least amount of regression potential. https://bugs.launchpad.net/neutron/+bug/1894117 https://bugs.launchpad.net/neutron/+bug/1903008
2021-06-08 14:31:48 Corey Bryant description I have an Openstack Ussuri 16.3.2 deployment using OVN. When I create a vm with one or more sriov ports it fails with: 2021-06-08 11:38:31.939 526862 WARNING nova.virt.libvirt.driver [req-c4be797e-7d7e-4e73-8406-f74ae51db192 696c98b722a44d229e16b6d6474a27d4 0b9102977dcc4d4ab662b48494bb3110 - 2e0bf6ec95c047d986a61f7570222149 2e0bf6ec95c047d986a61f7570222149] [instance: 7ab9b374-51eb-4e94-8055-c69e8a7d76b3] Timeout waiting for [('network-vif-plugged', 'c2b7c68d-c465-4ca2-869a-59bc73b2b595'), ('network-vif-plugged', 'a50de16a-29ac-4dca-9cb6-0247a932fbf3')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds A bit of analysis shows that nova-compute did its thing and sits there waiting on network-vif-plugged. The sriov-agent then notices new VFs configured and sends a get_devices_details_list() rpc call to neutron and neutron never responds. Reverting to 16.3.1 fixes the issue. Taking a closer look at 16.3.2 by reverting patches lead to [1] as the culprit. [1] https://github.com/openstack/neutron/commit/7cf9597570f288d27768dc5ff7be04824d09f8bc === Ubuntu SRU details === [Impact] [Test Case] See above. For now we are planning to revert the commit as a stop-gap to prevent further upgrades from being regressed. [Regression Potential] There is regression potential in that the patch being reverted contributes partial fixes to the following related bugs. Considering most openstack users are on ussuri at this point and 16.3.2 has not been available for very long the revert that we are proposing would seem to have the least amount of regression potential. https://bugs.launchpad.net/neutron/+bug/1894117 https://bugs.launchpad.net/neutron/+bug/1903008 I have an Openstack Ussuri 16.3.2 deployment using OVN. When I create a vm with one or more sriov ports it fails with: 2021-06-08 11:38:31.939 526862 WARNING nova.virt.libvirt.driver [req-c4be797e-7d7e-4e73-8406-f74ae51db192 696c98b722a44d229e16b6d6474a27d4 0b9102977dcc4d4ab662b48494bb3110 - 2e0bf6ec95c047d986a61f7570222149 2e0bf6ec95c047d986a61f7570222149] [instance: 7ab9b374-51eb-4e94-8055-c69e8a7d76b3] Timeout waiting for [('network-vif-plugged', 'c2b7c68d-c465-4ca2-869a-59bc73b2b595'), ('network-vif-plugged', 'a50de16a-29ac-4dca-9cb6-0247a932fbf3')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds A bit of analysis shows that nova-compute did its thing and sits there waiting on network-vif-plugged. The sriov-agent then notices new VFs configured and sends a get_devices_details_list() rpc call to neutron and neutron never responds. Reverting to 16.3.1 fixes the issue. Taking a closer look at 16.3.2 by reverting patches lead to [1] as the culprit. [1] https://github.com/openstack/neutron/commit/7cf9597570f288d27768dc5ff7be04824d09f8bc === Ubuntu SRU details === [Impact] [Test Case] See above. I think for testing we can run standard regression testing with OVN/neutron deployments plus tempest testing. For now we are planning to revert the commit as a stop-gap to prevent further upgrades from being regressed. [Regression Potential] There is regression potential in that the patch being reverted contributes partial fixes to the following related bugs. Considering most openstack users are on ussuri at this point and 16.3.2 has not been available for very long the revert that we are proposing would seem to have the least amount of regression potential. https://bugs.launchpad.net/neutron/+bug/1894117 https://bugs.launchpad.net/neutron/+bug/1903008
2021-06-08 16:54:24 Corey Bryant cloud-archive/victoria: importance High Undecided
2021-06-08 16:54:24 Corey Bryant cloud-archive/victoria: status Triaged Invalid
2021-06-08 16:54:43 Corey Bryant neutron (Ubuntu Groovy): status Triaged Invalid
2021-06-08 18:01:12 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2021-06-08 18:35:26 Brian Murray neutron (Ubuntu Hirsute): status Triaged Fix Committed
2021-06-08 18:35:31 Brian Murray bug added subscriber SRU Verification
2021-06-08 18:35:39 Brian Murray tags ovn sriov-pci-pt ovn sriov-pci-pt verification-needed verification-needed-hirsute
2021-06-08 18:43:52 Brian Murray neutron (Ubuntu Focal): status Triaged Fix Committed
2021-06-08 18:44:02 Brian Murray tags ovn sriov-pci-pt verification-needed verification-needed-hirsute ovn sriov-pci-pt verification-needed verification-needed-focal verification-needed-hirsute
2021-06-08 22:25:27 Dominique Poulain bug added subscriber Dominique Poulain
2021-06-08 23:50:59 James Troup bug added subscriber Canonical IS CREs
2021-06-09 02:54:20 Mathew Hodson bug task deleted neutron (Ubuntu Groovy)
2021-06-09 05:16:40 OpenStack Infra neutron: status New In Progress
2021-06-09 12:16:59 Corey Bryant cloud-archive: status Triaged Fix Committed
2021-06-09 12:18:02 Corey Bryant cloud-archive/ussuri: status Triaged Fix Committed
2021-06-09 12:18:06 Corey Bryant tags ovn sriov-pci-pt verification-needed verification-needed-focal verification-needed-hirsute ovn sriov-pci-pt verification-needed verification-needed-focal verification-needed-hirsute verification-ussuri-needed
2021-06-09 12:18:15 Corey Bryant cloud-archive/wallaby: status Triaged Fix Committed
2021-06-11 11:42:52 Edward Hope-Morley tags ovn sriov-pci-pt verification-needed verification-needed-focal verification-needed-hirsute verification-ussuri-needed ovn sriov-pci-pt verification-needed verification-needed-focal verification-needed-hirsute verification-ussuri-done
2021-06-11 13:37:17 Hemanth Nakkina tags ovn sriov-pci-pt verification-needed verification-needed-focal verification-needed-hirsute verification-ussuri-done ovn sriov-pci-pt verification-done-focal verification-needed verification-needed-hirsute verification-ussuri-done
2021-06-14 04:06:51 Hemanth Nakkina tags ovn sriov-pci-pt verification-done-focal verification-needed verification-needed-hirsute verification-ussuri-done ovn sriov-pci-pt verification-done-focal verification-done-hirsute verification-needed verification-ussuri-done
2021-06-14 06:41:25 Nobuto Murata bug added subscriber Nobuto Murata
2021-06-15 04:11:39 Hemanth Nakkina tags ovn sriov-pci-pt verification-done-focal verification-done-hirsute verification-needed verification-ussuri-done ovn sriov-pci-pt verification-done verification-done-focal verification-done-hirsute verification-ussuri-done verification-wallaby-done
2021-06-17 00:21:58 Launchpad Janitor neutron (Ubuntu Hirsute): status Fix Committed Fix Released
2021-06-17 00:22:16 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2021-06-17 00:26:13 Launchpad Janitor neutron (Ubuntu Focal): status Fix Committed Fix Released
2021-06-17 18:18:42 Corey Bryant cloud-archive: status Fix Committed Fix Released
2021-06-17 18:46:55 Corey Bryant cloud-archive/wallaby: status Fix Committed Fix Released
2021-06-17 18:51:02 Corey Bryant neutron (Ubuntu Impish): status Triaged Fix Released
2021-06-17 18:53:47 Corey Bryant cloud-archive/ussuri: status Fix Committed Fix Released
2021-06-24 15:28:42 Rodolfo Alonso neutron: status In Progress Fix Released