| 2022-05-13 07:50:27 |
Gregory Thiemonge |
bug |
|
|
added bug |
| 2022-05-13 07:51:42 |
Rodolfo Alonso |
neutron: importance |
Undecided |
High |
|
| 2022-05-13 07:51:48 |
Rodolfo Alonso |
neutron: status |
New |
Confirmed |
|
| 2022-05-13 07:57:12 |
Rodolfo Alonso |
neutron: status |
Confirmed |
New |
|
| 2022-05-13 08:28:45 |
Rodolfo Alonso |
neutron: status |
New |
Invalid |
|
| 2022-05-17 08:29:51 |
Rodolfo Alonso |
neutron: status |
Invalid |
In Progress |
|
| 2022-05-17 08:29:55 |
Rodolfo Alonso |
neutron: importance |
High |
Medium |
|
| 2022-05-17 08:29:57 |
Rodolfo Alonso |
neutron: assignee |
|
Rodolfo Alonso (rodolfo-alonso-hernandez) |
|
| 2022-05-18 09:35:45 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2022-05-23 09:16:41 |
Lajos Katona |
tags |
|
ovn |
|
| 2022-05-24 15:43:36 |
OpenStack Infra |
tags |
ovn |
in-stable-yoga ovn |
|
| 2022-05-30 12:01:17 |
OpenStack Infra |
tags |
in-stable-yoga ovn |
in-stable-wallaby in-stable-yoga ovn |
|
| 2022-05-30 15:29:53 |
OpenStack Infra |
tags |
in-stable-wallaby in-stable-yoga ovn |
in-stable-wallaby in-stable-xena in-stable-yoga ovn |
|
| 2022-06-02 09:06:21 |
OpenStack Infra |
tags |
in-stable-wallaby in-stable-xena in-stable-yoga ovn |
in-stable-ussuri in-stable-wallaby in-stable-xena in-stable-yoga ovn |
|
| 2022-06-02 20:35:19 |
OpenStack Infra |
tags |
in-stable-ussuri in-stable-wallaby in-stable-xena in-stable-yoga ovn |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn |
|
| 2022-06-12 01:02:59 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2022-08-23 13:08:56 |
Dmitrii Shcherbakov |
bug task added |
|
neutron (Ubuntu) |
|
| 2022-08-23 13:32:54 |
Edward Hope-Morley |
bug task added |
|
cloud-archive |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/xena |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/xena |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/yoga |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/yoga |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/zed |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/zed |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/ussuri |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/ussuri |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/victoria |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/victoria |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/wallaby |
|
| 2022-08-23 13:33:08 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/wallaby |
|
| 2022-08-23 14:05:38 |
Edward Hope-Morley |
cloud-archive/zed: status |
New |
Fix Released |
|
| 2022-08-23 14:05:45 |
Edward Hope-Morley |
cloud-archive/yoga: status |
New |
Fix Released |
|
| 2022-08-23 15:15:23 |
Edward Hope-Morley |
attachment added |
|
lp1973276-ussuri.debdiff https://bugs.launchpad.net/neutron/+bug/1973276/+attachment/5611003/+files/lp1973276-ussuri.debdiff |
|
| 2022-08-23 15:15:55 |
Edward Hope-Morley |
attachment added |
|
lp1973276-victoria.debdiff https://bugs.launchpad.net/neutron/+bug/1973276/+attachment/5611004/+files/lp1973276-victoria.debdiff |
|
| 2022-08-23 16:25:06 |
Ubuntu Foundations Team Bug Bot |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch |
|
| 2022-08-23 16:25:13 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2022-09-15 19:44:47 |
Corey Bryant |
cloud-archive/xena: status |
New |
Fix Released |
|
| 2022-09-15 19:44:51 |
Corey Bryant |
cloud-archive/wallaby: status |
New |
Fix Released |
|
| 2022-09-15 19:45:03 |
Corey Bryant |
cloud-archive/victoria: status |
New |
Triaged |
|
| 2022-09-15 19:47:00 |
Corey Bryant |
cloud-archive/ussuri: status |
New |
Triaged |
|
| 2022-09-15 19:47:05 |
Corey Bryant |
cloud-archive/victoria: importance |
Undecided |
High |
|
| 2022-09-15 19:47:08 |
Corey Bryant |
cloud-archive/ussuri: importance |
Undecided |
High |
|
| 2022-09-15 19:47:45 |
Corey Bryant |
nominated for series |
|
Ubuntu Focal |
|
| 2022-09-15 19:47:45 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Focal) |
|
| 2022-09-15 19:48:01 |
Corey Bryant |
neutron (Ubuntu): status |
New |
Fix Released |
|
| 2022-09-15 19:48:16 |
Corey Bryant |
neutron (Ubuntu Focal): importance |
Undecided |
High |
|
| 2022-09-15 19:48:16 |
Corey Bryant |
neutron (Ubuntu Focal): status |
New |
Triaged |
|
| 2022-09-15 19:49:14 |
Corey Bryant |
description |
Bug found in Octavia (master)
Octavia creates at least 2 ports for each load balancer:
- the VIP port, it is down, it keeps/stores the IP address of the LB
- the VRRP port, plugged into a VM, it has the VIP address in the allowed-address list (and the VIP address is configured on the interface in the VM)
When sending an ARP request for the VIP address, the VRRP port should reply with its mac-address.
In OVN the VIP port is marked as "type: virtual".
But when the VIP port is updated, it loses its "port: virtual" status and that breaks the ARP resolution (OVN replies to the ARP request by sending the mac-address of the VIP port - which is not used/down).
Quick reproducer that simulates the Octavia behavior:
===========================
import subprocess
import time
import openstack
conn = openstack.connect(cloud="devstack-admin-demo")
network = conn.network.find_network("public")
sg = conn.network.find_security_group('sg')
if not sg:
sg = conn.network.create_security_group(name='sg')
vip_port = conn.network.create_port(
name="lb-vip",
network_id=network.id,
device_id="lb-1",
device_owner="me",
is_admin_state_up=False)
vip_address = [
fixed_ip['ip_address']
for fixed_ip in vip_port.fixed_ips
if '.' in fixed_ip['ip_address']][0]
vrrp_port = conn.network.create_port(
name="lb-vrrp",
device_id="vrrp",
device_owner="vm",
network_id=network.id)
vrrp_port = conn.network.update_port(
vrrp_port,
allowed_address_pairs=[
{"ip_address": vip_address,
"mac_address": vrrp_port.mac_address}])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' in output:
print("Port is virtual, this is ok.")
print(output)
conn.network.update_port(
vip_port,
security_group_ids=[sg.id])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' not in output:
print("Port is not virtual, this is an issue.")
print(output)
===========================
In my env (devstack master on c9s):
$ python3 /mnt/host/virtual_port_issue.py
Port is virtual, this is ok.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
type: virtual
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
Port is not virtual, this is an issue.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
port 8ec36278-82b1-436b-bc5e-ea03ef22192f
In Octavia, the "port: virtual" is _sometimes_ back after other updates of the ports, but in some cases the LB is unreachable.
(and "ovn-nbctl lsp-set-type <vip-port-id> virtual" fixes the LB) |
Bug found in Octavia (master)
Octavia creates at least 2 ports for each load balancer:
- the VIP port, it is down, it keeps/stores the IP address of the LB
- the VRRP port, plugged into a VM, it has the VIP address in the allowed-address list (and the VIP address is configured on the interface in the VM)
When sending an ARP request for the VIP address, the VRRP port should reply with its mac-address.
In OVN the VIP port is marked as "type: virtual".
But when the VIP port is updated, it loses its "port: virtual" status and that breaks the ARP resolution (OVN replies to the ARP request by sending the mac-address of the VIP port - which is not used/down).
Quick reproducer that simulates the Octavia behavior:
===========================
import subprocess
import time
import openstack
conn = openstack.connect(cloud="devstack-admin-demo")
network = conn.network.find_network("public")
sg = conn.network.find_security_group('sg')
if not sg:
sg = conn.network.create_security_group(name='sg')
vip_port = conn.network.create_port(
name="lb-vip",
network_id=network.id,
device_id="lb-1",
device_owner="me",
is_admin_state_up=False)
vip_address = [
fixed_ip['ip_address']
for fixed_ip in vip_port.fixed_ips
if '.' in fixed_ip['ip_address']][0]
vrrp_port = conn.network.create_port(
name="lb-vrrp",
device_id="vrrp",
device_owner="vm",
network_id=network.id)
vrrp_port = conn.network.update_port(
vrrp_port,
allowed_address_pairs=[
{"ip_address": vip_address,
"mac_address": vrrp_port.mac_address}])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' in output:
print("Port is virtual, this is ok.")
print(output)
conn.network.update_port(
vip_port,
security_group_ids=[sg.id])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' not in output:
print("Port is not virtual, this is an issue.")
print(output)
===========================
In my env (devstack master on c9s):
$ python3 /mnt/host/virtual_port_issue.py
Port is virtual, this is ok.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
type: virtual
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
Port is not virtual, this is an issue.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
port 8ec36278-82b1-436b-bc5e-ea03ef22192f
In Octavia, the "port: virtual" is _sometimes_ back after other updates of the ports, but in some cases the LB is unreachable.
(and "ovn-nbctl lsp-set-type <vip-port-id> virtual" fixes the LB)
=== Ubuntu SRU Details ===
[Impact]
See above
[Test Case]
See above
[Where things could go wrong] |
|
| 2022-09-15 19:51:52 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-09-16 12:24:10 |
Corey Bryant |
cloud-archive/victoria: status |
Triaged |
Fix Committed |
|
| 2022-09-16 12:24:14 |
Corey Bryant |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-victoria-needed |
|
| 2022-09-20 15:23:49 |
Edward Hope-Morley |
description |
Bug found in Octavia (master)
Octavia creates at least 2 ports for each load balancer:
- the VIP port, it is down, it keeps/stores the IP address of the LB
- the VRRP port, plugged into a VM, it has the VIP address in the allowed-address list (and the VIP address is configured on the interface in the VM)
When sending an ARP request for the VIP address, the VRRP port should reply with its mac-address.
In OVN the VIP port is marked as "type: virtual".
But when the VIP port is updated, it loses its "port: virtual" status and that breaks the ARP resolution (OVN replies to the ARP request by sending the mac-address of the VIP port - which is not used/down).
Quick reproducer that simulates the Octavia behavior:
===========================
import subprocess
import time
import openstack
conn = openstack.connect(cloud="devstack-admin-demo")
network = conn.network.find_network("public")
sg = conn.network.find_security_group('sg')
if not sg:
sg = conn.network.create_security_group(name='sg')
vip_port = conn.network.create_port(
name="lb-vip",
network_id=network.id,
device_id="lb-1",
device_owner="me",
is_admin_state_up=False)
vip_address = [
fixed_ip['ip_address']
for fixed_ip in vip_port.fixed_ips
if '.' in fixed_ip['ip_address']][0]
vrrp_port = conn.network.create_port(
name="lb-vrrp",
device_id="vrrp",
device_owner="vm",
network_id=network.id)
vrrp_port = conn.network.update_port(
vrrp_port,
allowed_address_pairs=[
{"ip_address": vip_address,
"mac_address": vrrp_port.mac_address}])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' in output:
print("Port is virtual, this is ok.")
print(output)
conn.network.update_port(
vip_port,
security_group_ids=[sg.id])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' not in output:
print("Port is not virtual, this is an issue.")
print(output)
===========================
In my env (devstack master on c9s):
$ python3 /mnt/host/virtual_port_issue.py
Port is virtual, this is ok.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
type: virtual
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
Port is not virtual, this is an issue.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
port 8ec36278-82b1-436b-bc5e-ea03ef22192f
In Octavia, the "port: virtual" is _sometimes_ back after other updates of the ports, but in some cases the LB is unreachable.
(and "ovn-nbctl lsp-set-type <vip-port-id> virtual" fixes the LB)
=== Ubuntu SRU Details ===
[Impact]
See above
[Test Case]
See above
[Where things could go wrong] |
Bug found in Octavia (master)
Octavia creates at least 2 ports for each load balancer:
- the VIP port, it is down, it keeps/stores the IP address of the LB
- the VRRP port, plugged into a VM, it has the VIP address in the allowed-address list (and the VIP address is configured on the interface in the VM)
When sending an ARP request for the VIP address, the VRRP port should reply with its mac-address.
In OVN the VIP port is marked as "type: virtual".
But when the VIP port is updated, it loses its "port: virtual" status and that breaks the ARP resolution (OVN replies to the ARP request by sending the mac-address of the VIP port - which is not used/down).
Quick reproducer that simulates the Octavia behavior:
===========================
import subprocess
import time
import openstack
conn = openstack.connect(cloud="devstack-admin-demo")
network = conn.network.find_network("public")
sg = conn.network.find_security_group('sg')
if not sg:
sg = conn.network.create_security_group(name='sg')
vip_port = conn.network.create_port(
name="lb-vip",
network_id=network.id,
device_id="lb-1",
device_owner="me",
is_admin_state_up=False)
vip_address = [
fixed_ip['ip_address']
for fixed_ip in vip_port.fixed_ips
if '.' in fixed_ip['ip_address']][0]
vrrp_port = conn.network.create_port(
name="lb-vrrp",
device_id="vrrp",
device_owner="vm",
network_id=network.id)
vrrp_port = conn.network.update_port(
vrrp_port,
allowed_address_pairs=[
{"ip_address": vip_address,
"mac_address": vrrp_port.mac_address}])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' in output:
print("Port is virtual, this is ok.")
print(output)
conn.network.update_port(
vip_port,
security_group_ids=[sg.id])
time.sleep(1)
output = subprocess.check_output(
f"sudo ovn-nbctl show | grep -A2 'port {vip_port.id}'",
shell=True)
output = output.decode('utf-8')
if 'type: virtual' not in output:
print("Port is not virtual, this is an issue.")
print(output)
===========================
In my env (devstack master on c9s):
$ python3 /mnt/host/virtual_port_issue.py
Port is virtual, this is ok.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
type: virtual
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
Port is not virtual, this is an issue.
port e0fe2894-e306-42d9-8c5e-6e77b77659e2 (aka lb-vip)
addresses: ["fa:16:3e:93:00:8f 172.24.4.111 2001:db8::178"]
port 8ec36278-82b1-436b-bc5e-ea03ef22192f
In Octavia, the "port: virtual" is _sometimes_ back after other updates of the ports, but in some cases the LB is unreachable.
(and "ovn-nbctl lsp-set-type <vip-port-id> virtual" fixes the LB)
=== Ubuntu SRU Details ===
[Impact]
This bug causes loadbalancer vip ports to lose their "virtual" type in ovn and results in broken connectivity to amphora vms after failover. There are two patches, one that fixes new ports and one that retroactively fixes existing ones. We are backporting the former since it is clean and simple but the latter does not apply cleanly so we will defer.
[Test Case]
* deploy openstack ussuri or victoria with neutron + ovn and octavia
* create a loadbalancer
* check ovn-nbctl for the vip port and check that type is virtual
* failover the loadbalancer
* check ovn-nbctl for the vip port and check that type is still virtual and that lb vip is reachable
[Where things could go wrong]
There are not anticipated to be any regressions from this backport. |
|
| 2022-09-23 16:14:51 |
Brian Murray |
neutron (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2022-09-23 16:14:58 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2022-09-23 16:15:02 |
Brian Murray |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-victoria-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-needed verification-needed-focal verification-victoria-needed |
|
| 2022-09-23 20:48:05 |
Corey Bryant |
cloud-archive/ussuri: status |
Triaged |
Fix Committed |
|
| 2022-09-23 20:48:09 |
Corey Bryant |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-needed verification-needed-focal verification-victoria-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed |
|
| 2022-10-03 12:55:57 |
Edward Hope-Morley |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-done-focal verification-needed verification-ussuri-needed verification-victoria-needed |
|
| 2022-10-03 15:00:39 |
Edward Hope-Morley |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-done-focal verification-needed verification-ussuri-needed verification-victoria-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-done-focal verification-needed verification-ussuri-needed verification-victoria-done |
|
| 2022-10-03 18:49:58 |
Edward Hope-Morley |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-done-focal verification-needed verification-ussuri-needed verification-victoria-done |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovn patch verification-done verification-done-focal verification-ussuri-done verification-victoria-done |
|
| 2022-10-03 18:53:57 |
Corey Bryant |
cloud-archive/victoria: status |
Fix Committed |
Fix Released |
|
| 2022-10-05 03:28:07 |
Launchpad Janitor |
neutron (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2022-10-05 12:00:49 |
Corey Bryant |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
