| 2021-05-09 17:42:05 |
Alexandros Soumplis |
bug |
|
|
added bug |
| 2021-05-09 17:43:07 |
Launchpad Janitor |
neutron (Ubuntu): status |
New |
Confirmed |
|
| 2021-06-14 18:29:46 |
Corey Bryant |
bug task added |
|
neutron |
|
| 2021-06-14 23:16:13 |
Jared Baker |
bug |
|
|
added subscriber Jared Baker |
| 2021-06-16 07:18:47 |
Slawek Kaplonski |
tags |
|
l3-d |
|
| 2021-06-16 07:18:54 |
Slawek Kaplonski |
tags |
l3-d |
l3-dvr-backlog |
|
| 2021-06-16 07:20:54 |
Slawek Kaplonski |
neutron: importance |
Undecided |
Critical |
|
| 2021-06-16 08:21:19 |
Christian Rohmann |
bug |
|
|
added subscriber Christian Rohmann |
| 2021-06-17 16:59:52 |
David Negreira |
bug |
|
|
added subscriber David Negreira |
| 2021-06-17 21:16:11 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2021-06-18 14:24:20 |
Corey Bryant |
attachment added |
|
LP1927868-recreate-15.3.3-to-16.3.2.txt https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1927868/+attachment/5505559/+files/LP1927868-recreate-15.3.3-to-16.3.2.txt |
|
| 2021-06-18 14:25:22 |
Corey Bryant |
neutron (Ubuntu): status |
Confirmed |
Triaged |
|
| 2021-06-18 15:47:40 |
Jared Baker |
bug watch added |
|
https://bugzilla.redhat.com/show_bug.cgi?id=1929829 |
|
| 2021-06-22 14:27:19 |
Christian Rohmann |
bug |
|
|
added subscriber Hannes von Haugwitz |
| 2021-06-23 02:13:12 |
Corey Bryant |
attachment removed |
LP1927868-recreate-15.3.3-to-16.3.2.txt https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1927868/+attachment/5505559/+files/LP1927868-recreate-15.3.3-to-16.3.2.txt |
|
|
| 2021-06-23 13:03:15 |
Corey Bryant |
neutron (Ubuntu): status |
Triaged |
New |
|
| 2021-06-28 13:39:39 |
Corey Bryant |
neutron (Ubuntu): status |
New |
Incomplete |
|
| 2021-06-29 11:39:50 |
zibort |
bug |
|
|
added subscriber Lev Klimin |
| 2021-07-01 11:35:58 |
Kabanov Oleg |
bug |
|
|
added subscriber Kabanov Oleg |
| 2021-07-22 09:15:41 |
Fontenay Tony |
bug |
|
|
added subscriber Fontenay Tony |
| 2021-07-23 12:49:22 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2021-07-27 21:08:05 |
James Troup |
neutron (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2021-07-27 21:26:32 |
James Troup |
bug |
|
|
added subscriber Canonical IS CREs |
| 2021-07-28 20:45:27 |
Corey Bryant |
neutron (Ubuntu): status |
Confirmed |
Triaged |
|
| 2021-07-28 20:45:32 |
Corey Bryant |
neutron (Ubuntu): importance |
Undecided |
Critical |
|
| 2021-07-28 21:43:32 |
Corey Bryant |
attachment added |
|
bug-1927868-debugging.txt https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1927868/+attachment/5514384/+files/bug-1927868-debugging.txt |
|
| 2021-07-28 21:50:51 |
Corey Bryant |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-07-28 21:50:51 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Hirsute) |
|
| 2021-07-28 21:50:51 |
Corey Bryant |
nominated for series |
|
Ubuntu Impish |
|
| 2021-07-28 21:50:51 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Impish) |
|
| 2021-07-28 21:50:51 |
Corey Bryant |
nominated for series |
|
Ubuntu Focal |
|
| 2021-07-28 21:50:51 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Focal) |
|
| 2021-07-28 21:51:02 |
Corey Bryant |
neutron (Ubuntu Focal): status |
New |
Triaged |
|
| 2021-07-28 21:51:07 |
Corey Bryant |
neutron (Ubuntu Hirsute): importance |
Undecided |
Critical |
|
| 2021-07-28 21:51:10 |
Corey Bryant |
neutron (Ubuntu Focal): importance |
Undecided |
Critical |
|
| 2021-07-28 21:51:15 |
Corey Bryant |
neutron (Ubuntu Hirsute): status |
New |
Triaged |
|
| 2021-07-28 21:51:31 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
nominated for series |
|
cloud-archive/wallaby |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
bug task added |
|
cloud-archive/wallaby |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
nominated for series |
|
cloud-archive/victoria |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
bug task added |
|
cloud-archive/victoria |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
nominated for series |
|
cloud-archive/train |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
bug task added |
|
cloud-archive/train |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
nominated for series |
|
cloud-archive/ussuri |
|
| 2021-07-28 21:51:45 |
Corey Bryant |
bug task added |
|
cloud-archive/ussuri |
|
| 2021-07-28 21:52:27 |
Corey Bryant |
nominated for series |
|
cloud-archive/xena |
|
| 2021-07-28 21:52:27 |
Corey Bryant |
bug task added |
|
cloud-archive/xena |
|
| 2021-07-28 21:52:43 |
Corey Bryant |
cloud-archive/xena: importance |
Undecided |
Critical |
|
| 2021-07-28 21:52:43 |
Corey Bryant |
cloud-archive/xena: status |
New |
Triaged |
|
| 2021-07-28 21:52:57 |
Corey Bryant |
cloud-archive/wallaby: importance |
Undecided |
Critical |
|
| 2021-07-28 21:52:57 |
Corey Bryant |
cloud-archive/wallaby: status |
New |
Triaged |
|
| 2021-07-28 21:53:11 |
Corey Bryant |
cloud-archive/victoria: importance |
Undecided |
Critical |
|
| 2021-07-28 21:53:11 |
Corey Bryant |
cloud-archive/victoria: status |
New |
Triaged |
|
| 2021-07-28 21:53:27 |
Corey Bryant |
cloud-archive/ussuri: importance |
Undecided |
Critical |
|
| 2021-07-28 21:53:27 |
Corey Bryant |
cloud-archive/ussuri: status |
New |
Triaged |
|
| 2021-07-28 21:53:41 |
Corey Bryant |
cloud-archive/train: importance |
Undecided |
Critical |
|
| 2021-07-28 21:53:41 |
Corey Bryant |
cloud-archive/train: status |
New |
Triaged |
|
| 2021-07-28 22:02:47 |
Corey Bryant |
description |
We run a juju managed Openstack Ussuri on Bionic. After updating neutron packages from 16.3.0 to 16.3.1 all virtual routers stopped working. It seems that most (not all) namespaces are created but have only the lo interface and sometime the ha-XYZ interface in DOWN state. The underlying tap interfaces are also in down.
neutron-l3-agent has many logs similar to the following:
2021-05-08 15:01:45.286 39411 ERROR neutron.agent.l3.ha_router [-] Gateway interface for router 02945b59-639b-41be-8237-3b7933b4e32d was not set up; router will not work properly
and journal logs report at around the same time
May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.765 18596 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 62.62.62.62 on qg-5a6efe8c-6b in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d: Exit code: 2; Stdin: ; Stdout: Interface "qg-5a6efe8c-6b" is down
May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.767 18596 INFO neutron.agent.linux.ip_lib [-] Interface qg-5a6efe8c-6b or address 62.62.62.62 in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d was deleted concurrently
The neutron packages installed are:
ii neutron-common 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-metering-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metering agent
ii neutron-openvswitch-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python3-neutron 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Python library
ii python3-neutron-lib 2.3.0-0ubuntu1~cloud0 all Neutron shared routines and utilities - Python 3.x
ii python3-neutronclient 1:7.1.1-0ubuntu1~cloud0 all client API library for Neutron - Python 3.x
Downgrading to 16.3.0 resolves the issues. |
We run a juju managed Openstack Ussuri on Bionic. After updating neutron packages from 16.3.0 to 16.3.1 all virtual routers stopped working. It seems that most (not all) namespaces are created but have only the lo interface and sometime the ha-XYZ interface in DOWN state. The underlying tap interfaces are also in down.
neutron-l3-agent has many logs similar to the following:
2021-05-08 15:01:45.286 39411 ERROR neutron.agent.l3.ha_router [-] Gateway interface for router 02945b59-639b-41be-8237-3b7933b4e32d was not set up; router will not work properly
and journal logs report at around the same time
May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.765 18596 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 62.62.62.62 on qg-5a6efe8c-6b in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d: Exit code: 2; Stdin: ; Stdout: Interface "qg-5a6efe8c-6b" is down
May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.767 18596 INFO neutron.agent.linux.ip_lib [-] Interface qg-5a6efe8c-6b or address 62.62.62.62 in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d was deleted concurrently
The neutron packages installed are:
ii neutron-common 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-metering-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metering agent
ii neutron-openvswitch-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python3-neutron 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Python library
ii python3-neutron-lib 2.3.0-0ubuntu1~cloud0 all Neutron shared routines and utilities - Python 3.x
ii python3-neutronclient 1:7.1.1-0ubuntu1~cloud0 all client API library for Neutron - Python 3.x
Downgrading to 16.3.0 resolves the issues.
=================================
Ubuntu SRU details:
[Impact]
See above.
[Test Case]
Deploy openstack with l3ha and create several HA routers, the number required varies per environment. It is probably best to deploy a known bad version of the package, ensure it is failing, upgrade to the version in proposed, and re-test several times to confirm it is fixed.
Restarting neutron-l3-agent should expect all HA Routers are restored.
[Regression Potential]
This change is fixing a regression by reverting a patch that was introduced in a stable point release of neutron. |
|
| 2021-07-28 22:24:47 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-07-28 22:35:31 |
Brian Murray |
neutron (Ubuntu Hirsute): status |
Triaged |
Fix Committed |
|
| 2021-07-28 22:35:35 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2021-07-28 22:35:44 |
Brian Murray |
tags |
l3-dvr-backlog |
l3-dvr-backlog verification-needed verification-needed-hirsute |
|
| 2021-07-28 22:39:06 |
Brian Murray |
neutron (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2021-07-28 22:39:17 |
Brian Murray |
tags |
l3-dvr-backlog verification-needed verification-needed-hirsute |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute |
|
| 2021-07-28 22:40:25 |
Corey Bryant |
cloud-archive/victoria: status |
Triaged |
Fix Committed |
|
| 2021-07-28 22:40:28 |
Corey Bryant |
tags |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-victoria-needed |
|
| 2021-07-28 22:42:27 |
Corey Bryant |
cloud-archive/train: status |
Triaged |
Fix Committed |
|
| 2021-07-28 22:42:30 |
Corey Bryant |
tags |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-victoria-needed |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-victoria-needed |
|
| 2021-07-29 00:35:05 |
Corey Bryant |
cloud-archive: status |
Triaged |
Fix Committed |
|
| 2021-07-29 01:33:24 |
Launchpad Janitor |
neutron (Ubuntu Impish): status |
Triaged |
Fix Released |
|
| 2021-07-29 06:52:31 |
James Page |
cloud-archive/ussuri: status |
Triaged |
Fix Committed |
|
| 2021-07-29 06:52:35 |
James Page |
tags |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-victoria-needed |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-ussuri-needed verification-victoria-needed |
|
| 2021-07-29 10:21:51 |
Corey Bryant |
cloud-archive/wallaby: status |
Triaged |
Fix Committed |
|
| 2021-07-29 10:21:54 |
Corey Bryant |
tags |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-ussuri-needed verification-victoria-needed |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-ussuri-needed verification-victoria-needed verification-wallaby-needed |
|
| 2021-07-29 14:54:14 |
Corey Bryant |
bug task added |
|
oslo.privsep |
|
| 2021-07-30 08:55:38 |
Tobias Ketter |
bug |
|
|
added subscriber Tobias Ketter |
| 2021-08-03 13:29:01 |
Corey Bryant |
cloud-archive: status |
Fix Committed |
Fix Released |
|
| 2021-08-04 15:13:07 |
Edward Hope-Morley |
tags |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-ussuri-needed verification-victoria-needed verification-wallaby-needed |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-ussuri-done verification-victoria-needed verification-wallaby-needed |
|
| 2021-08-04 15:28:48 |
Chris MacNaughton |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
| 2021-08-10 15:17:30 |
Alin-Gabriel Serdean |
tags |
l3-dvr-backlog verification-needed verification-needed-focal verification-needed-hirsute verification-train-needed verification-ussuri-done verification-victoria-needed verification-wallaby-needed |
l3-dvr-backlog verification-hirsute-done verification-needed verification-needed-focal verification-train-needed verification-ussuri-done verification-victoria-needed verification-wallaby-needed |
|
| 2021-08-10 15:32:57 |
Alin-Gabriel Serdean |
tags |
l3-dvr-backlog verification-hirsute-done verification-needed verification-needed-focal verification-train-needed verification-ussuri-done verification-victoria-needed verification-wallaby-needed |
l3-dvr-backlog verification-focal-done verification-hirsute-done verification-needed verification-train-needed verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-08-10 20:11:04 |
Alin-Gabriel Serdean |
tags |
l3-dvr-backlog verification-focal-done verification-hirsute-done verification-needed verification-train-needed verification-ussuri-done verification-victoria-done verification-wallaby-done |
l3-dvr-backlog verification-bionic-done verification-focal-done verification-hirsute-done verification-needed verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-08-10 20:11:44 |
Alin-Gabriel Serdean |
tags |
l3-dvr-backlog verification-bionic-done verification-focal-done verification-hirsute-done verification-needed verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
l3-dvr-backlog verification-bionic-done verification-done verification-focal-done verification-hirsute-done verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-08-12 23:26:45 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
| 2021-08-16 13:40:47 |
Chris MacNaughton |
cloud-archive/wallaby: status |
Fix Committed |
Fix Released |
|
| 2021-08-16 15:15:45 |
Chris MacNaughton |
tags |
l3-dvr-backlog verification-bionic-done verification-done verification-focal-done verification-hirsute-done verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-08-16 16:08:18 |
Launchpad Janitor |
neutron (Ubuntu Hirsute): status |
Fix Committed |
Fix Released |
|
| 2021-08-16 16:08:36 |
Launchpad Janitor |
neutron (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-08-16 17:02:20 |
Chris MacNaughton |
cloud-archive/victoria: status |
Fix Committed |
Fix Released |
|
| 2021-08-16 17:04:21 |
Chris MacNaughton |
cloud-archive/train: status |
Fix Committed |
Fix Released |
|
| 2021-08-20 11:28:26 |
OpenStack Infra |
neutron: status |
New |
In Progress |
|
| 2021-08-20 11:28:59 |
Edward Hope-Morley |
neutron: assignee |
|
Edward Hope-Morley (hopem) |
|
| 2021-09-15 09:49:37 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2021-09-16 21:35:18 |
OpenStack Infra |
tags |
l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-09-23 23:14:08 |
OpenStack Infra |
tags |
in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
in-stable-train in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-09-23 23:14:17 |
OpenStack Infra |
tags |
in-stable-train in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
in-stable-train in-stable-ussuri in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-09-24 12:37:25 |
OpenStack Infra |
tags |
in-stable-train in-stable-ussuri in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-10-12 12:01:31 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~hopem/ubuntu/+source/neutron/+git/neutron/+merge/410053 |
|
| 2021-11-17 15:48:28 |
Bernard Cafarelli |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby l3-dvr-backlog neutron-proactive-backport-potential verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
| 2021-12-10 13:50:57 |
Slawek Kaplonski |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby l3-dvr-backlog neutron-proactive-backport-potential verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby l3-dvr-backlog verification-bionic-done verification-done verification-done-focal verification-done-hirsute verification-train-done verification-ussuri-done verification-victoria-done verification-wallaby-done |
|
