We run a juju managed Openstack Ussuri on Bionic. After updating neutron packages from 16.3.0 to 16.3.1 all virtual routers stopped working. It seems that most (not all) namespaces are created but have only the lo interface and sometime the ha-XYZ interface in DOWN state. The underlying tap interfaces are also in down.
neutron-l3-agent has many logs similar to the following:
2021-05-08 15:01:45.286 39411 ERROR neutron.agent.l3.ha_router [-] Gateway interface for router 02945b59-639b-41be-8237-3b7933b4e32d was not set up; router will not work properly
and journal logs report at around the same time
May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.765 18596 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 62.62.62.62 on qg-5a6efe8c-6b in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d: Exit code: 2; Stdin: ; Stdout: Interface "qg-5a6efe8c-6b" is down
May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.767 18596 INFO neutron.agent.linux.ip_lib [-] Interface qg-5a6efe8c-6b or address 62.62.62.62 in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d was deleted concurrently
The neutron packages installed are:
ii neutron-common 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-metering-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metering agent
ii neutron-openvswitch-agent 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python3-neutron 2:16.3.1-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Python library
ii python3-neutron-lib 2.3.0-0ubuntu1~cloud0 all Neutron shared routines and utilities - Python 3.x
ii python3-neutronclient 1:7.1.1-0ubuntu1~cloud0 all client API library for Neutron - Python 3.x
We run a juju managed Openstack Ussuri on Bionic. After updating neutron packages from 16.3.0 to 16.3.1 all virtual routers stopped working. It seems that most (not all) namespaces are created but have only the lo interface and sometime the ha-XYZ interface in DOWN state. The underlying tap interfaces are also in down.
neutron-l3-agent has many logs similar to the following: agent.l3. ha_router [-] Gateway interface for router 02945b59- 639b-41be- 8237-3b7933b4e3 2d was not set up; router will not work properly
2021-05-08 15:01:45.286 39411 ERROR neutron.
and journal logs report at around the same time srv-louros. grnet.gr neutron- keepalived- state-change[ 18596]: 2021-05-08 15:01:40.765 18596 INFO neutron. agent.linux. ip_lib [-] Failed sending gratuitous ARP to 62.62.62.62 on qg-5a6efe8c-6b in namespace qrouter- 02945b59- 639b-41be- 8237-3b7933b4e3 2d: Exit code: 2; Stdin: ; Stdout: Interface "qg-5a6efe8c-6b" is down srv-louros. grnet.gr neutron- keepalived- state-change[ 18596]: 2021-05-08 15:01:40.767 18596 INFO neutron. agent.linux. ip_lib [-] Interface qg-5a6efe8c-6b or address 62.62.62.62 in namespace qrouter- 02945b59- 639b-41be- 8237-3b7933b4e3 2d was deleted concurrently
May 08 15:01:40 lar1615.
May 08 15:01:40 lar1615.
The neutron packages installed are:
ii neutron-common 2:16.3. 1-0ubuntu1~ cloud0 all Neutron is a virtual network service for Openstack - common 1-0ubuntu1~ cloud0 all Neutron is a virtual network service for Openstack - DHCP agent 1-0ubuntu1~ cloud0 all Neutron is a virtual network service for Openstack - l3 agent metadata- agent 2:16.3. 1-0ubuntu1~ cloud0 all Neutron is a virtual network service for Openstack - metadata agent metering- agent 2:16.3. 1-0ubuntu1~ cloud0 all Neutron is a virtual network service for Openstack - metering agent openvswitch- agent 2:16.3. 1-0ubuntu1~ cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent 1-0ubuntu1~ cloud0 all Neutron is a virtual network service for Openstack - Python library cloud0 all Neutron shared routines and utilities - Python 3.x neutronclient 1:7.1.1- 0ubuntu1~ cloud0 all client API library for Neutron - Python 3.x
ii neutron-dhcp-agent 2:16.3.
ii neutron-l3-agent 2:16.3.
ii neutron-
ii neutron-
ii neutron-
ii python3-neutron 2:16.3.
ii python3-neutron-lib 2.3.0-0ubuntu1~
ii python3-
Downgrading to 16.3.0 resolves the issues.