Activity log for bug #1871850

Date Who What changed Old value New value Message
2020-04-09 13:22:14 LIU Yulong bug added bug
2020-04-09 19:46:44 Brian Haley bug added subscriber Brian Haley
2020-04-09 21:19:24 Slawek Kaplonski tags l3-dvr-backlog
2020-04-11 00:49:01 OpenStack Infra neutron: status Confirmed In Progress
2020-04-11 00:49:01 OpenStack Infra neutron: assignee LIU Yulong (dragon889)
2020-04-24 21:45:20 OpenStack Infra neutron: assignee LIU Yulong (dragon889) Brian Haley (brian-haley)
2020-04-27 15:54:05 OpenStack Infra neutron: status In Progress Fix Released
2020-06-04 09:18:11 Bernard Cafarelli tags l3-dvr-backlog l3-dvr-backlog neutron-proactive-backport-potential
2020-08-27 19:38:06 OpenStack Infra tags l3-dvr-backlog neutron-proactive-backport-potential in-stable-rocky l3-dvr-backlog neutron-proactive-backport-potential
2020-08-28 03:34:36 OpenStack Infra tags in-stable-rocky l3-dvr-backlog neutron-proactive-backport-potential in-stable-rocky in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential
2020-08-28 11:56:16 OpenStack Infra tags in-stable-rocky in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential
2020-08-28 18:31:01 OpenStack Infra tags in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential
2020-08-29 14:37:58 OpenStack Infra tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog neutron-proactive-backport-potential
2020-10-09 09:22:00 Slawek Kaplonski tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog neutron-proactive-backport-potential in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog
2020-11-25 02:40:12 Nobuto Murata bug added subscriber Nobuto Murata
2020-12-08 05:19:09 Trent Lloyd bug task added cloud-archive
2020-12-11 04:14:11 Dominique Poulain bug added subscriber Dominique Poulain
2020-12-11 06:35:41 Erwin Dwight Paler summary [L3] existing router resources are partial deleted unexceptedly when MQ is gone [L3] existing router resources are partial deleted unexpectedly when MQ is gone
2021-01-04 10:29:02 Chris MacNaughton nominated for series cloud-archive/queens
2021-01-04 10:29:02 Chris MacNaughton bug task added cloud-archive/queens
2021-01-04 10:29:02 Chris MacNaughton nominated for series cloud-archive/rocky
2021-01-04 10:29:02 Chris MacNaughton bug task added cloud-archive/rocky
2021-01-04 10:29:02 Chris MacNaughton nominated for series cloud-archive/ussuri
2021-01-04 10:29:02 Chris MacNaughton bug task added cloud-archive/ussuri
2021-01-04 10:29:02 Chris MacNaughton nominated for series cloud-archive/train
2021-01-04 10:29:02 Chris MacNaughton bug task added cloud-archive/train
2021-01-04 10:29:02 Chris MacNaughton nominated for series cloud-archive/stein
2021-01-04 10:29:02 Chris MacNaughton bug task added cloud-archive/stein
2021-01-04 10:30:52 Chris MacNaughton cloud-archive/stein: status New Fix Released
2021-01-04 10:31:02 Chris MacNaughton cloud-archive/train: status New Fix Released
2021-01-04 10:31:14 Chris MacNaughton cloud-archive/ussuri: status New Fix Released
2021-01-04 10:31:26 Chris MacNaughton cloud-archive: status New Invalid
2021-02-18 17:17:54 Dan Streetman bug task added neutron (Ubuntu)
2021-02-18 17:18:04 Dan Streetman nominated for series Ubuntu Bionic
2021-02-18 17:18:04 Dan Streetman bug task added neutron (Ubuntu Bionic)
2021-02-18 17:18:14 Dan Streetman neutron (Ubuntu): status New Fix Released
2021-02-18 17:18:33 Dan Streetman neutron (Ubuntu Bionic): assignee Trent Lloyd (lathiat)
2021-02-18 17:18:39 Dan Streetman neutron (Ubuntu Bionic): importance Undecided Critical
2021-02-18 17:18:42 Dan Streetman neutron (Ubuntu Bionic): status New In Progress
2021-02-18 17:20:02 Dan Streetman description ENV: meet this issue on our stable/queens deployment, but master branch has the same code logic When the L3 agent get a router update notification, it will try to retrieve the router info from DB server [1]. But at this time, if the message queue is down/unreachable. It will get exceptions related message queue. A resync action will be run then [2]. Sometimes, from my personal experience, rabbitMQ cluster is not so much easy to recover. Long time MQ recover time will cause the router info sync RPC never get successful until it meets the max retry time [3]. So the bad thing happens, L3 agent is trying to remove the router now [4]. It basically shutdown all the existing L3 traffic of this router. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L705 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L710 [3] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L666 [4] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L671 (For SRU template, please see bug 1869808, as the SRU info there applies to this bug also) ENV: meet this issue on our stable/queens deployment, but master branch has the same code logic When the L3 agent get a router update notification, it will try to retrieve the router info from DB server [1]. But at this time, if the message queue is down/unreachable. It will get exceptions related message queue. A resync action will be run then [2]. Sometimes, from my personal experience, rabbitMQ cluster is not so much easy to recover. Long time MQ recover time will cause the router info sync RPC never get successful until it meets the max retry time [3]. So the bad thing happens, L3 agent is trying to remove the router now [4]. It basically shutdown all the existing L3 traffic of this router. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L705 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L710 [3] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L666 [4] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L671
2021-02-18 18:33:36 Corey Bryant cloud-archive/rocky: importance Undecided Critical
2021-02-18 18:33:36 Corey Bryant cloud-archive/rocky: status New Triaged
2021-02-18 18:33:48 Corey Bryant cloud-archive/queens: status New Incomplete
2021-02-18 18:33:52 Corey Bryant cloud-archive/queens: status Incomplete Triaged
2021-02-18 18:33:56 Corey Bryant cloud-archive/queens: importance Undecided Critical
2021-02-19 16:04:01 Corey Bryant cloud-archive/rocky: status Triaged Fix Committed
2021-02-19 16:04:05 Corey Bryant tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog verification-rocky-needed
2021-03-18 15:35:53 Łukasz Zemczak neutron (Ubuntu Bionic): status In Progress Fix Committed
2021-03-18 15:35:56 Łukasz Zemczak bug added subscriber Ubuntu Stable Release Updates Team
2021-03-18 15:35:59 Łukasz Zemczak bug added subscriber SRU Verification
2021-03-18 15:36:03 Łukasz Zemczak tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog verification-rocky-needed in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog verification-needed verification-needed-bionic verification-rocky-needed
2021-03-22 21:01:45 Corey Bryant cloud-archive/queens: status Triaged Fix Committed
2021-03-22 21:01:50 Corey Bryant tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog verification-needed verification-needed-bionic verification-rocky-needed in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed
2021-04-07 10:19:09 Edward Hope-Morley tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog verification-done verification-done-bionic verification-queens-done verification-rocky-done
2021-04-08 09:00:43 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2021-04-08 09:05:26 Launchpad Janitor neutron (Ubuntu Bionic): status Fix Committed Fix Released
2021-04-12 13:52:58 Corey Bryant cloud-archive/queens: status Fix Committed Fix Released
2021-04-12 18:12:51 Corey Bryant cloud-archive/rocky: status Fix Committed Fix Released
2021-10-12 12:01:14 Launchpad Janitor merge proposal linked https://code.launchpad.net/~hopem/ubuntu/+source/neutron/+git/neutron/+merge/410051