Activity log for bug #1891673

Date Who What changed Old value New value Message
2020-08-14 15:30:20 Edward Hope-Morley bug added bug
2020-08-14 15:34:52 Edward Hope-Morley tags sts
2020-08-14 15:57:41 James Troup bug added subscriber Canonical IS CREs
2020-08-14 16:44:10 Edward Hope-Morley neutron: assignee Edward Hope-Morley (hopem)
2020-08-14 16:48:11 OpenStack Infra neutron: status New In Progress
2020-08-17 08:03:46 Bence Romsics neutron: status In Progress Confirmed
2020-08-17 08:03:52 Bence Romsics neutron: importance Undecided High
2020-08-17 08:04:17 Bence Romsics tags sts l3-dvr-backlog sts
2020-08-17 09:09:06 Edward Hope-Morley neutron: status Confirmed In Progress
2020-09-03 23:22:08 Dominique Poulain bug added subscriber Dominique Poulain
2020-09-07 14:15:30 Edward Hope-Morley bug task added cloud-archive
2020-09-07 14:15:48 Edward Hope-Morley nominated for series cloud-archive/train
2020-09-07 14:15:48 Edward Hope-Morley bug task added cloud-archive/train
2020-09-07 14:15:48 Edward Hope-Morley nominated for series cloud-archive/victoria
2020-09-07 14:15:48 Edward Hope-Morley bug task added cloud-archive/victoria
2020-09-07 14:15:48 Edward Hope-Morley nominated for series cloud-archive/ussuri
2020-09-07 14:15:48 Edward Hope-Morley bug task added cloud-archive/ussuri
2020-09-07 14:15:48 Edward Hope-Morley nominated for series cloud-archive/rocky
2020-09-07 14:15:48 Edward Hope-Morley bug task added cloud-archive/rocky
2020-09-07 14:15:48 Edward Hope-Morley nominated for series cloud-archive/queens
2020-09-07 14:15:48 Edward Hope-Morley bug task added cloud-archive/queens
2020-09-07 14:15:48 Edward Hope-Morley nominated for series cloud-archive/stein
2020-09-07 14:15:48 Edward Hope-Morley bug task added cloud-archive/stein
2020-09-08 10:18:00 OpenStack Infra neutron: status In Progress Fix Released
2020-09-09 08:43:42 Edward Hope-Morley bug task added neutron (Ubuntu)
2020-09-09 08:44:10 Edward Hope-Morley nominated for series Ubuntu Focal
2020-09-09 08:44:10 Edward Hope-Morley bug task added neutron (Ubuntu Focal)
2020-09-09 08:44:10 Edward Hope-Morley nominated for series Ubuntu Groovy
2020-09-09 08:44:10 Edward Hope-Morley bug task added neutron (Ubuntu Groovy)
2020-09-09 08:44:10 Edward Hope-Morley nominated for series Ubuntu Bionic
2020-09-09 08:44:10 Edward Hope-Morley bug task added neutron (Ubuntu Bionic)
2020-09-09 08:45:15 Edward Hope-Morley attachment added lp1891673-focal-ussuri.debdiff https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1891673/+attachment/5408960/+files/lp1891673-focal-ussuri.debdiff
2020-09-09 08:47:13 Edward Hope-Morley attachment added lp1891673-bionic-train-uca.debdiff https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1891673/+attachment/5408962/+files/lp1891673-bionic-train-uca.debdiff
2020-09-09 08:55:18 Edward Hope-Morley description With Bionic Stein using dvr_snat if I add a floating ip to a vm then remove the floating ip, the corresponding ip rules in the associated qrouter ns local to the instance are not deleted which results in no longer being able to reach the external network because packets are still sent to the fip namespace (via rfp-/fpr-) e.g. in my compute host running a vm whose address is 192.168.21.28 for which i have removed the fip I still see: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip rule list 0: from all lookup local 32765: from 192.168.21.28 lookup 16 32766: from all lookup main 32767: from all lookup default 3232240897: from 192.168.21.1/24 lookup 3232240897 3232241231: from 192.168.22.79/24 lookup 3232241231 And table 16 leads to: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip route show table 16 default via 169.254.109.249 dev rfp-5e45608f-3 Which results in the instance no longer being able to reach the external network (packets are never sent to the snat- ns in my case). The workaround is to delete that ip rule but neutron should be taking care of this. Looks like the culprit is in neutron/agent/l3/dvr_local_router.py:floating_ip_removed_dist Note that the NAT rules were successfully removed from iptables so looks like it is just this bit that is left behind. [Impact] neutron-l3-agent restart causes partial loss of fip information such that fip removal from vm results in ip rules left behind which breaks external network access for that vm. [Test Case] * deploy openstack with dvr enabled * create distributed router, network etc * create a vm and attach a floating ip * go to compute host on which vm is running and restart neutron-l3-agent * tail -f /var/log/neutron/neutron-l3-agent.log until it settles * remove fip from vm * run https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b on that compute node * should return with "nothing to do" [Regression Potential] none expected [Other Info] patched neutron l3 agent will reload info for *used* floating ips when restarted BUT if there are ip rules left behind from fips removed prior to using a pathed neutron then manual cleanup is still required and for that you can use https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b. -------------------------------------------------------------------------- With Bionic Stein using dvr_snat if I add a floating ip to a vm then remove the floating ip, the corresponding ip rules in the associated qrouter ns local to the instance are not deleted which results in no longer being able to reach the external network because packets are still sent to the fip namespace (via rfp-/fpr-) e.g. in my compute host running a vm whose address is 192.168.21.28 for which i have removed the fip I still see: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip rule list 0: from all lookup local 32765: from 192.168.21.28 lookup 16 32766: from all lookup main 32767: from all lookup default 3232240897: from 192.168.21.1/24 lookup 3232240897 3232241231: from 192.168.22.79/24 lookup 3232241231 And table 16 leads to: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip route show table 16 default via 169.254.109.249 dev rfp-5e45608f-3 Which results in the instance no longer being able to reach the external network (packets are never sent to the snat- ns in my case). The workaround is to delete that ip rule but neutron should be taking care of this. Looks like the culprit is in neutron/agent/l3/dvr_local_router.py:floating_ip_removed_dist Note that the NAT rules were successfully removed from iptables so looks like it is just this bit that is left behind.
2020-09-09 08:55:52 Edward Hope-Morley tags l3-dvr-backlog sts l3-dvr-backlog sts sts-sru-needed
2020-09-09 12:28:03 Ubuntu Foundations Team Bug Bot tags l3-dvr-backlog sts sts-sru-needed l3-dvr-backlog patch sts sts-sru-needed
2020-09-09 12:28:09 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Sponsors Team
2020-09-09 13:08:19 Edward Hope-Morley attachment added lp1891673-bionic-stein-uca.debdiff https://bugs.launchpad.net/neutron/+bug/1891673/+attachment/5409037/+files/lp1891673-bionic-stein-uca.debdiff
2020-09-09 13:41:32 Edward Hope-Morley attachment added lp1891673-bionic-queens.debdiff https://bugs.launchpad.net/neutron/+bug/1891673/+attachment/5409048/+files/lp1891673-bionic-queens.debdiff
2020-09-09 13:57:03 Edward Hope-Morley attachment added lp1891673-bionic-rocky.debdiff https://bugs.launchpad.net/neutron/+bug/1891673/+attachment/5409049/+files/lp1891673-bionic-rocky.debdiff
2020-09-10 19:49:24 Corey Bryant cloud-archive/victoria: importance Undecided High
2020-09-10 19:49:24 Corey Bryant cloud-archive/victoria: status New Triaged
2020-09-10 19:49:59 Corey Bryant cloud-archive/queens: importance Undecided High
2020-09-10 19:49:59 Corey Bryant cloud-archive/queens: status In Progress Triaged
2020-09-10 19:50:05 Corey Bryant cloud-archive/rocky: importance Undecided High
2020-09-10 19:50:05 Corey Bryant cloud-archive/rocky: status In Progress Triaged
2020-09-10 19:50:12 Corey Bryant cloud-archive/stein: importance Undecided High
2020-09-10 19:50:12 Corey Bryant cloud-archive/stein: status In Progress Triaged
2020-09-10 19:50:20 Corey Bryant cloud-archive/train: importance Undecided High
2020-09-10 19:50:20 Corey Bryant cloud-archive/train: status In Progress Triaged
2020-09-10 19:50:28 Corey Bryant cloud-archive/ussuri: importance Undecided High
2020-09-10 19:50:28 Corey Bryant cloud-archive/ussuri: status In Progress Triaged
2020-09-10 19:51:02 Corey Bryant neutron (Ubuntu Bionic): importance Undecided High
2020-09-10 19:51:02 Corey Bryant neutron (Ubuntu Bionic): status New Triaged
2020-09-10 19:51:11 Corey Bryant neutron (Ubuntu Focal): importance Undecided High
2020-09-10 19:51:11 Corey Bryant neutron (Ubuntu Focal): status New Triaged
2020-09-10 19:51:19 Corey Bryant neutron (Ubuntu Groovy): importance Undecided High
2020-09-10 19:51:19 Corey Bryant neutron (Ubuntu Groovy): status New Triaged
2020-09-11 15:32:35 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2020-09-16 14:08:23 Robie Basak neutron (Ubuntu Bionic): status Triaged Incomplete
2020-09-16 14:08:41 Robie Basak neutron (Ubuntu Focal): status Triaged Incomplete
2020-09-16 14:08:51 Robie Basak neutron (Ubuntu Groovy): status Triaged Incomplete
2020-09-16 15:42:33 Edward Hope-Morley description [Impact] neutron-l3-agent restart causes partial loss of fip information such that fip removal from vm results in ip rules left behind which breaks external network access for that vm. [Test Case] * deploy openstack with dvr enabled * create distributed router, network etc * create a vm and attach a floating ip * go to compute host on which vm is running and restart neutron-l3-agent * tail -f /var/log/neutron/neutron-l3-agent.log until it settles * remove fip from vm * run https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b on that compute node * should return with "nothing to do" [Regression Potential] none expected [Other Info] patched neutron l3 agent will reload info for *used* floating ips when restarted BUT if there are ip rules left behind from fips removed prior to using a pathed neutron then manual cleanup is still required and for that you can use https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b. -------------------------------------------------------------------------- With Bionic Stein using dvr_snat if I add a floating ip to a vm then remove the floating ip, the corresponding ip rules in the associated qrouter ns local to the instance are not deleted which results in no longer being able to reach the external network because packets are still sent to the fip namespace (via rfp-/fpr-) e.g. in my compute host running a vm whose address is 192.168.21.28 for which i have removed the fip I still see: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip rule list 0: from all lookup local 32765: from 192.168.21.28 lookup 16 32766: from all lookup main 32767: from all lookup default 3232240897: from 192.168.21.1/24 lookup 3232240897 3232241231: from 192.168.22.79/24 lookup 3232241231 And table 16 leads to: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip route show table 16 default via 169.254.109.249 dev rfp-5e45608f-3 Which results in the instance no longer being able to reach the external network (packets are never sent to the snat- ns in my case). The workaround is to delete that ip rule but neutron should be taking care of this. Looks like the culprit is in neutron/agent/l3/dvr_local_router.py:floating_ip_removed_dist Note that the NAT rules were successfully removed from iptables so looks like it is just this bit that is left behind. [Impact] neutron-l3-agent restart causes partial loss of fip information such that fip removal from vm results in ip rules left behind which breaks external network access for that vm. [Test Case] * deploy openstack with dvr enabled * create distributed router, network etc * create a vm and attach a floating ip * go to compute host on which vm is running and restart neutron-l3-agent * tail -f /var/log/neutron/neutron-l3-agent.log until it settles * remove fip from vm * run https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b on that compute node * should return with "nothing to do" [Regression Potential] the patch is reloading, on agent startup, information associated with floating ips, specifically the information needed to delete ip rules and rule priorities associated with a floating ip. Since that is essentially read-only I don't envisage a regression potential. When the l3-agent comes to use that information to delete the floating ip an error could occur if the information it is trying to delete no longer exists but that would not be a problem introduced by this patch so again, I don't envisage any potential for regressions from this patch since it doesn't change behavior in any way other than allowing the l3-agent to behave the same as if it hadn't been restarted. [Other Info] patched neutron l3 agent will reload info for *used* floating ips when restarted BUT if there are ip rules left behind from fips removed prior to using a pathed neutron then manual cleanup is still required and for that you can use https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b. -------------------------------------------------------------------------- With Bionic Stein using dvr_snat if I add a floating ip to a vm then remove the floating ip, the corresponding ip rules in the associated qrouter ns local to the instance are not deleted which results in no longer being able to reach the external network because packets are still sent to the fip namespace (via rfp-/fpr-) e.g. in my compute host running a vm whose address is 192.168.21.28 for which i have removed the fip I still see: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip rule list 0: from all lookup local 32765: from 192.168.21.28 lookup 16 32766: from all lookup main 32767: from all lookup default 3232240897: from 192.168.21.1/24 lookup 3232240897 3232241231: from 192.168.22.79/24 lookup 3232241231 And table 16 leads to: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip route show table 16 default via 169.254.109.249 dev rfp-5e45608f-3 Which results in the instance no longer being able to reach the external network (packets are never sent to the snat- ns in my case). The workaround is to delete that ip rule but neutron should be taking care of this. Looks like the culprit is in neutron/agent/l3/dvr_local_router.py:floating_ip_removed_dist Note that the NAT rules were successfully removed from iptables so looks like it is just this bit that is left behind.
2020-09-21 08:07:26 Edward Hope-Morley neutron (Ubuntu Bionic): status Incomplete Triaged
2020-09-21 08:07:40 Edward Hope-Morley neutron (Ubuntu Focal): status Incomplete Triaged
2020-09-21 08:07:50 Edward Hope-Morley neutron (Ubuntu Groovy): status Incomplete Triaged
2020-09-21 12:26:11 OpenStack Infra cloud-archive/ussuri: status Triaged Fix Committed
2020-09-21 12:28:23 Edward Hope-Morley cloud-archive/ussuri: status Fix Committed Triaged
2020-09-22 10:01:45 Edward Hope-Morley cloud-archive/victoria: status Triaged Fix Committed
2020-09-22 10:01:59 Edward Hope-Morley neutron (Ubuntu Groovy): status Triaged Fix Committed
2020-09-22 18:41:36 Brian Murray neutron (Ubuntu Focal): status Triaged Fix Committed
2020-09-22 18:41:43 Brian Murray bug added subscriber SRU Verification
2020-09-22 18:41:48 Brian Murray tags l3-dvr-backlog patch sts sts-sru-needed l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-focal
2020-09-22 18:43:46 Brian Murray neutron (Ubuntu Bionic): status Triaged Fix Committed
2020-09-22 18:43:54 Brian Murray tags l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-focal l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal
2020-09-22 18:43:59 Brian Murray removed subscriber Ubuntu Sponsors Team
2020-09-22 23:59:10 Corey Bryant cloud-archive/ussuri: status Triaged Fix Committed
2020-09-22 23:59:18 Corey Bryant tags l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-ussuri-needed
2020-09-22 23:59:21 Corey Bryant cloud-archive/stein: status Triaged Fix Committed
2020-09-22 23:59:26 Corey Bryant cloud-archive/train: status Triaged Fix Committed
2020-09-22 23:59:46 Corey Bryant cloud-archive/rocky: status Triaged Fix Committed
2020-09-22 23:59:48 Corey Bryant tags l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-ussuri-needed l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-rocky-needed verification-ussuri-needed
2020-09-22 23:59:56 Corey Bryant cloud-archive/queens: status Triaged Fix Committed
2020-09-22 23:59:59 Corey Bryant tags l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-rocky-needed verification-ussuri-needed l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-queens-needed verification-rocky-needed verification-ussuri-needed
2020-09-23 12:42:28 Edward Hope-Morley tags l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-queens-needed verification-rocky-needed verification-ussuri-needed l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-queens-needed verification-rocky-needed verification-ussuri-done
2020-09-23 16:07:52 Edward Hope-Morley tags l3-dvr-backlog patch sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-focal verification-queens-needed verification-rocky-needed verification-ussuri-done l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-ussuri-done
2020-09-23 18:11:50 Edward Hope-Morley tags l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-ussuri-done l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-train-done verification-ussuri-done
2020-09-24 15:03:10 Edward Hope-Morley tags l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-train-done verification-ussuri-done l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-stein-done verification-train-done verification-ussuri-done
2020-09-24 18:15:44 Edward Hope-Morley tags l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-stein-done verification-train-done verification-ussuri-done l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-09-24 19:24:25 Edward Hope-Morley tags l3-dvr-backlog patch sts sts-sru-needed verification-done-focal verification-needed verification-needed-bionic verification-queens-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done l3-dvr-backlog patch sts sts-sru-needed verification-done-bionic verification-done-focal verification-needed verification-queens-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-09-24 20:10:57 Edward Hope-Morley tags l3-dvr-backlog patch sts sts-sru-needed verification-done-bionic verification-done-focal verification-needed verification-queens-needed verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done l3-dvr-backlog patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-09-29 20:58:50 Launchpad Janitor neutron (Ubuntu Focal): status Fix Committed Fix Released
2020-09-29 20:58:56 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2020-09-29 21:00:13 Launchpad Janitor neutron (Ubuntu Bionic): status Fix Committed Fix Released
2020-09-30 13:21:39 Corey Bryant cloud-archive/victoria: status Fix Committed Fix Released
2020-09-30 13:21:51 Corey Bryant neutron (Ubuntu Groovy): status Fix Committed Fix Released
2020-09-30 13:49:14 Corey Bryant cloud-archive/ussuri: status Fix Committed Fix Released
2020-09-30 13:52:20 Corey Bryant cloud-archive/train: status Fix Committed Fix Released
2020-09-30 13:55:27 Corey Bryant cloud-archive/stein: status Fix Committed Fix Released
2020-09-30 13:56:27 Corey Bryant cloud-archive/rocky: status Fix Committed Fix Released
2020-09-30 13:59:12 Corey Bryant cloud-archive/queens: status Fix Committed Fix Released
2020-10-08 11:40:04 Bernard Cafarelli tags l3-dvr-backlog patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-11-04 16:29:46 OpenStack Infra tags l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-11-04 16:29:54 OpenStack Infra tags in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-11-11 17:23:08 OpenStack Infra tags in-stable-queens in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-11-15 12:19:09 OpenStack Infra tags in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2020-12-11 11:05:07 Slawek Kaplonski tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2021-08-06 12:37:54 Bernard Cafarelli tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2021-09-23 19:53:57 OpenStack Infra tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2021-09-24 11:54:23 OpenStack Infra tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri in-stable-wallaby l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done
2021-09-27 02:16:12 OpenStack Infra tags in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri in-stable-wallaby l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done in-stable-queens in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby l3-dvr-backlog neutron-proactive-backport-potential patch sts sts-sru-needed verification-done verification-done-bionic verification-done-focal verification-queens-done verification-rocky-done verification-stein-done verification-train-done verification-ussuri-done