| 2019-03-05 08:03:32 |
Slawek Kaplonski |
bug |
|
|
added bug |
| 2019-03-05 16:42:25 |
Miguel Lavalle |
tags |
functional-tests gate-failure l3-ha |
functional-tests gate-failure l3-dvr-backlog |
|
| 2019-03-06 12:06:17 |
Slawek Kaplonski |
neutron: assignee |
|
Slawek Kaplonski (slaweq) |
|
| 2019-03-08 15:49:07 |
OpenStack Infra |
neutron: status |
Confirmed |
In Progress |
|
| 2019-03-14 19:08:36 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2019-03-15 10:08:22 |
Bernard Cafarelli |
tags |
functional-tests gate-failure l3-dvr-backlog |
functional-tests gate-failure l3-dvr-backlog neutron-proactive-backport-potential |
|
| 2019-03-15 10:08:28 |
Bernard Cafarelli |
tags |
functional-tests gate-failure l3-dvr-backlog neutron-proactive-backport-potential |
functional-tests gate-failure l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
|
| 2019-03-15 16:15:06 |
OpenStack Infra |
tags |
functional-tests gate-failure l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
functional-tests gate-failure in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
|
| 2019-03-15 16:15:18 |
OpenStack Infra |
tags |
functional-tests gate-failure in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
functional-tests gate-failure in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
|
| 2019-03-16 23:19:13 |
OpenStack Infra |
tags |
functional-tests gate-failure in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
|
| 2019-03-25 13:55:26 |
Edward Hope-Morley |
description |
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
[Impact]
Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.
[Test Case]
* deploy Openstack (any version that supports l3ha)
* create HA router with max-l3-agents=2
* check neutron l3-agent-list-hosting-router for master location
* on both hosts that are running the l3-agent do
pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
ps -f --ppid $pid
pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
ps -f --ppid $pid <<<<<<<<<<< this should return nothing now
pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"
* without this patch you should now see both agents reporting the router as "active"
* with the patch this should not happen (once neutron-keepalived-state-change has been restarted)
[Regression Potential]
====================================================================
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
|
| 2019-03-25 13:55:46 |
Edward Hope-Morley |
description |
[Impact]
Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.
[Test Case]
* deploy Openstack (any version that supports l3ha)
* create HA router with max-l3-agents=2
* check neutron l3-agent-list-hosting-router for master location
* on both hosts that are running the l3-agent do
pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
ps -f --ppid $pid
pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
ps -f --ppid $pid <<<<<<<<<<< this should return nothing now
pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"
* without this patch you should now see both agents reporting the router as "active"
* with the patch this should not happen (once neutron-keepalived-state-change has been restarted)
[Regression Potential]
====================================================================
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
[Impact]
Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.
[Test Case]
* deploy Openstack (any version that supports l3ha)
* create HA router with max-l3-agents=2
* check neutron l3-agent-list-hosting-router for master location
* on both hosts that are running the l3-agent do
pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
ps -f --ppid $pid
pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
ps -f --ppid $pid <<<<<<<<<<< this should return nothing now
pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"
* without this patch you should now see both agents reporting the router as "active"
* with the patch this should not happen (once neutron-keepalived-state-change has been restarted)
[Regression Potential]
None expected.
====================================================================
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
|
| 2019-03-25 13:55:54 |
Edward Hope-Morley |
summary |
Various L3HA functional tests fails often |
[SRU] Various L3HA functional tests fails often |
|
| 2019-03-25 13:57:19 |
Edward Hope-Morley |
bug task added |
|
cloud-archive |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/pike |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/pike |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/rocky |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/rocky |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/stein |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/stein |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/queens |
|
| 2019-03-25 13:57:38 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/queens |
|
| 2019-03-25 19:43:17 |
Corey Bryant |
cloud-archive/stein: status |
New |
Fix Released |
|
| 2019-03-25 19:43:25 |
Corey Bryant |
cloud-archive/rocky: status |
New |
Triaged |
|
| 2019-03-25 19:43:29 |
Corey Bryant |
cloud-archive/rocky: importance |
Undecided |
Critical |
|
| 2019-03-25 19:43:33 |
Corey Bryant |
cloud-archive/rocky: importance |
Critical |
High |
|
| 2019-03-25 19:43:36 |
Corey Bryant |
cloud-archive/queens: status |
New |
Triaged |
|
| 2019-03-25 19:43:37 |
Corey Bryant |
cloud-archive/queens: importance |
Undecided |
High |
|
| 2019-03-25 19:43:40 |
Corey Bryant |
cloud-archive/pike: status |
New |
Triaged |
|
| 2019-03-25 19:43:42 |
Corey Bryant |
cloud-archive/pike: importance |
Undecided |
High |
|
| 2019-03-25 19:44:05 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu) |
|
| 2019-03-25 19:44:52 |
Corey Bryant |
nominated for series |
|
Ubuntu Disco |
|
| 2019-03-25 19:44:52 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Disco) |
|
| 2019-03-25 19:44:52 |
Corey Bryant |
nominated for series |
|
Ubuntu Cosmic |
|
| 2019-03-25 19:44:52 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Cosmic) |
|
| 2019-03-25 19:44:52 |
Corey Bryant |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-03-25 19:44:52 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Bionic) |
|
| 2019-03-25 19:45:06 |
Corey Bryant |
neutron (Ubuntu Bionic): importance |
Undecided |
High |
|
| 2019-03-25 19:45:06 |
Corey Bryant |
neutron (Ubuntu Bionic): status |
New |
Triaged |
|
| 2019-03-25 19:45:39 |
Corey Bryant |
neutron (Ubuntu Cosmic): importance |
Undecided |
High |
|
| 2019-03-25 19:45:39 |
Corey Bryant |
neutron (Ubuntu Cosmic): status |
New |
Triaged |
|
| 2019-03-25 19:46:03 |
Corey Bryant |
neutron (Ubuntu Disco): status |
New |
Fix Released |
|
| 2019-03-25 19:46:19 |
Corey Bryant |
neutron (Ubuntu Disco): importance |
Undecided |
High |
|
| 2019-03-25 19:46:31 |
Corey Bryant |
cloud-archive/stein: importance |
Undecided |
High |
|
| 2019-03-26 13:32:36 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-03-26 13:49:33 |
Corey Bryant |
cloud-archive/pike: status |
Triaged |
Fix Committed |
|
| 2019-03-26 13:49:36 |
Corey Bryant |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-pike-needed |
|
| 2019-03-26 16:01:40 |
Corey Bryant |
description |
[Impact]
Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.
[Test Case]
* deploy Openstack (any version that supports l3ha)
* create HA router with max-l3-agents=2
* check neutron l3-agent-list-hosting-router for master location
* on both hosts that are running the l3-agent do
pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
ps -f --ppid $pid
pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
ps -f --ppid $pid <<<<<<<<<<< this should return nothing now
pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"
* without this patch you should now see both agents reporting the router as "active"
* with the patch this should not happen (once neutron-keepalived-state-change has been restarted)
[Regression Potential]
None expected.
====================================================================
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
[Impact]
Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.
[Test Case]
* deploy Openstack (any version that supports l3ha)
* create HA router with max-l3-agents=2
* check neutron l3-agent-list-hosting-router for master location
* on both hosts that are running the l3-agent do
pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
ps -f --ppid $pid
pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
ps -f --ppid $pid <<<<<<<<<<< this should return nothing now
pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"
* without this patch you should now see both agents reporting the router as "active"
* with the patch this should not happen (once neutron-keepalived-state-change has been restarted)
[Regression Potential]
These patches have already landed in corresponding upstream branches and therefore have undergone reviews + unit and functional testing upstream, therefore regression potential is expected to be low.
====================================================================
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
|
| 2019-03-26 16:06:02 |
Brian Murray |
neutron (Ubuntu Cosmic): status |
Triaged |
Fix Committed |
|
| 2019-03-26 16:06:07 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2019-03-26 16:06:11 |
Brian Murray |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-pike-needed |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-cosmic verification-pike-needed |
|
| 2019-03-26 16:55:20 |
Brian Murray |
neutron (Ubuntu Bionic): status |
Triaged |
Fix Committed |
|
| 2019-03-26 16:55:26 |
Brian Murray |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-cosmic verification-pike-needed |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed |
|
| 2019-03-26 20:11:15 |
Corey Bryant |
cloud-archive/rocky: status |
Triaged |
Fix Committed |
|
| 2019-03-26 20:11:23 |
Corey Bryant |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-rocky-needed |
|
| 2019-04-01 13:46:51 |
Corey Bryant |
cloud-archive/queens: status |
Triaged |
Fix Committed |
|
| 2019-04-01 13:46:53 |
Corey Bryant |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-rocky-needed |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed |
|
| 2019-04-04 09:26:40 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed |
|
| 2019-04-04 12:00:49 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-failed-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
|
| 2019-04-04 12:12:25 |
Magnus Bergman |
bug |
|
|
added subscriber Magnus Bergman |
| 2019-04-08 17:21:40 |
OpenStack Infra |
tags |
functional-tests gate-failure in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-failed-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-failed-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
|
| 2019-04-08 22:07:07 |
OpenStack Infra |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-failed-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-failed-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
|
| 2019-04-12 07:48:32 |
Tobias Urdin |
bug |
|
|
added subscriber Tobias Urdin |
| 2019-04-15 13:45:38 |
Łukasz Zemczak |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-failed-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed |
|
| 2019-04-15 20:23:28 |
Edward Hope-Morley |
description |
[Impact]
Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.
[Test Case]
* deploy Openstack (any version that supports l3ha)
* create HA router with max-l3-agents=2
* check neutron l3-agent-list-hosting-router for master location
* on both hosts that are running the l3-agent do
pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
ps -f --ppid $pid
pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
ps -f --ppid $pid <<<<<<<<<<< this should return nothing now
pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"
* without this patch you should now see both agents reporting the router as "active"
* with the patch this should not happen (once neutron-keepalived-state-change has been restarted)
[Regression Potential]
These patches have already landed in corresponding upstream branches and therefore have undergone reviews + unit and functional testing upstream, therefore regression potential is expected to be low.
====================================================================
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
[Impact]
Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.
[Test Case]
* deploy Openstack (any version that supports l3ha)
* create HA router with max-l3-agents=2
* check neutron l3-agent-list-hosting-router for master location
* on both hosts that are running the l3-agent do
pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
ps -f --ppid $pid
pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"
pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
ps -f --ppid $pid # <<<<<<<<<<< this should return nothing now
* without this patch you should now see both agents reporting the router as "active"
* with the patch this should not happen (once neutron-keepalived-state-change has been restarted by neutron-l3-agent)
[Regression Potential]
These patches have already landed in corresponding upstream branches and therefore have undergone reviews + unit and functional testing upstream, therefore regression potential is expected to be low.
====================================================================
Recently many L3 HA related functional tests are failing.
The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.
Example stack trace:
ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/base.py", line 174, in func
return f(self, *args, **kwargs)
File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
self._router_lifecycle(enable_ha=True, router_info=router_info)
File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
common_utils.wait_until_true(lambda: router.ha_state == 'master')
File "neutron/common/utils.py", line 690, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
Example failure: http://logs.openstack.org/79/633979/21/check/neutron-functional-python27/ce7ef07/logs/testr_results.html.gz
Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22 |
|
| 2019-04-15 20:26:10 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
|
| 2019-05-01 01:50:37 |
Chris Halse Rogers |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-05-01 02:00:46 |
Launchpad Janitor |
neutron (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-05-07 14:42:21 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-needed |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-done |
|
| 2019-05-07 16:25:10 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-cosmic verification-needed verification-needed-bionic verification-pike-needed verification-queens-needed verification-rocky-done |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-bionic verification-done-cosmic verification-needed verification-pike-needed verification-queens-needed verification-rocky-done |
|
| 2019-05-07 18:01:04 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-bionic verification-done-cosmic verification-needed verification-pike-needed verification-queens-needed verification-rocky-done |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-bionic verification-done-cosmic verification-needed verification-pike-needed verification-queens-done verification-rocky-done |
|
| 2019-05-08 10:39:46 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-bionic verification-done-cosmic verification-needed verification-pike-needed verification-queens-done verification-rocky-done |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-bionic verification-done-cosmic verification-needed verification-pike-done verification-queens-done verification-rocky-done |
|
| 2019-05-08 10:43:23 |
Edward Hope-Morley |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done-bionic verification-done-cosmic verification-needed verification-pike-done verification-queens-done verification-rocky-done |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done verification-done-bionic verification-done-cosmic verification-pike-done verification-queens-done verification-rocky-done |
|
| 2019-05-09 09:59:28 |
Launchpad Janitor |
neutron (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-05-13 19:15:30 |
Corey Bryant |
cloud-archive/rocky: status |
Fix Committed |
Fix Released |
|
| 2019-05-13 19:19:16 |
Corey Bryant |
cloud-archive/pike: status |
Fix Committed |
Fix Released |
|
| 2019-05-13 19:32:45 |
Corey Bryant |
cloud-archive/queens: status |
Fix Committed |
Fix Released |
|
| 2019-09-13 10:42:36 |
Slawek Kaplonski |
tags |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog neutron-easy-proactive-backport-potential neutron-proactive-backport-potential sts-sru-needed verification-done verification-done-bionic verification-done-cosmic verification-pike-done verification-queens-done verification-rocky-done |
functional-tests gate-failure in-stable-ocata in-stable-pike in-stable-queens in-stable-rocky in-stable-stein l3-dvr-backlog sts-sru-needed verification-done verification-done-bionic verification-done-cosmic verification-pike-done verification-queens-done verification-rocky-done |
|
