| 2021-02-03 14:42:36 |
Chris MacNaughton |
bug |
|
|
added bug |
| 2021-02-03 14:42:59 |
Chris MacNaughton |
summary |
[SRU] |
[SRU] Recent update broke message handling |
|
| 2021-02-03 14:43:15 |
Chris MacNaughton |
summary |
[SRU] Recent update broke message handling |
[SRU] MessageTimeout and DuplicateMessage errors after udpate |
|
| 2021-02-03 15:39:27 |
Corey Bryant |
bug task added |
|
python-oslo.messaging (Ubuntu) |
|
| 2021-02-03 15:39:38 |
Corey Bryant |
nominated for series |
|
Ubuntu Bionic |
|
| 2021-02-03 15:39:38 |
Corey Bryant |
bug task added |
|
python-oslo.messaging (Ubuntu Bionic) |
|
| 2021-02-03 15:39:45 |
Corey Bryant |
python-oslo.messaging (Ubuntu Bionic): status |
New |
Triaged |
|
| 2021-02-03 15:39:47 |
Corey Bryant |
python-oslo.messaging (Ubuntu Bionic): importance |
Undecided |
Critical |
|
| 2021-02-03 15:39:50 |
Corey Bryant |
python-oslo.messaging (Ubuntu): status |
New |
Invalid |
|
| 2021-02-03 15:39:59 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2021-02-03 15:40:17 |
Corey Bryant |
nominated for series |
|
cloud-archive/rocky |
|
| 2021-02-03 15:40:17 |
Corey Bryant |
bug task added |
|
cloud-archive/rocky |
|
| 2021-02-03 15:40:17 |
Corey Bryant |
nominated for series |
|
cloud-archive/stein |
|
| 2021-02-03 15:40:17 |
Corey Bryant |
bug task added |
|
cloud-archive/stein |
|
| 2021-02-03 15:40:17 |
Corey Bryant |
nominated for series |
|
cloud-archive/queens |
|
| 2021-02-03 15:40:17 |
Corey Bryant |
bug task added |
|
cloud-archive/queens |
|
| 2021-02-03 15:40:34 |
Corey Bryant |
cloud-archive/stein: status |
New |
Triaged |
|
| 2021-02-03 15:40:37 |
Corey Bryant |
cloud-archive/rocky: status |
New |
Triaged |
|
| 2021-02-03 15:40:40 |
Corey Bryant |
cloud-archive/queens: status |
New |
Triaged |
|
| 2021-02-03 15:40:44 |
Corey Bryant |
cloud-archive/stein: importance |
Undecided |
Critical |
|
| 2021-02-03 15:40:45 |
Corey Bryant |
cloud-archive/rocky: importance |
Undecided |
Critical |
|
| 2021-02-03 15:40:47 |
Corey Bryant |
cloud-archive/queens: importance |
Undecided |
Critical |
|
| 2021-02-03 15:40:50 |
Corey Bryant |
cloud-archive: status |
New |
Invalid |
|
| 2021-02-03 15:41:54 |
Chris MacNaughton |
bug task deleted |
cloud-archive/queens |
|
|
| 2021-02-03 15:50:07 |
Corey Bryant |
nominated for series |
|
cloud-archive/queens |
|
| 2021-02-03 15:50:07 |
Corey Bryant |
bug task added |
|
cloud-archive/queens |
|
| 2021-02-03 15:50:14 |
Corey Bryant |
cloud-archive/queens: status |
New |
Incomplete |
|
| 2021-02-03 15:50:16 |
Corey Bryant |
cloud-archive/queens: status |
Incomplete |
Triaged |
|
| 2021-02-03 15:50:18 |
Corey Bryant |
cloud-archive/queens: importance |
Undecided |
Critical |
|
| 2021-02-03 15:54:32 |
Peter Sabaini |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2021-02-03 17:17:43 |
Robie Basak |
tags |
regression-updates |
regression-update |
|
| 2021-02-03 17:18:06 |
Robie Basak |
description |
A recent update to oslo.messaging to resolve #1789177 causes failures.
(Below comments copied form the original bug):
After a partial upgrade (only one side, producers or consumers), there are a lot of MessageTimeout and DuplicateMessage errors in the logs. Downgrading back to 5.35.0-0ubuntu1~cloud0 fixed the problem.
Right after restarted n-ovs-agent, I can see a lot of errors in rabbitmq log[1]
which is the same as the error when rabbitmq failover issue ( the original issue of this LP )
Then after I upgraded oslo.messaging in neutron-api unit and restarted neutron-server, below errors are gone and I was able to create instance again.
After upgrading oslo.messaging in n-ovs only, exchange they communicate didn't match.
As changing exchanges they use depends on publisher-cosumer relation.
So I think there are two ways.
1. revert this patch for Q ( original failover problem will be there )
2. upgrade them with maintenance window
Thanks a lot
[1]
################################################################################
=ERROR REPORT==== 3-Feb-2021::03:25:26 ===
Channel error on connection <0.2379.1> (10.0.0.32:60430 -> 10.0.0.34:5672, vhost: 'openstack', user: 'neutron'), channel 1:
{amqp_error,not_found,
"no exchange 'reply_7da3cecc31b34bdeb96c866dc84e3044' in vhost 'openstack'",
'basic.publish'}
10.0.0.32 is neutron-api unit |
A recent update to oslo.messaging to resolve bug 1789177 causes failures.
(Below comments copied form the original bug):
After a partial upgrade (only one side, producers or consumers), there are a lot of MessageTimeout and DuplicateMessage errors in the logs. Downgrading back to 5.35.0-0ubuntu1~cloud0 fixed the problem.
Right after restarted n-ovs-agent, I can see a lot of errors in rabbitmq log[1]
which is the same as the error when rabbitmq failover issue ( the original issue of this LP )
Then after I upgraded oslo.messaging in neutron-api unit and restarted neutron-server, below errors are gone and I was able to create instance again.
After upgrading oslo.messaging in n-ovs only, exchange they communicate didn't match.
As changing exchanges they use depends on publisher-cosumer relation.
So I think there are two ways.
1. revert this patch for Q ( original failover problem will be there )
2. upgrade them with maintenance window
Thanks a lot
[1]
################################################################################
=ERROR REPORT==== 3-Feb-2021::03:25:26 ===
Channel error on connection <0.2379.1> (10.0.0.32:60430 -> 10.0.0.34:5672, vhost: 'openstack', user: 'neutron'), channel 1:
{amqp_error,not_found,
"no exchange 'reply_7da3cecc31b34bdeb96c866dc84e3044' in vhost 'openstack'",
'basic.publish'}
10.0.0.32 is neutron-api unit |
|
| 2021-02-03 17:54:15 |
Robie Basak |
python-oslo.messaging (Ubuntu Bionic): status |
Triaged |
Fix Committed |
|
| 2021-02-03 17:54:17 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-02-03 17:54:19 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2021-02-03 17:54:22 |
Robie Basak |
tags |
regression-update |
regression-update verification-needed verification-needed-bionic |
|
| 2021-02-03 18:02:53 |
Corey Bryant |
description |
A recent update to oslo.messaging to resolve bug 1789177 causes failures.
(Below comments copied form the original bug):
After a partial upgrade (only one side, producers or consumers), there are a lot of MessageTimeout and DuplicateMessage errors in the logs. Downgrading back to 5.35.0-0ubuntu1~cloud0 fixed the problem.
Right after restarted n-ovs-agent, I can see a lot of errors in rabbitmq log[1]
which is the same as the error when rabbitmq failover issue ( the original issue of this LP )
Then after I upgraded oslo.messaging in neutron-api unit and restarted neutron-server, below errors are gone and I was able to create instance again.
After upgrading oslo.messaging in n-ovs only, exchange they communicate didn't match.
As changing exchanges they use depends on publisher-cosumer relation.
So I think there are two ways.
1. revert this patch for Q ( original failover problem will be there )
2. upgrade them with maintenance window
Thanks a lot
[1]
################################################################################
=ERROR REPORT==== 3-Feb-2021::03:25:26 ===
Channel error on connection <0.2379.1> (10.0.0.32:60430 -> 10.0.0.34:5672, vhost: 'openstack', user: 'neutron'), channel 1:
{amqp_error,not_found,
"no exchange 'reply_7da3cecc31b34bdeb96c866dc84e3044' in vhost 'openstack'",
'basic.publish'}
10.0.0.32 is neutron-api unit |
[Impact]
A recent update to oslo.messaging to resolve #1789177 causes failures.
(Below comments copied form the original bug):
After a partial upgrade (only one side, producers or consumers), there are a lot of MessageTimeout and DuplicateMessage errors in the logs. Downgrading back to 5.35.0-0ubuntu1~cloud0 fixed the problem.
Right after restarted n-ovs-agent, I can see a lot of errors in rabbitmq log[1]
which is the same as the error when rabbitmq failover issue ( the original issue of this LP )
Then after I upgraded oslo.messaging in neutron-api unit and restarted neutron-server, below errors are gone and I was able to create instance again.
After upgrading oslo.messaging in n-ovs only, exchange they communicate didn't match.
As changing exchanges they use depends on publisher-cosumer relation.
So I think there are two ways.
1. revert this patch for Q ( original failover problem will be there )
2. upgrade them with maintenance window
Thanks a lot
[1]
################################################################################
=ERROR REPORT==== 3-Feb-2021::03:25:26 ===
Channel error on connection <0.2379.1> (10.0.0.32:60430 -> 10.0.0.34:5672, vhost: 'openstack', user: 'neutron'), channel 1:
{amqp_error,not_found,
"no exchange 'reply_7da3cecc31b34bdeb96c866dc84e3044' in vhost 'openstack'",
'basic.publish'}
10.0.0.32 is neutron-api unit
[Test Case]
This SRU needs the following scenarios tested:
1) partial upgrade of n-ovs at 5.35.0-0ubuntu3 [1] and n-api/n-gateway at 5.35.0-0ubuntu1 - instance creation will be successful
2) partial upgrade of n-api/n-gateway at 5.35.0-0ubuntu3 [1] and n-ovs at 5.35.0-0ubuntu1 - instance creation will be successful
3) partial upgrade of n-ovs at 5.35.0-0ubuntu2 [1] and n-api/n-gateway at 5.35.0-0ubuntu3 - instance creation will fail (see regression potential)
4) partial upgrade of n-api/n-gateway at 5.35.0-0ubuntu3 [1] and n-ovs at 5.35.0-0ubuntu2 - instance creation will fail (see regression potential)
5) test all neutron nodes at 5.35.0-0ubunt3 - instance creation will be successful
[1] and neutron* services restarted
[Regression Potential]
There is regression potential for clouds that have already upgraded to 5.35.0-0ubuntu2. This needs to be tested but if a cloud has fully upgraded to 5.35.0-0ubuntu2, then the same disruption that this SRU is trying to solve may once again occur in a cloud with some services running 5.35.0-0ubuntu2 and some running 5.35.0-0ubuntu3. Once that cloud is entirely at 5.35.0-0ubuntu3, messages will no longer timeout. |
|
| 2021-02-03 18:15:18 |
Corey Bryant |
cloud-archive/stein: status |
Triaged |
Fix Committed |
|
| 2021-02-03 18:15:20 |
Corey Bryant |
tags |
regression-update verification-needed verification-needed-bionic |
regression-update verification-needed verification-needed-bionic verification-stein-needed |
|
| 2021-02-03 19:31:30 |
Corey Bryant |
cloud-archive/rocky: status |
Triaged |
Fix Committed |
|
| 2021-02-03 19:31:31 |
Corey Bryant |
tags |
regression-update verification-needed verification-needed-bionic verification-stein-needed |
regression-update verification-needed verification-needed-bionic verification-rocky-needed verification-stein-needed |
|
| 2021-02-03 21:09:54 |
Corey Bryant |
cloud-archive/queens: status |
Triaged |
Fix Committed |
|
| 2021-02-03 21:09:55 |
Corey Bryant |
tags |
regression-update verification-needed verification-needed-bionic verification-rocky-needed verification-stein-needed |
regression-update verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-stein-needed |
|
| 2021-02-03 22:07:52 |
Corey Bryant |
tags |
regression-update verification-needed verification-needed-bionic verification-queens-needed verification-rocky-needed verification-stein-needed |
regression-update verification-done-bionic verification-needed verification-rocky-done verification-stein-done |
|
| 2021-02-03 22:08:01 |
Corey Bryant |
tags |
regression-update verification-done-bionic verification-needed verification-rocky-done verification-stein-done |
regression-update verification-done verification-done-bionic verification-rocky-done verification-stein-done |
|
| 2021-02-03 22:08:50 |
Corey Bryant |
tags |
regression-update verification-done verification-done-bionic verification-rocky-done verification-stein-done |
regression-update verification-done verification-done-bionic verification-queens-needed verification-rocky-done verification-stein-done |
|
| 2021-02-03 22:09:51 |
Corey Bryant |
cloud-archive/stein: status |
Fix Committed |
Fix Released |
|
| 2021-02-03 22:10:33 |
Corey Bryant |
cloud-archive/rocky: status |
Fix Committed |
Fix Released |
|
| 2021-02-03 23:23:05 |
Corey Bryant |
tags |
regression-update verification-done verification-done-bionic verification-queens-needed verification-rocky-done verification-stein-done |
regression-update verification-done verification-done-bionic verification-queens-done verification-rocky-done verification-stein-done |
|
| 2021-02-04 08:36:24 |
Nikolay Vinogradov |
bug |
|
|
added subscriber Canonical Field Critical |
| 2021-02-04 13:15:12 |
Launchpad Janitor |
python-oslo.messaging (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2021-02-04 13:17:05 |
Corey Bryant |
cloud-archive/queens: status |
Fix Committed |
Fix Released |
|
| 2021-02-04 13:38:25 |
Nikolay Vinogradov |
removed subscriber Canonical Field Critical |
|
|
|
| 2021-02-04 13:43:52 |
Corey Bryant |
oslo.messaging: status |
New |
Invalid |
|
| 2021-02-04 21:46:23 |
Brian Murray |
summary |
[SRU] MessageTimeout and DuplicateMessage errors after udpate |
[SRU] MessageTimeout and DuplicateMessage errors after update |
|
| 2021-02-05 02:58:17 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
| 2021-02-18 16:39:24 |
James Page |
nominated for series |
|
cloud-archive/train |
|
| 2021-02-18 16:39:24 |
James Page |
bug task added |
|
cloud-archive/train |
|
| 2021-02-19 15:57:34 |
Victor Morales |
bug |
|
|
added subscriber Victor Morales |
