| 2015-03-11 19:40:00 |
Assaf Muller |
bug |
|
|
added bug |
| 2015-03-11 19:40:13 |
Assaf Muller |
neutron: assignee |
|
Assaf Muller (amuller) |
|
| 2015-03-11 20:17:57 |
Assaf Muller |
description |
Several patches merged as a part of:
https://review.openstack.org/#/q/status:merged+project:openstack/neutron+branch:master+topic:bp/rpc-docs-and-namespaces,n,z
Broke Neutron rolling upgrade (Specifically: Upgrading the server(s) before the agents or vice versa). This was done knowingly and discussed in the spec process. While we don't test a rolling upgrade scenario, there is no reason to break it knowingly. I've spoken to operators that have successfully performed such an upgrade from I to J and it will be very surprising to them if the same doesn't work from J to K.
The breakage comes from the introduction of RPC namespaces, a very useful concept of putting RPC endpoints in separate namespaces. i.e. you may place the same method name listening in the same process if it belongs to different namespaces.
Possible solutions:
Have the server listen on both the new namespaces, and in the root namespace. However, this effectively brings all such methods into one big namespace, so I don't see the point. Why not revert and forgo the usage of namespaces altogether? We could delay by making a change in Oslo messaging where if a new and optional backwards_compatibility flag is passed in to a target along with a namespace, then the dispatcher will check against the namespace as well as the root namespace, and we simply stop passing the flag in the L cycle (This means that we only support rolling upgrades from version N to N+1).
Testing:
I've been working on basic RPC tests:
https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:rpc_tests,n,z
But I don't think such a framework will allow us to test rolling upgrades. I can't think of an alternative to actually performing one and seeing what happens (A spin on the grenade job). |
Several patches merged as a part of:
https://review.openstack.org/#/q/status:merged+project:openstack/neutron+branch:master+topic:bp/rpc-docs-and-namespaces,n,z
Broke Neutron rolling upgrade (Specifically: Upgrading the server(s) before the agents or vice versa). This was done knowingly and discussed in the spec process. While we don't test a rolling upgrade scenario, there is no reason to break it knowingly. I've spoken to operators that have successfully performed such an upgrade from I to J and it will be very surprising to them if the same doesn't work from J to K.
The breakage comes from the introduction of RPC namespaces, a very useful concept of putting RPC endpoints in separate namespaces. i.e. you may place the same method name listening in the same process if it belongs to different namespaces.
Possible solutions:
Have the server listen on both the new namespaces, and in the root namespace. However, this effectively brings all such methods into one big namespace, so this kind of defeats the purpose of namespacing. We could delay by making a change in Oslo messaging where if a new and optional backwards_compatibility flag is passed in to a target along with a namespace, then the dispatcher will check against the namespace as well as the root namespace, and we simply stop passing the flag in the L cycle (This means that we only support rolling upgrades from version N to N+1). In order to support a scenario where an agent is upgraded before a sever, then even with the proposed solution, all of the K agents would have to implement a fallback.
Testing:
I've been working on basic RPC tests:
https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:rpc_tests,n,z
But I don't think such a framework will allow us to test rolling upgrades. I can't think of an alternative to actually performing one and seeing what happens (A spin on the grenade job). |
|
| 2015-03-11 20:18:35 |
Assaf Muller |
description |
Several patches merged as a part of:
https://review.openstack.org/#/q/status:merged+project:openstack/neutron+branch:master+topic:bp/rpc-docs-and-namespaces,n,z
Broke Neutron rolling upgrade (Specifically: Upgrading the server(s) before the agents or vice versa). This was done knowingly and discussed in the spec process. While we don't test a rolling upgrade scenario, there is no reason to break it knowingly. I've spoken to operators that have successfully performed such an upgrade from I to J and it will be very surprising to them if the same doesn't work from J to K.
The breakage comes from the introduction of RPC namespaces, a very useful concept of putting RPC endpoints in separate namespaces. i.e. you may place the same method name listening in the same process if it belongs to different namespaces.
Possible solutions:
Have the server listen on both the new namespaces, and in the root namespace. However, this effectively brings all such methods into one big namespace, so this kind of defeats the purpose of namespacing. We could delay by making a change in Oslo messaging where if a new and optional backwards_compatibility flag is passed in to a target along with a namespace, then the dispatcher will check against the namespace as well as the root namespace, and we simply stop passing the flag in the L cycle (This means that we only support rolling upgrades from version N to N+1). In order to support a scenario where an agent is upgraded before a sever, then even with the proposed solution, all of the K agents would have to implement a fallback.
Testing:
I've been working on basic RPC tests:
https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:rpc_tests,n,z
But I don't think such a framework will allow us to test rolling upgrades. I can't think of an alternative to actually performing one and seeing what happens (A spin on the grenade job). |
Several patches merged as a part of:
https://review.openstack.org/#/q/status:merged+project:openstack/neutron+branch:master+topic:bp/rpc-docs-and-namespaces,n,z
Broke Neutron rolling upgrade (Specifically: Upgrading the server(s) before the agents or vice versa). This was done knowingly and discussed in the spec process. While we don't test a rolling upgrade scenario, there is no reason to break it knowingly. I've spoken to operators that have successfully performed such an upgrade from I to J and it will be very surprising to them if the same doesn't work from J to K.
The breakage comes from the introduction of RPC namespaces, a very useful concept of putting RPC endpoints in separate namespaces. i.e. you may place the same method name listening in the same process if it belongs to different namespaces.
Possible solutions:
Have the server listen on both the new namespaces, and in the root namespace. However, this effectively brings all such methods into one big namespace, so this kind of defeats the purpose of namespacing. We could delay by making a change in Oslo messaging where if a new and optional backwards_compatibility flag is passed in to a target along with a namespace, then the dispatcher will check against the namespace as well as the root namespace, and we simply stop passing the flag in the L cycle (This means that we only support rolling upgrades from version N to N+1). In order to support a scenario where an agent is upgraded before a server, then even with the proposed solution, all of the K agents would have to implement a fallback.
Testing:
I've been working on basic RPC tests:
https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:rpc_tests,n,z
But I don't think such a framework will allow us to test rolling upgrades. I can't think of an alternative to actually performing one and seeing what happens (A spin on the grenade job). |
|
| 2015-03-11 20:27:18 |
Assaf Muller |
bug task added |
|
oslo.messaging |
|
| 2015-03-12 00:05:53 |
Takashi Natsume |
bug |
|
|
added subscriber Takashi NATSUME |
| 2015-03-12 01:39:52 |
Assaf Muller |
oslo.messaging: assignee |
|
Assaf Muller (amuller) |
|
| 2015-03-12 01:51:56 |
OpenStack Infra |
oslo.messaging: status |
New |
In Progress |
|
| 2015-03-12 05:55:57 |
Vijay Chundury |
bug |
|
|
added subscriber Vijay Chundury |
| 2015-03-12 10:25:24 |
gustavo panizzo |
bug |
|
|
added subscriber gustavo panizzo |
| 2015-03-12 11:35:57 |
Ihar Hrachyshka |
neutron: milestone |
|
kilo-3 |
|
| 2015-03-13 11:38:29 |
OpenStack Infra |
oslo.messaging: status |
In Progress |
Fix Committed |
|
| 2015-03-16 13:42:04 |
Assaf Muller |
neutron: status |
New |
In Progress |
|
| 2015-03-19 14:10:41 |
Kyle Mestery |
neutron: milestone |
kilo-3 |
kilo-rc1 |
|
| 2015-03-23 17:51:20 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2015-03-23 21:31:36 |
Kyle Mestery |
neutron: importance |
Undecided |
High |
|
| 2015-03-23 21:32:02 |
Kyle Mestery |
neutron: status |
In Progress |
Fix Committed |
|
| 2015-03-24 17:19:08 |
OpenStack Infra |
tags |
|
in-stable-kilo |
|
| 2015-03-25 14:04:35 |
Doug Hellmann |
oslo.messaging: status |
Fix Committed |
Fix Released |
|
| 2015-03-25 14:04:35 |
Doug Hellmann |
oslo.messaging: milestone |
|
1.8.1 |
|
| 2015-04-09 18:43:42 |
Thierry Carrez |
neutron: status |
Fix Committed |
Fix Released |
|
| 2015-04-30 09:50:03 |
Thierry Carrez |
neutron: milestone |
kilo-rc1 |
2015.1.0 |
|
