| 2014-10-31 12:58:29 |
James Page |
bug |
|
|
added bug |
| 2014-10-31 13:01:08 |
James Page |
description |
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
Switches power state reporting from being a serial process on each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance densities, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread. |
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
Switches power state reporting from being a serial process on each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance densities, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread.
The net result of this activity is that it places increase demands on the message broker, which has to deal with more parallel connections, and the conductors as they try to consume all of the RPC requests; if the message broker hits its memory high water mark it will stop publishers publishing any more messages until the memory usage drops below the high water mark again - this might not be achievable if all conductor processes are tied up with existing RPC calls try to send replies, resulting in a message broker lockup and collapse of all RPC in the cloud. |
|
| 2014-10-31 13:01:22 |
James Page |
description |
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
Switches power state reporting from being a serial process on each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance densities, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread.
The net result of this activity is that it places increase demands on the message broker, which has to deal with more parallel connections, and the conductors as they try to consume all of the RPC requests; if the message broker hits its memory high water mark it will stop publishers publishing any more messages until the memory usage drops below the high water mark again - this might not be achievable if all conductor processes are tied up with existing RPC calls try to send replies, resulting in a message broker lockup and collapse of all RPC in the cloud. |
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
Switches power state reporting from being a serial process for each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance densities, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread.
The net result of this activity is that it places increase demands on the message broker, which has to deal with more parallel connections, and the conductors as they try to consume all of the RPC requests; if the message broker hits its memory high water mark it will stop publishers publishing any more messages until the memory usage drops below the high water mark again - this might not be achievable if all conductor processes are tied up with existing RPC calls try to send replies, resulting in a message broker lockup and collapse of all RPC in the cloud. |
|
| 2014-10-31 13:02:01 |
James Page |
bug task added |
|
nova |
|
| 2014-10-31 13:02:19 |
James Page |
summary |
Parallel periodic power state reporting from compute nodes has high impact on conductors |
Parallel periodic power state reporting from compute nodes has high impact on conductors and message broker |
|
| 2014-10-31 13:04:37 |
James Page |
description |
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
Switches power state reporting from being a serial process for each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance densities, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread.
The net result of this activity is that it places increase demands on the message broker, which has to deal with more parallel connections, and the conductors as they try to consume all of the RPC requests; if the message broker hits its memory high water mark it will stop publishers publishing any more messages until the memory usage drops below the high water mark again - this might not be achievable if all conductor processes are tied up with existing RPC calls try to send replies, resulting in a message broker lockup and collapse of all RPC in the cloud. |
Environment: OpenStack Juno release/Ubuntu 14.04/480 compute nodes/8 cloud controllers/40,000 instances +
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
switches power state reporting from being a serial process for each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance densities, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread.
The net result of this activity is that it places increase demands on the message broker, which has to deal with more parallel connections, and the conductors as they try to consume all of the RPC requests; if the message broker hits its memory high water mark it will stop publishers publishing any more messages until the memory usage drops below the high water mark again - this might not be achievable if all conductor processes are tied up with existing RPC calls try to send replies, resulting in a message broker lockup and collapse of all RPC in the cloud. |
|
| 2014-10-31 13:06:50 |
James Page |
summary |
Parallel periodic power state reporting from compute nodes has high impact on conductors and message broker |
Parallel periodic instance power state reporting from compute nodes has high impact on conductors and message broker |
|
| 2014-10-31 13:07:21 |
James Page |
description |
Environment: OpenStack Juno release/Ubuntu 14.04/480 compute nodes/8 cloud controllers/40,000 instances +
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
switches power state reporting from being a serial process for each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance densities, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread.
The net result of this activity is that it places increase demands on the message broker, which has to deal with more parallel connections, and the conductors as they try to consume all of the RPC requests; if the message broker hits its memory high water mark it will stop publishers publishing any more messages until the memory usage drops below the high water mark again - this might not be achievable if all conductor processes are tied up with existing RPC calls try to send replies, resulting in a message broker lockup and collapse of all RPC in the cloud. |
Environment: OpenStack Juno release/Ubuntu 14.04/480 compute nodes/8 cloud controllers/40,000 instances +
The change made in:
https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545
switches power state reporting from being a serial process for each instance on a hypervisor to being a parallel thread for every instance; for clouds running high instance counts, this has quite an impact on the conductor processes as they try to deal with N instance refresh calls in parallel where N is the number of instances running on the cloud.
It might be better to throttle this to a configurable parallel level so that period RPC load can be managed effectively in a larger cloud, or to continue todo this process in series but outside of the main thread.
The net result of this activity is that it places increase demands on the message broker, which has to deal with more parallel connections, and the conductors as they try to consume all of the RPC requests; if the message broker hits its memory high water mark it will stop publishers publishing any more messages until the memory usage drops below the high water mark again - this might not be achievable if all conductor processes are tied up with existing RPC calls try to send replies, resulting in a message broker lockup and collapse of all RPC in the cloud. |
|
| 2014-10-31 13:15:53 |
James Page |
tags |
|
juno scale-testing |
|
| 2014-11-12 17:29:31 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2014-11-12 17:29:31 |
OpenStack Infra |
nova: assignee |
|
James Page (james-page) |
|
| 2014-11-28 15:18:53 |
Chuck Short |
nova (Ubuntu): status |
New |
In Progress |
|
| 2014-12-03 15:04:22 |
James Page |
nominated for series |
|
Ubuntu Vivid |
|
| 2014-12-03 15:04:22 |
James Page |
bug task added |
|
nova (Ubuntu Vivid) |
|
| 2014-12-03 15:04:22 |
James Page |
nominated for series |
|
Ubuntu Utopic |
|
| 2014-12-03 15:04:22 |
James Page |
bug task added |
|
nova (Ubuntu Utopic) |
|
| 2014-12-16 16:08:10 |
James Page |
nova (Ubuntu Utopic): status |
New |
In Progress |
|
| 2014-12-16 16:08:16 |
James Page |
nova (Ubuntu Utopic): status |
In Progress |
Fix Committed |
|
| 2015-01-13 16:10:33 |
James Page |
nova (Ubuntu Vivid): status |
In Progress |
Fix Released |
|
| 2015-01-13 16:10:41 |
James Page |
nova (Ubuntu Utopic): status |
Fix Committed |
Fix Released |
|
| 2015-02-18 15:39:45 |
Davanum Srinivas (DIMS) |
nova: importance |
Undecided |
Medium |
|
| 2015-10-20 10:21:30 |
James Page |
nova: assignee |
James Page (james-page) |
|
|
| 2016-03-03 23:47:47 |
Davanum Srinivas (DIMS) |
nova: status |
In Progress |
Confirmed |
|
| 2016-07-05 09:54:23 |
Markus Zoeller (markus_z) |
nova: importance |
Medium |
Undecided |
|
| 2016-07-05 09:54:23 |
Markus Zoeller (markus_z) |
nova: status |
Confirmed |
Expired |
|
