| 2015-10-26 19:03:06 |
Diana Clarke |
bug |
|
|
added bug |
| 2015-10-26 19:03:36 |
Diana Clarke |
description |
Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
* List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
* Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
* Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
* The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
Here's example output demonstrating the issue:
http://paste.openstack.org/show/477404/
See bug #1450438 for more context:
https://bugs.launchpad.net/oslo.service/+bug/1450438
Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.
https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
Oslo Service: version 0.11
Nova: Master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2) |
Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
1. List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
2. Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
3. Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
4. The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
Here's example output demonstrating the issue:
http://paste.openstack.org/show/477404/
See bug #1450438 for more context:
https://bugs.launchpad.net/oslo.service/+bug/1450438
Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.
https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
Oslo Service: version 0.11
Nova: Master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2) |
|
| 2015-10-26 19:04:14 |
Diana Clarke |
description |
Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
1. List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
2. Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
3. Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
4. The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
Here's example output demonstrating the issue:
http://paste.openstack.org/show/477404/
See bug #1450438 for more context:
https://bugs.launchpad.net/oslo.service/+bug/1450438
Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.
https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
Oslo Service: version 0.11
Nova: Master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2) |
Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
1. List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
2. Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
3. Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
4. The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
Here's example output demonstrating the issue:
http://paste.openstack.org/show/477404/
See bug #1450438 for more context:
https://bugs.launchpad.net/oslo.service/+bug/1450438
Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.
https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
Oslo Service: version 0.11
Nova: master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2) |
|
| 2015-10-26 19:06:59 |
Matt Riedemann |
tags |
|
oslo |
|
| 2015-10-26 19:12:00 |
Diana Clarke |
description |
Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
1. List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
2. Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
3. Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
4. The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
Here's example output demonstrating the issue:
http://paste.openstack.org/show/477404/
See bug #1450438 for more context:
https://bugs.launchpad.net/oslo.service/+bug/1450438
Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.
https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
Oslo Service: version 0.11
Nova: master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2) |
Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
1. List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
2. Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
3. Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
4. The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
5. You'll see a log message like the following when the heartbeats stop:
2015-10-26 17:14:10.538 DEBUG nova.servicegroup.drivers.db [req-c41a2ad7-e5a5-4914-bdc8-6c1ca8b224c6 None None] Seems service is down. Last heartbeat was 2015-10-26 17:20:20. Elapsed time is -369.461679 from (pid=13994) is_up /opt/stack/nova/nova/servicegroup/drivers/db.py:80
Here's example output demonstrating the issue:
http://paste.openstack.org/show/477404/
See bug #1450438 for more context:
https://bugs.launchpad.net/oslo.service/+bug/1450438
Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.
https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
Oslo Service: version 0.11
Nova: master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2) |
|
| 2015-10-26 19:14:49 |
Matt Riedemann |
bug task added |
|
oslo.service |
|
| 2015-10-26 19:14:57 |
Matt Riedemann |
oslo.service: status |
New |
Confirmed |
|
| 2015-10-26 19:15:03 |
Matt Riedemann |
bug task deleted |
nova |
|
|
| 2015-11-03 18:45:29 |
Chet Burgess |
bug |
|
|
added subscriber Chet Burgess |
| 2016-03-01 03:07:56 |
Davanum Srinivas (DIMS) |
bug task added |
|
nova |
|
| 2016-03-01 17:26:03 |
Sean Dague |
bug task deleted |
nova |
|
|
| 2016-04-13 17:31:17 |
Diana Clarke |
oslo.service: status |
Confirmed |
Fix Released |
|
| 2017-02-22 16:42:14 |
Roman Podoliaka |
bug task added |
|
nova |
|
| 2017-02-22 16:42:31 |
Roman Podoliaka |
nova: assignee |
|
Roman Podoliaka (rpodolyaka) |
|
| 2017-02-22 16:42:34 |
Roman Podoliaka |
nova: status |
New |
In Progress |
|
| 2017-09-06 13:46:58 |
OpenStack Infra |
nova: assignee |
Roman Podoliaka (rpodolyaka) |
Stephen Finucane (stephenfinucane) |
|
| 2017-09-07 21:01:02 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2017-09-14 04:21:43 |
Dinesh Bhor |
bug task added |
|
masakari |
|
| 2017-09-14 04:22:09 |
Dinesh Bhor |
masakari: assignee |
|
Dinesh Bhor (dinesh-bhor) |
|
| 2017-09-14 04:23:14 |
OpenStack Infra |
masakari: status |
New |
In Progress |
|
| 2017-11-14 07:15:54 |
OpenStack Infra |
masakari: assignee |
Dinesh Bhor (dinesh-bhor) |
SamP (sampath-priyankara) |
|
| 2017-11-14 07:49:22 |
OpenStack Infra |
masakari: status |
In Progress |
Fix Released |
|
| 2018-01-25 22:27:45 |
Matt Riedemann |
nominated for series |
|
nova/pike |
|
| 2018-01-25 22:27:45 |
Matt Riedemann |
bug task added |
|
nova/pike |
|
| 2018-01-25 22:28:11 |
Matt Riedemann |
nova: assignee |
Stephen Finucane (stephenfinucane) |
Roman Podoliaka (rpodolyaka) |
|
| 2018-01-25 22:28:17 |
Matt Riedemann |
nova/pike: status |
New |
In Progress |
|
| 2018-01-25 22:28:39 |
Matt Riedemann |
nova/pike: assignee |
|
John Smith (wang-zengzhi) |
|
| 2018-02-13 16:39:39 |
OpenStack Infra |
nova/pike: status |
In Progress |
Fix Committed |
|
