services hang when time is jumping forward and backward

[Eugene Nikanorov]

MOS 9.0

Consider the following scenario:
The cluster is power-cycled, some nodes have their system time reset due to hardware.
When the cluster is up, ntp starts synchronizing the time. Meanwhile, services try to connect to rabbitmq and waiting for rabbitmq to be available using time.sleep(), monkeypatched by eventlet.
For some reason during the sync the time gets adjusted forward and backwards several years (that much because default systime is 200x-01-01), that makes monkeypatched time.sleep($interval) to wait additional $interval time.

The root cause is that eventlet is using non-monotonic timer.
Related patch that fixes the issue for oslo: https://review.openstack.org/#/c/190372/
That approach should be extended to all services that utilize oslo_service

[Roman Podoliaka (rpodolyaka)]

>>> That approach should be extended to all services that utilize oslo_service

My understanding is that oslo_service part of the problem is actually fixed and now we need to make eventlet use a monotonic clock everywhere. The main thing we are interested in is probably Hub class: https://github.com/eventlet/eventlet/blob/master/eventlet/hubs/hub.py#L116

But it's initialized implicitly when used in OpenStack:

http://paste.openstack.org/show/589381/

thus, we'll probably need to patch eventlet directly. I took a quick look at https://github.com/eventlet/eventlet/blob/master/eventlet/hubs/hub.py module and looks like using of a monotonic clock there must be fine, as we are not really interested in time value itself, but only in differences between two given points of time.

But there are more usages of time.time() in eventlet - http://paste.openstack.org/show/589384/ - we'll need to check those as well.

I don't think this is suitable for a stable release, though. This must be thoroughly tested in our current development branches first.

[Kevin Benton (kevinbenton)]

This doesn't actually have anything to do with the eventlet monkey patch of time.sleep IIUC. It's in the calculation of the $interval that we pass into it.

At least that's what I've gathered from the fix applied in that patch, they are fixing their calculations of time deltas by not using clock references. I don't see where they are replacing a call to time.sleep itself.

[Kevin Benton (kevinbenton)]

Ah, based on Roman's response, there could be both cases. eventlet itself could be busted by clock jumps and the services (e.g. Neutron) could be screwed up by clock jumps when calculating intervals based on the clock.

[Dmitry Teselkin (teselkin-d)]

Related to python components, passing to mos-packaging.

[Vitaly Sedelnik (vsedelnik)]

Making all services sustainable against clock jumps seems to be pretty big effort, let's try to fix for 10.0 and see what could be backported to 9.x.

As for 9.2 fix: Anton - you set it to customer-found - are time jumps reproducing constantly in customer environment? Can we just fix NTP to get things to work?

[Eugene Nikanorov (enikanorov)]

Well, when I said "all services" I meant making change in some library that is used by most of them.
One option is oslo_service, the work there has been done partially already.

[Dmitry Mescheryakov (dmitrymex)]

I have backorted https://review.openstack.org/#/c/286838/ into 9.2 here: https://review.fuel-infra.org/#/c/28713/

https://review.openstack.org/#/c/190372/ is already there starting from 8.0.

But as Roman pointed out, there are a lot more places in eventlet where time still is calculated incorrectly and we need to fix them before we can declare the bug to be fixed.

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 9.2, 8.0-updates, 7.0-updates as this is not a bug but feature request. We will consider implementing it in 10.

[Roman Podoliaka (rpodolyaka)]

Upstream patches:

nova - https://review.openstack.org/434327
eventlet - https://github.com/eventlet/eventlet/commit/f92b49f77e073dad40a71858ef9149b543e07d67