Don't use the local compute time when syncing

Bug #1101839 reported by Joshua Harlow
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Opinion
Wishlist
Unassigned

Bug Description

Right now there is a strong tendency to rely on NTP for determining if services are up or down, especially compute nodes. This has been problematic since it is very fragile in its implementation (aka when NTP gets slightly out of sync on any compute node then that compute node will no longer be useable). It seems simpler to let the database decide what is "time" using its own internal functions like NOW() and such and not worry about time being in sync on the other nodes...

Examples of this:

https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L502 (note the time is from the caller, not from the db)... and https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L276

Tags: db
Revision history for this message
Joshua Harlow (harlowja) wrote :
Revision history for this message
Boris Pavlovic (boris-42) wrote :

I don't like also using time from nodes. But it is not so simple to use SQL NOW() in our case, because there is no support for NOW() in sqlite. So it is not so simple as change nova.openstack.common.timeutils.utcnow() to return SQL NOW().

Revision history for this message
Russell Bryant (russellb) wrote :

We don't require ultra-precise time precision between nodes. Getting a few seconds out of sync even should be ok ... so relying on NTP seems reasonable to me.

Anyway, setting this to Opinion/Wishlist as a feature request.

Changed in nova:
importance: Undecided → Wishlist
status: New → Opinion
Revision history for this message
Tim Daly Jr. (timjr) wrote :

It's obviously fragile, since simple clock skew can render nova-compute processes unavailable. You should not compare times from different machines in a distributed system in this way, even with NTP. You might try reading something like "Time, Clocks, and the Ordering of Events in a Distributed System" by Lamport, and then see how you feel about it.

SQLite does support a "now" function, but it's not spelled the same:

sqlite> create table foo(zot);
sqlite> insert into foo select datetime('now');
sqlite> insert into foo select datetime('now');
sqlite> insert into foo select datetime('now');
sqlite> select * from foo;
2013-02-26 04:04:22
2013-02-26 04:04:33
2013-02-26 04:04:34
sqlite>

aeva black (tenbrae)
tags: added: db
Changed in nova:
assignee: nobody → Boris Pavlovic (boris-42)
Revision history for this message
aeva black (tenbrae) wrote :

Having accurate timing in distributed systems is actually really important, and time skew can cause issues (eg, scheduler thinking a compute node is dead). Even if I might be tempted to blame the deployer for not properly managing ntpd, the problem is preventable by not relying on each compute host's local clock where possible.

Changed in nova:
importance: Wishlist → High
status: Opinion → Triaged
Revision history for this message
Joshua Harlow (harlowja) wrote :

I wish I could blame the deployer but sometimes its out of the deployers hand or configuration gets misapplied, or a file gets adjusted by accident, or configuration management makes an oopsies. All things that yes, shouldn't happen, but do just because computers are ran by programmers and programmers are human and humans make mistakes (thus computers make mistakes). Having a system which can be resistant to said 'fragility' (caused by said mistakes or misc. reasons) makes the total system more robust in general.

Revision history for this message
aeva black (tenbrae) wrote :

Exactly.

However, I think the risk of introducing new bugs when we start changing how processes record time is high enough that we shouldn't change it this late in the cycle. Targeting to H-1.

Changed in nova:
milestone: none → havana-1
Revision history for this message
Boris Pavlovic (boris-42) wrote :

I agree with Devananda. This work is for H-1

Revision history for this message
Joshua Harlow (harlowja) wrote :

Agreed. In general maybe we should talk at the summit about getting rid of other places that use localtime. Thoughts?

Revision history for this message
Boris Pavlovic (boris-42) wrote :

I made BP for this thing because it is pretty complex for implementation
https://blueprints.launchpad.net/nova/+spec/use-db-time

Revision history for this message
aeva black (tenbrae) wrote :

BP has been updated based on (my recollection of) the discussion we had at the summit. summary:
- nova compute and scheduler shouldn't start if time skew is "too large"
- periodic task checks time skew, does configurable $thing if skew is "too large"

Changed in nova:
milestone: havana-1 → none
Revision history for this message
Jay Pipes (jaypipes) wrote :

Unassigning from Boris (sorry, Boris!) and putting to Incomplete status, since there is a blueprint (not a new-style spec, though), and I'm not sure what the direction here should be, and what might have changed in the >1 year since this was updated.

Changed in nova:
status: Triaged → Incomplete
assignee: Boris Pavlovic (boris-42) → nobody
Revision history for this message
Sean Dague (sdague) wrote :

Putting into wishlist state, this is really a new feature, and clearly no one is working on it.

Changed in nova:
importance: High → Wishlist
status: Incomplete → Confirmed
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

This wishlist bug has been open a year without any activity. I'm going
to move it to "Opinion / Wishlist", which is an easily-obtainable queue
of older requests that have come on. If you decide to work on this
consider using a blueprint [1] (with spec [2]). I'll recommend to
read [3] if not yet done.

If there are any questions left, feel free to contact me (markus_z)
in the IRC channel #openstack-nova

References:
[1] https://blueprints.launchpad.net/nova/
[2] https://github.com/openstack/nova-specs
[3] https://wiki.openstack.org/wiki/Blueprints

Changed in nova:
status: Confirmed → Opinion
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.