VBox VMs has to sync time after restoring

[Timur Sufiev]

Image the bug was reproced on: fuel-5.1-4-2014-09-12_00-01-11.iso

The POST request to http://172.16.0.2:5000/v3/auth/tokens is sent via Chrome REST Client on 2014-09-15, headers: Content-Type: application/json, Body: {"auth": {"identity": {"methods": ["password"], "password": {"user": {"name": "tsufiev", "password": "swordfish", "domain": {"id": "default"}}}}}}

Notice that JSON data being returned has "expires_at": "2014-09-12T17:03:47.285887Z" - i.e. 3 days before the actual request was issued!

{"token": {"methods": ["password"], "roles": [{"id": "706b49e5f9094163b4e108bda8a48beb", "name": "admin"}], "expires_at": "2014-09-12T17:03:47.285887Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "135b0cc9a8bf41b999b8046163ba0204", "name": "tsufiev"}, "catalog": [{"endpoints": [{"url": "http://192.168.0.3:8773/services/Cloud", "region": "RegionOne", "interface": "internal", "id": "574995c65e23402d93cba68ba2f997e1"}, {"url": "http://172.16.0.2:8773/services/Cloud", "region": "RegionOne", "interface": "public", "id": "cb50ae885343455fb33b95e6e7913660"}, {"url": "http://192.168.0.3:8773/services/Admin", "region": "RegionOne", "interface": "admin", "id": "dfc28979710b4127be449251a6b8868f"}], "type": "ec2", "id": "0f2cd49e19a74d5d92188c381babbc2d"}, {"endpoints": [{"url": "http://192.168.0.3:9696", "region": "RegionOne", "interface": "admin", "id": "654156fe52444dc98073543df9e71ffa"}, {"url": "http://172.16.0.2:9696", "region": "RegionOne", "interface": "public", "id": "957283b1f8374ac8a0edadc4c692fdfa"}, {"url": "http://192.168.0.3:9696", "region": "RegionOne", "interface": "internal", "id": "d49443933aeb47669cf183587db3192f"}], "type": "network", "id": "3de9b7a1297b4aa38db2dcce29a3a6c5"}, {"endpoints": [{"url": "http://192.168.0.3:8004/v1/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "internal", "id": "58e73ec335d44c158464e2ed9165d50e"}, {"url": "http://172.16.0.2:8004/v1/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "public", "id": "83d228e362e3495189028b9e6d11bfc0"}, {"url": "http://192.168.0.3:8004/v1/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "admin", "id": "bfa588db8b6c46aebc45de75f086aebf"}], "type": "orchestration", "id": "6b7ce42c03624140b968f1bd216096ab"}, {"endpoints": [{"url": "http://172.16.0.2:9292", "region": "RegionOne", "interface": "public", "id": "212e12b3538542edb83a1a85d0f17481"}, {"url": "http://192.168.0.3:9292", "region": "RegionOne", "interface": "admin", "id": "2c4be5ec811a456c92b10ebd6b6cedc4"}, {"url": "http://192.168.0.3:9292", "region": "RegionOne", "interface": "internal", "id": "e6b31148c2994d6ca78bafc039146a76"}], "type": "image", "id": "6f0d67860a104168b04bfa4daec41aa0"}, {"endpoints": [{"url": "http://192.168.0.3:8000/v1/", "region": "RegionOne", "interface": "internal", "id": "342ab018c92641c89189a298244f0b2e"}, {"url": "http://172.16.0.2:8000/v1/", "region": "RegionOne", "interface": "public", "id": "8dbd3b77e1b24deaa75b590e9128656a"}, {"url": "http://192.168.0.3:8000/v1/", "region": "RegionOne", "interface": "admin", "id": "e270dc50e5eb48b389b745cb5edb4a29"}], "type": "cloudformation", "id": "7e7d5eff27234942a99fc087f936fee5"}, {"endpoints": [{"url": "http://192.168.0.3:35357/v2.0", "region": "RegionOne", "interface": "admin", "id": "3dc50fcdee994d33af0efe11681224b4"}, {"url": "http://172.16.0.2:5000/v2.0", "region": "RegionOne", "interface": "public", "id": "705c8eb2c36b4bba9576ccf8ee09d990"}, {"url": "http://192.168.0.3:5000/v2.0", "region": "RegionOne", "interface": "internal", "id": "fedd2b427e9e49a88e829d65c29e4e63"}], "type": "identity", "id": "93fe3e7dbc2d49719d0758486797f30e"}, {"endpoints": [{"url": "http://192.168.0.3:8776/v1/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "admin", "id": "62998780462244ca99514274f3b472ca"}, {"url": "http://172.16.0.2:8776/v1/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "public", "id": "cbf578ef580b4f6db595b2d3590b45e3"}, {"url": "http://192.168.0.3:8776/v1/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "internal", "id": "d8b55d9c00034696a73a4e4ec5298cfe"}], "type": "volume", "id": "a90305c1c54a44a49dda4ff1bae72a3b"}, {"endpoints": [{"url": "http://172.16.0.2:8776/v2/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "public", "id": "0d6e2e28a3374abbb70591c5ddf5a15a"}, {"url": "http://192.168.0.3:8776/v2/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "admin", "id": "1d34e2dbac524350a98beba5c384bdb1"}, {"url": "http://192.168.0.3:8776/v2/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "internal", "id": "f32ea6614bfa4d2d89069a0bab6d5b7b"}], "type": "volumev2", "id": "ba20ddf8987542869438fb42f1fc5532"}, {"endpoints": [{"url": "http://192.168.0.3:8774/v2/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "internal", "id": "21d2ca87a564498886dd6be8652e3a01"}, {"url": "http://172.16.0.2:8774/v2/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "public", "id": "3bc80261b5e04c39b85dda64573fab53"}, {"url": "http://192.168.0.3:8774/v2/135b0cc9a8bf41b999b8046163ba0204", "region": "RegionOne", "interface": "admin", "id": "dd94634b1a0d49fa8327528e7b43d8d8"}], "type": "compute", "id": "fd874904421444e0abf485196afb5443"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "64913abf93fa474896f3fadf05fa22c7", "name": "tsufiev"}, "issued_at": "2014-09-12T16:03:47.285916Z"}}

[Alexander Makarov (amakarov)]

Bug appears just after restoring suspended VirtualBox VM.
Local time on the machine is in the past until synchronized with a host.
In that period Keystone returns cached tokens valid for it's time but expired for host.

Solution: synchronize VM time with host after restoring it.

[Alexander Makarov (amakarov)]

Quik & dirty way:
on host & guest system manually run
$ sudo ntpdate pool.ntp.org

[Timur Sufiev (tsufiev-x)]

I believe that making guest time-synced with host after resume is something that VirtualBox should already be able to do, perhaps with some proper settings... So, reassigning this one to the Fuel folks who are dealing with virtualbox scripts.

[Miroslav Anashkin (manashkin)]

To install VBox Guest Additions and related software from host to guest we need:

1. Verify , that all our local repos on master node contains all the VBox Guest Additions requirements.

2. Either use VirtualBox API directly, or with XPCOM or use pyvbox - it requires significant scripts modification. But as a result we will use guest machine console directly instead of using TCL Expect. I vote to use Python with venv as OS independent solution, but Java, Perl and C# may be discussed:-)

3. Use and automatically start vboxwebsrv to process API calls from our scripts. It also adds remote VBox management.

[Matthew Mosesohn (raytrac3r)]

Moved back to fuel-library. Still need to decide on a solution for this.

[Vladimir Kuklin (vkuklin)]

This bug is not of high priority. Restoring of environment from VirtualBox snapshot is valuable only for development purposes. If developer needs, he can manuall install all he needs and synchronize the time. Though it is possible to detect virtualbox and install all the packages, we need to consider adding of guest additions into our bundle carefully in order not to violate any licensing restrictions. Moving to 6.1 milestone.

[Stanislaw Bogatkin (sbogatkin)]

We need someone to verify this bug for current release due to significant changes into time sync process in previous one. Seems for me that this bug is invalid for 7.0

[Ilya Sviridov (isviridov)]

Tested with fuel-7.0-138-2015-08-05_08-38-26.iso

sudo ntpdate pool.ntp.org helps after VM resuming
keystone, horizon handles time changes correctly, no components restarts are needed

Syncing time accross fuel controlled nodes is a rule of thumb after waking up of VMs