Fuel for OpenStack

[system tests] Need to sync time after failovers

Bug #1420449 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Fuel QA Team
Fuel for OpenStack 6.1

Bug Description

In failover tests, where reboot / force of of vms take place we need to sync time on slaves as well.
http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.thread_5/28/consoleFull

a lot of tests failed here on OSTF step with message instance in Error state

At the same time nova-api.log says that no valid host. We need to sync time after revert/snapshot, b4ecause for now we can see that computes go to the past

[root@node-2 ~]# nova-manage service list
2015-02-09 20:18:56.673 14921 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.6/site-packages/nova/servicegroup/api.py:65
2015-02-09 20:18:57.476 14921 DEBUG oslo.db.sqlalchemy.session [req-c2bf6af6-b176-4de2-977f-69e47500e301 ] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/session.py:482
Binary Host Zone Status State Updated_At
2015-02-09 20:18:57.759 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:17. Elapsed time is -8899.240669 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-consoleauth node-2.test.domain.local internal enabled XXX 2015-02-10 06:47:17
2015-02-09 20:18:57.760 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:17. Elapsed time is -8899.239764 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-scheduler node-2.test.domain.local internal enabled XXX 2015-02-10 06:47:17
2015-02-09 20:18:57.761 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:17. Elapsed time is -8899.238533 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-conductor node-2.test.domain.local internal enabled XXX 2015-02-10 06:47:17
2015-02-09 20:18:57.762 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:17. Elapsed time is -8899.237415 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-cert node-2.test.domain.local internal enabled XXX 2015-02-10 06:47:17
2015-02-09 20:18:57.763 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:44. Elapsed time is -8926.236529 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-consoleauth node-4.test.domain.local internal enabled XXX 2015-02-10 06:47:44
2015-02-09 20:18:57.764 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:44. Elapsed time is -8926.235369 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-scheduler node-4.test.domain.local internal enabled XXX 2015-02-10 06:47:44
2015-02-09 20:18:57.765 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:44. Elapsed time is -8926.234289 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-conductor node-4.test.domain.local internal enabled XXX 2015-02-10 06:47:44
2015-02-09 20:18:57.767 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:48:11. Elapsed time is -8953.233118 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-consoleauth node-5.test.domain.local internal enabled XXX 2015-02-10 06:48:11
2015-02-09 20:18:57.768 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:48:11. Elapsed time is -8953.232065 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-scheduler node-5.test.domain.local internal enabled XXX 2015-02-10 06:48:11
2015-02-09 20:18:57.769 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:48:11. Elapsed time is -8953.231075 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-conductor node-5.test.domain.local internal enabled XXX 2015-02-10 06:48:11
2015-02-09 20:18:57.770 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:47:44. Elapsed time is -8926.230024 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-cert node-4.test.domain.local internal enabled XXX 2015-02-10 06:47:44
2015-02-09 20:18:57.771 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 06:48:11. Elapsed time is -8953.228972 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-cert node-5.test.domain.local internal enabled XXX 2015-02-10 06:48:11
2015-02-09 20:18:57.772 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 04:03:35. Elapsed time is 922.77215 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-compute node-3.test.domain.local nova enabled XXX 2015-02-10 04:03:35
2015-02-09 20:18:57.773 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 04:03:35. Elapsed time is 922.773551 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-network node-3.test.domain.local internal enabled XXX 2015-02-10 04:03:35
2015-02-09 20:18:57.774 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 04:03:50. Elapsed time is 907.774367 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-compute node-1.test.domain.local nova enabled XXX 2015-02-10 04:03:50
2015-02-09 20:18:57.775 14921 DEBUG nova.servicegroup.drivers.db [req-c2bf6af6-b176-4de2-977f-69e47500e301 None] Seems service is down. Last heartbeat was 2015-02-10 04:03:50. Elapsed time is 907.775351 is_up /usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py:75
nova-network node-1.test.domain.local internal enabled XXX 2015-02-10 04:03:50
[root@node-2 ~]# crm mon_1

Tags: system-tests
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

move to invalid, issue discussion moved to the https://bugs.launchpad.net/fuel/+bug/1421114

Changed in fuel:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.