OpenStack Compute (nova)

can not nova reboot vm

Bug #1478450 reported by hiyonger-ZTE_TECS on 2015-07-27

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Won't Fix	Low	Unassigned

Bug Description

I have 10 vms, run nova reboot the 10 vms. After reboot 285 times,can not nova reboot again.The detail information:

The nova-compute.log:

2015-07-22 14:51:43.808 15940 WARNING nova.openstack.common.deadlock_monitor [-] service function nova.servicegroup.drivers.db._report_state is deadlocked!
2015-07-22 14:51:43.984 15940 WARNING nova.openstack.common.deadlock_monitor [-] deadlock traceback is [' File "/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner\n self.f(*self.args, **self.kw)\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/deadlock_monitor.py", line 58, in __deco\n ret = func(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 116, in _report_state\n service.service_ref, state_catalog)\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 218, in service_update\n return self._manager.service_update(context, service, values)\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 330, in service_update\n service=service_p, values=values)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call\n wait_for_reply=True, timeout=timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send\n timeout=timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 400, in _send\n conn.topic_send(topic, msg, timeout=timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py", line 147, in __exit__\n self._done()\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py", line 136, in _done\n self.connection.reset()\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 656, in reset\n self.channel.close()\n', ' File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 163, in close\n (20, 41), # Channel.close_ok\n', ' File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait\n self.channel_id, allowed_methods)\n', ' File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 204, in _wait_method\n self.method_reader.read_method()\n', ' File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method\n self._next_method()\n', ' File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 112, in _next_method\n frame_type, channel, payload = read_frame()\n', ' File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 147, in read_frame\n frame_type, channel, size = unpack(\'>BHI\', read(7, True))\n', ' File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 245, in _read\n s = recv(131072)\n', ' File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 264, in recv\n timeout_exc=socket.timeout("timed out"))\n', ' File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 155, in trampoline\n return hub.switch()\n', ' File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch\n return self.greenlet.switch()\n']
2015-07-22 14:51:44.007 15940 WARNING nova.openstack.common.deadlock_monitor [-] nova compute exit!

But the openstack-nova-compute status is always active:
[root@slot13 drivers(keystone_admin)]# systemctl status openstack-nova-compute.service
openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled)
   Active: active (running) since Tue 2015-07-21 22:18:38 CST; 19h ago
Main PID: 15940 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           忖15940 /usr/bin/python /usr/bin/nova-compute

Jul 22 14:44:25 slot13 sudo[22639]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --tim...8af615-3b
Jul 22 14:44:25 slot13 sudo[22642]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link set qbr...1fa-53 up
Jul 22 14:44:25 slot13 sudo[22646]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --tim...48a15b-ae
Jul 22 14:44:25 slot13 sudo[22666]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --tim...2021fa-53

Tags:

Markus Zoeller (markus_z) (mzoeller) on 2015-07-28

tags:

added: reboot

Revision history for this message

Matt Riedemann (mriedem) wrote on 2015-07-29:

What release of nova? Are there errors in the nova-compute log?

Changed in nova:
status:	New → Incomplete

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2015-07-29:

I can't seem to find nova/openstack/common/deadlock_monitor.py anywhere....

Revision history for this message

hiyonger-ZTE_TECS (huanghuayong) wrote on 2015-07-31:

Download full text (58.2 KiB)

the I version,and nova is 2.17.0.
[root@slot13 qemu(keystone_admin)]# nova --version
2.17.0

but the nova/openstack/common/deadlock_monitor.pynova/openstack/common/deadlock_monitor.py is added by persional to check the nova-comupte service status. but the nova-compute proccess is not killed. if not killed, maybe our added code deadlock_monitor.py has bug. but the first bug come from the nova. the log is here.thks

the I version,and nova is 2.17.0.
[root@slot13 qemu(keystone_admin)]# nova --version
2.17.0

but the  nova/openstack/common/deadlock_monitor.pynova/openstack/common/deadlock_monitor.py is added by persional to check the nova-comupte service status. but the nova-compute proccess is not killed. if not killed, maybe our added code deadlock_monitor.py has bug. but the first bug come from the nova. the log is here.thks

2015-07-22 14:39:13.553 15940 AUDIT nova.compute.manager [req-5747a10e-74b6-4dc2-aaa5-429fbb6c3a9b 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Rebooting instance
2015-07-22 14:39:13.571 15940 AUDIT nova.compute.manager [req-0ab6aa09-1ead-427e-a251-a633fd083260 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Rebooting instance
2015-07-22 14:39:13.587 15940 AUDIT nova.compute.manager [req-2e757b4f-200e-41e7-b8ac-e5ca3e4b1b6f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Rebooting instance
2015-07-22 14:39:13.593 15940 AUDIT nova.compute.manager [req-30e78818-7801-4800-8726-c28b0f08276a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Rebooting instance
2015-07-22 14:39:13.812 15940 AUDIT nova.compute.manager [req-8172d0d4-c1a3-465e-8039-90f5a4dae6e8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Rebooting instance
2015-07-22 14:39:13.819 15940 AUDIT nova.compute.manager [req-baf4bc5a-8f92-455b-9027-7e88372b4eb8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Rebooting instance
2015-07-22 14:39:40.669 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:39:41.742 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334
2015-07-22 14:39:41.742 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85
2015-07-22 14:39:41.742 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6
2015-07-22 14:39:41.786 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13
2015-07-22 14:40:01.624 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.626 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.626 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.626 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.627 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:01.627 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:40:40.671 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:40:41.715 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334
2015-07-22 14:40:41.716 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85
2015-07-22 14:40:41.716 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6
2015-07-22 14:40:41.754 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13
2015-07-22 14:41:01.626 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.627 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.627 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.627 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.629 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:01.629 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:14.584 15940 WARNING nova.virt.libvirt.driver [req-a798452a-cf63-41b9-ae7d-220f2d7f56ad 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:14.586 15940 WARNING nova.virt.libvirt.driver [req-d4951655-2a79-412d-9957-d0709a185241 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:14.587 15940 WARNING nova.virt.libvirt.driver [req-d51fac32-3a38-4844-b6ce-65aebc819baf 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:14.796 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Stopped (Lifecycle Event)
2015-07-22 14:41:14.806 15940 WARNING nova.virt.libvirt.driver [req-25d80430-f5e2-461e-aaef-8bc28d527108 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:14.937 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:14.937 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Stopped (Lifecycle Event)
2015-07-22 14:41:15.014 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance destroyed successfully.
2015-07-22 14:41:15.019 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance destroyed successfully.
2015-07-22 14:41:15.024 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance destroyed successfully.
2015-07-22 14:41:15.083 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:15.084 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Stopped (Lifecycle Event)
2015-07-22 14:41:15.162 15940 INFO nova.virt.libvirt.driver [req-a798452a-cf63-41b9-ae7d-220f2d7f56ad 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:15.164 15940 INFO nova.virt.libvirt.driver [req-d51fac32-3a38-4844-b6ce-65aebc819baf 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:15.186 15940 INFO nova.virt.libvirt.driver [req-d4951655-2a79-412d-9957-d0709a185241 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:15.235 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:15.236 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Stopped (Lifecycle Event)
2015-07-22 14:41:15.247 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance destroyed successfully.
2015-07-22 14:41:15.358 15940 INFO nova.virt.libvirt.driver [req-25d80430-f5e2-461e-aaef-8bc28d527108 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:15.381 15940 WARNING nova.virt.libvirt.driver [req-2e757b4f-200e-41e7-b8ac-e5ca3e4b1b6f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:15.437 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:15.567 15940 WARNING nova.virt.libvirt.driver [req-0ab6aa09-1ead-427e-a251-a633fd083260 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:15.594 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Stopped (Lifecycle Event)
2015-07-22 14:41:15.613 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance destroyed successfully.
2015-07-22 14:41:15.805 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:15.826 15940 INFO nova.virt.libvirt.driver [req-2e757b4f-200e-41e7-b8ac-e5ca3e4b1b6f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:15.875 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Stopped (Lifecycle Event)
2015-07-22 14:41:15.930 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance destroyed successfully.
2015-07-22 14:41:16.011 15940 WARNING nova.virt.libvirt.driver [req-30e78818-7801-4800-8726-c28b0f08276a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:16.013 15940 WARNING nova.virt.libvirt.driver [req-8172d0d4-c1a3-465e-8039-90f5a4dae6e8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:16.024 15940 INFO nova.virt.libvirt.driver [req-0ab6aa09-1ead-427e-a251-a633fd083260 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:16.025 15940 WARNING nova.virt.libvirt.driver [req-5747a10e-74b6-4dc2-aaa5-429fbb6c3a9b 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:16.028 15940 WARNING nova.virt.libvirt.driver [req-baf4bc5a-8f92-455b-9027-7e88372b4eb8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:41:16.057 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:16.241 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Stopped (Lifecycle Event)
2015-07-22 14:41:16.261 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance destroyed successfully.
2015-07-22 14:41:16.359 15940 INFO nova.virt.libvirt.driver [req-30e78818-7801-4800-8726-c28b0f08276a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:16.413 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:16.414 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Stopped (Lifecycle Event)
2015-07-22 14:41:16.472 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance destroyed successfully.
2015-07-22 14:41:16.573 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:16.574 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Stopped (Lifecycle Event)
2015-07-22 14:41:16.584 15940 INFO nova.virt.libvirt.driver [req-8172d0d4-c1a3-465e-8039-90f5a4dae6e8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:16.685 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance destroyed successfully.
2015-07-22 14:41:16.698 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance destroyed successfully.
2015-07-22 14:41:16.743 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:16.743 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Stopped (Lifecycle Event)
2015-07-22 14:41:16.874 15940 INFO nova.virt.libvirt.driver [req-5747a10e-74b6-4dc2-aaa5-429fbb6c3a9b 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:16.948 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance rebooted successfully.
2015-07-22 14:41:16.954 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance rebooted successfully.
2015-07-22 14:41:16.984 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance rebooted successfully.
2015-07-22 14:41:16.988 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:16.998 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Resumed (Lifecycle Event)
2015-07-22 14:41:17.005 15940 INFO nova.virt.libvirt.driver [req-baf4bc5a-8f92-455b-9027-7e88372b4eb8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:41:17.142 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance rebooted successfully.
2015-07-22 14:41:17.256 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:41:17.257 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Started (Lifecycle Event)
2015-07-22 14:41:17.481 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Resumed (Lifecycle Event)
2015-07-22 14:41:17.491 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance rebooted successfully.
2015-07-22 14:41:17.651 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Started (Lifecycle Event)
2015-07-22 14:41:17.786 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance rebooted successfully.
2015-07-22 14:41:17.789 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance rebooted successfully.
2015-07-22 14:41:17.834 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Resumed (Lifecycle Event)
2015-07-22 14:41:18.099 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance rebooted successfully.
2015-07-22 14:41:18.151 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Started (Lifecycle Event)
2015-07-22 14:41:18.182 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance rebooted successfully.
2015-07-22 14:41:18.341 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance rebooted successfully.
2015-07-22 14:41:18.359 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Resumed (Lifecycle Event)
2015-07-22 14:41:18.515 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Started (Lifecycle Event)
2015-07-22 14:41:18.666 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Resumed (Lifecycle Event)
2015-07-22 14:41:18.889 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Started (Lifecycle Event)
2015-07-22 14:41:19.098 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Resumed (Lifecycle Event)
2015-07-22 14:41:19.270 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Started (Lifecycle Event)
2015-07-22 14:41:19.440 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Resumed (Lifecycle Event)
2015-07-22 14:41:19.620 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Started (Lifecycle Event)
2015-07-22 14:41:19.776 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Resumed (Lifecycle Event)
2015-07-22 14:41:19.941 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Started (Lifecycle Event)
2015-07-22 14:41:20.132 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Resumed (Lifecycle Event)
2015-07-22 14:41:20.337 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Started (Lifecycle Event)
2015-07-22 14:41:20.539 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Resumed (Lifecycle Event)
2015-07-22 14:41:20.740 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Started (Lifecycle Event)
2015-07-22 14:41:40.675 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:41:41.724 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334
2015-07-22 14:41:41.724 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85
2015-07-22 14:41:41.724 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6
2015-07-22 14:41:41.764 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13
2015-07-22 14:42:20.462 15940 AUDIT nova.compute.manager [req-daf138b2-9715-4efe-afbb-82c254e1e744 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Rebooting instance
2015-07-22 14:42:21.169 15940 AUDIT nova.compute.manager [req-6c1382a6-a826-4aef-9fed-e7b49c4e75a6 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Rebooting instance
2015-07-22 14:42:21.399 15940 AUDIT nova.compute.manager [req-ecf1d7a3-dd4d-492a-85ad-b859ca034e47 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Rebooting instance
2015-07-22 14:42:21.444 15940 AUDIT nova.compute.manager [req-40c01fce-999b-4b4c-aeb3-b9136ca15746 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Rebooting instance
2015-07-22 14:42:21.714 15940 AUDIT nova.compute.manager [req-5d5e08dd-7e76-4891-98a3-48e3cb04a0f5 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Rebooting instance
2015-07-22 14:42:21.762 15940 AUDIT nova.compute.manager [req-602a0d93-37b4-49b3-930a-7b86c5f9824f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Rebooting instance
2015-07-22 14:42:21.858 15940 AUDIT nova.compute.manager [req-48f878c6-65aa-4500-a01b-89dcfaf996d8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Rebooting instance
2015-07-22 14:42:22.064 15940 AUDIT nova.compute.manager [req-c3942a2b-b7e5-4e55-907f-0b436b0d925c 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Rebooting instance
2015-07-22 14:42:22.139 15940 AUDIT nova.compute.manager [req-c5cbf3a6-43db-45b1-b8a1-16c1368507cb 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Rebooting instance
2015-07-22 14:42:22.230 15940 AUDIT nova.compute.manager [req-9e774fca-88e8-45fb-8475-458a7535965f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Rebooting instance
2015-07-22 14:42:40.677 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:42:41.730 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334
2015-07-22 14:42:41.731 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85
2015-07-22 14:42:41.731 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6
2015-07-22 14:42:41.769 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13
2015-07-22 14:43:01.649 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.650 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.650 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.650 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.652 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:01.652 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:43:40.679 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:43:41.718 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334
2015-07-22 14:43:41.718 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85
2015-07-22 14:43:41.718 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6
2015-07-22 14:43:41.762 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13
2015-07-22 14:44:01.631 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.631 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.631 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.633 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.633 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:01.633 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:22.722 15940 WARNING nova.virt.libvirt.driver [req-daf138b2-9715-4efe-afbb-82c254e1e744 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:22.930 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Stopped (Lifecycle Event)
2015-07-22 14:44:22.935 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance destroyed successfully.
2015-07-22 14:44:23.013 15940 WARNING nova.virt.libvirt.driver [req-6c1382a6-a826-4aef-9fed-e7b49c4e75a6 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:23.046 15940 INFO nova.virt.libvirt.driver [req-daf138b2-9715-4efe-afbb-82c254e1e744 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:23.101 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:23.231 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Stopped (Lifecycle Event)
2015-07-22 14:44:23.245 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance destroyed successfully.
2015-07-22 14:44:23.349 15940 INFO nova.virt.libvirt.driver [req-6c1382a6-a826-4aef-9fed-e7b49c4e75a6 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:23.502 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:23.539 15940 WARNING nova.virt.libvirt.driver [req-40c01fce-999b-4b4c-aeb3-b9136ca15746 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:23.703 15940 WARNING nova.virt.libvirt.driver [req-602a0d93-37b4-49b3-930a-7b86c5f9824f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:23.705 15940 WARNING nova.virt.libvirt.driver [req-ecf1d7a3-dd4d-492a-85ad-b859ca034e47 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:23.706 15940 WARNING nova.virt.libvirt.driver [req-5d5e08dd-7e76-4891-98a3-48e3cb04a0f5 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:23.779 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Stopped (Lifecycle Event)
2015-07-22 14:44:23.933 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:24.000 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Stopped (Lifecycle Event)
2015-07-22 14:44:24.008 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance destroyed successfully.
2015-07-22 14:44:24.105 15940 INFO nova.virt.libvirt.driver [req-40c01fce-999b-4b4c-aeb3-b9136ca15746 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:24.157 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:24.157 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Stopped (Lifecycle Event)
2015-07-22 14:44:24.215 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance destroyed successfully.
2015-07-22 14:44:24.232 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance destroyed successfully.
2015-07-22 14:44:24.236 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance destroyed successfully.
2015-07-22 14:44:24.237 15940 WARNING nova.virt.libvirt.driver [req-c3942a2b-b7e5-4e55-907f-0b436b0d925c 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:24.242 15940 WARNING nova.virt.libvirt.driver [req-48f878c6-65aa-4500-a01b-89dcfaf996d8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:24.245 15940 WARNING nova.virt.libvirt.driver [req-9e774fca-88e8-45fb-8475-458a7535965f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:24.250 15940 WARNING nova.virt.libvirt.driver [req-c5cbf3a6-43db-45b1-b8a1-16c1368507cb 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:44:24.307 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:24.308 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Stopped (Lifecycle Event)
2015-07-22 14:44:24.403 15940 INFO nova.virt.libvirt.driver [req-ecf1d7a3-dd4d-492a-85ad-b859ca034e47 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:24.428 15940 INFO nova.virt.libvirt.driver [req-5d5e08dd-7e76-4891-98a3-48e3cb04a0f5 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:24.438 15940 INFO nova.virt.libvirt.driver [req-602a0d93-37b4-49b3-930a-7b86c5f9824f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:24.483 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:24.484 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Stopped (Lifecycle Event)
2015-07-22 14:44:24.661 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:24.661 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Stopped (Lifecycle Event)
2015-07-22 14:44:24.805 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:24.806 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Stopped (Lifecycle Event)
2015-07-22 14:44:24.870 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance destroyed successfully.
2015-07-22 14:44:24.891 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance destroyed successfully.
2015-07-22 14:44:24.963 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance destroyed successfully.
2015-07-22 14:44:25.014 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:25.015 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Stopped (Lifecycle Event)
2015-07-22 14:44:25.025 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance destroyed successfully.
2015-07-22 14:44:25.028 15940 INFO nova.virt.libvirt.driver [req-c3942a2b-b7e5-4e55-907f-0b436b0d925c 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:25.071 15940 INFO nova.virt.libvirt.driver [req-9e774fca-88e8-45fb-8475-458a7535965f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:25.115 15940 INFO nova.virt.libvirt.driver [req-48f878c6-65aa-4500-a01b-89dcfaf996d8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:25.184 15940 INFO nova.virt.libvirt.driver [req-c5cbf3a6-43db-45b1-b8a1-16c1368507cb 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 
2015-07-22 14:44:25.216 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance rebooted successfully.
2015-07-22 14:44:25.222 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance rebooted successfully.
2015-07-22 14:44:25.235 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:25.236 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Resumed (Lifecycle Event)
2015-07-22 14:44:25.452 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:25.453 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Started (Lifecycle Event)
2015-07-22 14:44:25.710 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Resumed (Lifecycle Event)
2015-07-22 14:44:25.900 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Started (Lifecycle Event)
2015-07-22 14:44:26.064 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Resumed (Lifecycle Event)
2015-07-22 14:44:26.219 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:44:26.220 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Started (Lifecycle Event)
2015-07-22 14:44:26.295 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance rebooted successfully.
2015-07-22 14:44:26.309 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance rebooted successfully.
2015-07-22 14:44:26.330 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance rebooted successfully.
2015-07-22 14:44:26.346 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance rebooted successfully.
2015-07-22 14:44:26.546 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Resumed (Lifecycle Event)
2015-07-22 14:44:26.597 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance rebooted successfully.
2015-07-22 14:44:26.808 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance rebooted successfully.
2015-07-22 14:44:26.851 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance rebooted successfully.
2015-07-22 14:44:26.880 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Started (Lifecycle Event)
2015-07-22 14:44:27.076 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Resumed (Lifecycle Event)
2015-07-22 14:44:27.198 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance rebooted successfully.
2015-07-22 14:44:27.246 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Started (Lifecycle Event)
2015-07-22 14:44:27.397 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Resumed (Lifecycle Event)
2015-07-22 14:44:27.569 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Started (Lifecycle Event)
2015-07-22 14:44:27.771 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Resumed (Lifecycle Event)
2015-07-22 14:44:27.922 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Started (Lifecycle Event)
2015-07-22 14:44:28.077 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Resumed (Lifecycle Event)
2015-07-22 14:44:28.269 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Started (Lifecycle Event)
2015-07-22 14:44:28.546 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Resumed (Lifecycle Event)
2015-07-22 14:44:28.701 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Started (Lifecycle Event)
2015-07-22 14:44:28.862 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Resumed (Lifecycle Event)
2015-07-22 14:44:29.033 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Started (Lifecycle Event)
2015-07-22 14:44:40.680 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:44:41.732 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334
2015-07-22 14:44:41.733 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85
2015-07-22 14:44:41.733 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6
2015-07-22 14:44:41.772 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13
2015-07-22 14:45:28.435 15940 AUDIT nova.compute.manager [req-ad9a7e60-8dce-45ee-b572-eddaa7837680 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Rebooting instance
2015-07-22 14:45:28.531 15940 AUDIT nova.compute.manager [req-cc39a956-c951-4c50-8d62-c8a9ef680d41 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Rebooting instance
2015-07-22 14:45:29.227 15940 AUDIT nova.compute.manager [req-50f891f1-eccf-45ff-9283-dccc85f0d87f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Rebooting instance
2015-07-22 14:45:29.414 15940 AUDIT nova.compute.manager [req-e8299f7f-0e3e-40ab-ba59-71a615eb8a7a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Rebooting instance
2015-07-22 14:45:29.454 15940 AUDIT nova.compute.manager [req-f122ba36-fe74-4966-b593-c1f20f8ed1ed 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Rebooting instance
2015-07-22 14:45:29.663 15940 AUDIT nova.compute.manager [req-785f9111-ffa4-45c2-9904-c32459c18e40 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Rebooting instance
2015-07-22 14:45:29.777 15940 AUDIT nova.compute.manager [req-6e4f3e1a-4d16-49c8-949b-4c8dbf78d2da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Rebooting instance
2015-07-22 14:45:29.934 15940 AUDIT nova.compute.manager [req-ae8a97ee-f9cd-41bf-9e6c-1c7020583039 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Rebooting instance
2015-07-22 14:45:29.959 15940 AUDIT nova.compute.manager [req-fc60e213-fe51-475c-9546-4e3a14344591 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Rebooting instance
2015-07-22 14:45:30.112 15940 AUDIT nova.compute.manager [req-0494b0fe-f0c4-4b33-b596-f919429932da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Rebooting instance
2015-07-22 14:45:40.682 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:45:41.810 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334
2015-07-22 14:45:41.811 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85
2015-07-22 14:45:41.811 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6
2015-07-22 14:45:41.854 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13
2015-07-22 14:46:01.643 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.643 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.645 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.645 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:01.645 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip.
2015-07-22 14:46:40.684 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-07-22 14:47:30.484 15940 WARNING nova.virt.libvirt.driver [req-cc39a956-c951-4c50-8d62-c8a9ef680d41 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:30.695 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Stopped (Lifecycle Event)
2015-07-22 14:47:30.701 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance destroyed successfully.
2015-07-22 14:47:30.928 15940 WARNING nova.virt.libvirt.driver [req-ad9a7e60-8dce-45ee-b572-eddaa7837680 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:30.963 15940 WARNING nova.virt.libvirt.driver [req-50f891f1-eccf-45ff-9283-dccc85f0d87f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:31.125 15940 WARNING nova.virt.libvirt.driver [req-f122ba36-fe74-4966-b593-c1f20f8ed1ed 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:31.348 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance destroyed successfully.
2015-07-22 14:47:31.558 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance destroyed successfully.
2015-07-22 14:47:31.562 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance destroyed successfully.
2015-07-22 14:47:31.699 15940 WARNING nova.virt.libvirt.driver [req-e8299f7f-0e3e-40ab-ba59-71a615eb8a7a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:31.895 15940 WARNING nova.virt.libvirt.driver [req-785f9111-ffa4-45c2-9904-c32459c18e40 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:31.911 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance destroyed successfully.
2015-07-22 14:47:32.118 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance destroyed successfully.
2015-07-22 14:47:32.138 15940 WARNING nova.virt.libvirt.driver [req-6e4f3e1a-4d16-49c8-949b-4c8dbf78d2da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:32.350 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance destroyed successfully.
2015-07-22 14:47:32.427 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672
2015-07-22 14:47:32.556 15940 WARNING nova.virt.libvirt.driver [req-ae8a97ee-f9cd-41bf-9e6c-1c7020583039 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:32.561 15940 WARNING nova.virt.libvirt.driver [req-fc60e213-fe51-475c-9546-4e3a14344591 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:32.702 15940 WARNING nova.virt.libvirt.driver [req-0494b0fe-f0c4-4b33-b596-f919429932da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Failed to soft reboot instance. Trying hard reboot.
2015-07-22 14:47:33.179 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance destroyed successfully.
2015-07-22 14:47:33.184 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance destroyed successfully.
2015-07-22 14:47:33.187 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance destroyed successfully.
2015-07-22 14:47:33.320 15940 WARNING oslo.messaging._drivers.amqpdriver [-] Number of call queues is greater than warning threshold: 10. There could be a leak.
2015-07-22 14:47:33.369 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672
2015-07-22 14:47:33.369 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672
2015-07-22 14:47:33.370 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672
2015-07-22 14:51:43.808 15940 WARNING nova.openstack.common.deadlock_monitor [-] service function nova.servicegroup.drivers.db._report_state is deadlocked!
2015-07-22 14:51:43.984 15940 WARNING nova.openstack.common.deadlock_monitor [-] deadlock traceback is ['  File "/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner\n    self.f(*self.args, **self.kw)\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/deadlock_monitor.py", line 58, in __deco\n    ret = func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 116, in _report_state\n    service.service_ref, state_catalog)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 218, in service_update\n    return self._manager.service_update(context, service, values)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 330, in service_update\n    service=service_p, values=values)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call\n    wait_for_reply=True, timeout=timeout)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send\n    timeout=timeout)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send\n    return self._send(target, ctxt, message, wait_for_reply, timeout)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 400, in _send\n    conn.topic_send(topic, msg, timeout=timeout)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py", line 147, in __exit__\n    self._done()\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py", line 136, in _done\n    self.connection.reset()\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 656, in reset\n    self.channel.close()\n', '  File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 163, in close\n    (20, 41),  # Channel.close_ok\n', '  File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait\n    self.channel_id, allowed_methods)\n', '  File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 204, in _wait_method\n    self.method_reader.read_method()\n', '  File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method\n    self._next_method()\n', '  File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 112, in _next_method\n    frame_type, channel, payload = read_frame()\n', '  File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 147, in read_frame\n    frame_type, channel, size = unpack(\'>BHI\', read(7, True))\n', '  File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 245, in _read\n    s = recv(131072)\n', '  File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 264, in recv\n    timeout_exc=socket.timeout("timed out"))\n', '  File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 155, in trampoline\n    return hub.switch()\n', '  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch\n    return self.greenlet.switch()\n']
2015-07-22 14:51:44.007 15940 WARNING nova.openstack.common.deadlock_monitor [-] nova compute exit!
2015-07-22 23:22:08.095 28664 INFO nova.openstack.common.periodic_task [-] Skipping periodic task update_host_ssh_pubkey because its interval is negative
2015-07-22 23:22:08.124 28664 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-07-22 23:22:08.196 28664 INFO nova.virt.driver [-] Loading compute driver 'nova.virt.libvirt.LibvirtDriver'
2015-07-22 23:22:08.223 28664 INFO nova.vcpu_affinity.policy [-] no vcpupin policy is configed
2015-07-22 23:22:08.242 28664 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672
2015-07-22 23:22:08.257 28664 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672
2015-07-23 21:27:11.687 7924 INFO nova.openstack.common.periodic_task [-] Skipping periodic task update_host_ssh_pubkey because its interval is negative
2015-07-23 21:27:11.814 7924 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-07-23 21:27:11.884 7924 INFO nova.virt.driver [-] Loading compute driver 'nova.virt.libvirt.LibvirtDriver'
2015-07-23 21:27:12.051 7924 INFO nova.vcpu_affinity.policy [-] no vcpupin policy is configed
2015-07-23 21:27:12.067 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-07-23 21:27:13.069 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:27:13.069 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-07-23 21:27:14.078 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2015-07-23 21:27:17.079 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:27:17.079 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-07-23 21:27:18.088 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2015-07-23 21:27:23.092 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:27:23.092 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-07-23 21:27:24.101 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
2015-07-23 21:27:31.104 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:27:31.104 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-07-23 21:27:32.113 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
2015-07-23 21:27:41.117 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:27:41.118 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-07-23 21:27:42.127 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 2015-07-23 21:43:36.141 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:43:36.142 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-07-23 21:43:37.154 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
2015-07-23 21:44:07.165 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:44:07.165 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-07-23 21:44:08.174 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
2015-07-23 21:44:38.182 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672
2015-07-23 21:44:38.182 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...

def deadlock_check():          
    while True:
        LOG.debug(_("service deadlock_check!, time is %f"), timeutils.monotonic_time())
        now = timeutils.monotonic_time()
        for func in deadlock_dict.keys():
            if deadlock_dict[func] < now:
                LOG.warn(_("service function %s is deadlocked!"),func)
                _print_deadlock_greenthreads(func)
                LOG.warn(_("nova compute exit!"))
                cmd = "ps -e|grep nova-compute | awk '{print $1}'|xargs kill -9"
                os.system(cmd)
                sys.exit(0)
        greenthread.sleep(10)

hiyonger-ZTE_TECS (huanghuayong) on 2015-08-02

Changed in nova:
status:	Incomplete → New

Revision history for this message

Sean Dague (sdague) wrote on 2015-09-23:

So it looks like if you tight loop reboot all of your instances continously, we're hitting an update lock issue. That's not great, and investigating it further would be good, but it's also a low likelihood issue in the field, so marking as Low.

Changed in nova:
status:	New → Confirmed
importance:	Undecided → Low

Revision history for this message

Sean Dague (sdague) wrote on 2016-02-20:

Closing this out as Invalid as it's a really low likelyhood event, and requires just stressing the whole environment to the point of breakdown. We know that if you do that stuff will shake out. Exactly why isn't hugely clear here. Without a better tight reproduce or problem determination this bug just isn't actionable.

Changed in nova:
status:	Confirmed → Won't Fix

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.