the I version,and nova is 2.17.0. [root@slot13 qemu(keystone_admin)]# nova --version 2.17.0 but the nova/openstack/common/deadlock_monitor.pynova/openstack/common/deadlock_monitor.py is added by persional to check the nova-comupte service status. but the nova-compute proccess is not killed. if not killed, maybe our added code deadlock_monitor.py has bug. but the first bug come from the nova. the log is here.thks 2015-07-22 14:39:13.553 15940 AUDIT nova.compute.manager [req-5747a10e-74b6-4dc2-aaa5-429fbb6c3a9b 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Rebooting instance 2015-07-22 14:39:13.571 15940 AUDIT nova.compute.manager [req-0ab6aa09-1ead-427e-a251-a633fd083260 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Rebooting instance 2015-07-22 14:39:13.587 15940 AUDIT nova.compute.manager [req-2e757b4f-200e-41e7-b8ac-e5ca3e4b1b6f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Rebooting instance 2015-07-22 14:39:13.593 15940 AUDIT nova.compute.manager [req-30e78818-7801-4800-8726-c28b0f08276a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Rebooting instance 2015-07-22 14:39:13.812 15940 AUDIT nova.compute.manager [req-8172d0d4-c1a3-465e-8039-90f5a4dae6e8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Rebooting instance 2015-07-22 14:39:13.819 15940 AUDIT nova.compute.manager [req-baf4bc5a-8f92-455b-9027-7e88372b4eb8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Rebooting instance 2015-07-22 14:39:40.669 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:39:41.742 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334 2015-07-22 14:39:41.742 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85 2015-07-22 14:39:41.742 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6 2015-07-22 14:39:41.786 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13 2015-07-22 14:40:01.624 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.625 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.626 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.626 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.626 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.627 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:01.627 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:40:40.671 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:40:41.715 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334 2015-07-22 14:40:41.716 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85 2015-07-22 14:40:41.716 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6 2015-07-22 14:40:41.754 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13 2015-07-22 14:41:01.626 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.627 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.627 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.627 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.628 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.629 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:01.629 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:14.584 15940 WARNING nova.virt.libvirt.driver [req-a798452a-cf63-41b9-ae7d-220f2d7f56ad 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:14.586 15940 WARNING nova.virt.libvirt.driver [req-d4951655-2a79-412d-9957-d0709a185241 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:14.587 15940 WARNING nova.virt.libvirt.driver [req-d51fac32-3a38-4844-b6ce-65aebc819baf 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:14.796 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Stopped (Lifecycle Event) 2015-07-22 14:41:14.806 15940 WARNING nova.virt.libvirt.driver [req-25d80430-f5e2-461e-aaef-8bc28d527108 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:14.937 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:14.937 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Stopped (Lifecycle Event) 2015-07-22 14:41:15.014 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance destroyed successfully. 2015-07-22 14:41:15.019 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance destroyed successfully. 2015-07-22 14:41:15.024 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance destroyed successfully. 2015-07-22 14:41:15.083 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:15.084 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Stopped (Lifecycle Event) 2015-07-22 14:41:15.162 15940 INFO nova.virt.libvirt.driver [req-a798452a-cf63-41b9-ae7d-220f2d7f56ad 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:15.164 15940 INFO nova.virt.libvirt.driver [req-d51fac32-3a38-4844-b6ce-65aebc819baf 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:15.186 15940 INFO nova.virt.libvirt.driver [req-d4951655-2a79-412d-9957-d0709a185241 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:15.235 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:15.236 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Stopped (Lifecycle Event) 2015-07-22 14:41:15.247 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance destroyed successfully. 2015-07-22 14:41:15.358 15940 INFO nova.virt.libvirt.driver [req-25d80430-f5e2-461e-aaef-8bc28d527108 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:15.381 15940 WARNING nova.virt.libvirt.driver [req-2e757b4f-200e-41e7-b8ac-e5ca3e4b1b6f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:15.437 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:15.567 15940 WARNING nova.virt.libvirt.driver [req-0ab6aa09-1ead-427e-a251-a633fd083260 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:15.594 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Stopped (Lifecycle Event) 2015-07-22 14:41:15.613 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance destroyed successfully. 2015-07-22 14:41:15.805 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:15.826 15940 INFO nova.virt.libvirt.driver [req-2e757b4f-200e-41e7-b8ac-e5ca3e4b1b6f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:15.875 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Stopped (Lifecycle Event) 2015-07-22 14:41:15.930 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance destroyed successfully. 2015-07-22 14:41:16.011 15940 WARNING nova.virt.libvirt.driver [req-30e78818-7801-4800-8726-c28b0f08276a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:16.013 15940 WARNING nova.virt.libvirt.driver [req-8172d0d4-c1a3-465e-8039-90f5a4dae6e8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:16.024 15940 INFO nova.virt.libvirt.driver [req-0ab6aa09-1ead-427e-a251-a633fd083260 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:16.025 15940 WARNING nova.virt.libvirt.driver [req-5747a10e-74b6-4dc2-aaa5-429fbb6c3a9b 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:16.028 15940 WARNING nova.virt.libvirt.driver [req-baf4bc5a-8f92-455b-9027-7e88372b4eb8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:41:16.057 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:16.241 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Stopped (Lifecycle Event) 2015-07-22 14:41:16.261 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance destroyed successfully. 2015-07-22 14:41:16.359 15940 INFO nova.virt.libvirt.driver [req-30e78818-7801-4800-8726-c28b0f08276a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:16.413 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:16.414 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Stopped (Lifecycle Event) 2015-07-22 14:41:16.472 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance destroyed successfully. 2015-07-22 14:41:16.573 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:16.574 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Stopped (Lifecycle Event) 2015-07-22 14:41:16.584 15940 INFO nova.virt.libvirt.driver [req-8172d0d4-c1a3-465e-8039-90f5a4dae6e8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:16.685 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance destroyed successfully. 2015-07-22 14:41:16.698 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance destroyed successfully. 2015-07-22 14:41:16.743 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:16.743 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Stopped (Lifecycle Event) 2015-07-22 14:41:16.874 15940 INFO nova.virt.libvirt.driver [req-5747a10e-74b6-4dc2-aaa5-429fbb6c3a9b 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:16.948 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance rebooted successfully. 2015-07-22 14:41:16.954 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance rebooted successfully. 2015-07-22 14:41:16.984 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance rebooted successfully. 2015-07-22 14:41:16.988 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:16.998 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Resumed (Lifecycle Event) 2015-07-22 14:41:17.005 15940 INFO nova.virt.libvirt.driver [req-baf4bc5a-8f92-455b-9027-7e88372b4eb8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:41:17.142 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance rebooted successfully. 2015-07-22 14:41:17.256 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:41:17.257 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Started (Lifecycle Event) 2015-07-22 14:41:17.481 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Resumed (Lifecycle Event) 2015-07-22 14:41:17.491 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance rebooted successfully. 2015-07-22 14:41:17.651 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Started (Lifecycle Event) 2015-07-22 14:41:17.786 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance rebooted successfully. 2015-07-22 14:41:17.789 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance rebooted successfully. 2015-07-22 14:41:17.834 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Resumed (Lifecycle Event) 2015-07-22 14:41:18.099 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance rebooted successfully. 2015-07-22 14:41:18.151 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Started (Lifecycle Event) 2015-07-22 14:41:18.182 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance rebooted successfully. 2015-07-22 14:41:18.341 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance rebooted successfully. 2015-07-22 14:41:18.359 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Resumed (Lifecycle Event) 2015-07-22 14:41:18.515 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Started (Lifecycle Event) 2015-07-22 14:41:18.666 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Resumed (Lifecycle Event) 2015-07-22 14:41:18.889 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Started (Lifecycle Event) 2015-07-22 14:41:19.098 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Resumed (Lifecycle Event) 2015-07-22 14:41:19.270 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Started (Lifecycle Event) 2015-07-22 14:41:19.440 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Resumed (Lifecycle Event) 2015-07-22 14:41:19.620 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Started (Lifecycle Event) 2015-07-22 14:41:19.776 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Resumed (Lifecycle Event) 2015-07-22 14:41:19.941 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Started (Lifecycle Event) 2015-07-22 14:41:20.132 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Resumed (Lifecycle Event) 2015-07-22 14:41:20.337 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Started (Lifecycle Event) 2015-07-22 14:41:20.539 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Resumed (Lifecycle Event) 2015-07-22 14:41:20.740 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Started (Lifecycle Event) 2015-07-22 14:41:40.675 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:41:41.724 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334 2015-07-22 14:41:41.724 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85 2015-07-22 14:41:41.724 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6 2015-07-22 14:41:41.764 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13 2015-07-22 14:42:20.462 15940 AUDIT nova.compute.manager [req-daf138b2-9715-4efe-afbb-82c254e1e744 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Rebooting instance 2015-07-22 14:42:21.169 15940 AUDIT nova.compute.manager [req-6c1382a6-a826-4aef-9fed-e7b49c4e75a6 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Rebooting instance 2015-07-22 14:42:21.399 15940 AUDIT nova.compute.manager [req-ecf1d7a3-dd4d-492a-85ad-b859ca034e47 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Rebooting instance 2015-07-22 14:42:21.444 15940 AUDIT nova.compute.manager [req-40c01fce-999b-4b4c-aeb3-b9136ca15746 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Rebooting instance 2015-07-22 14:42:21.714 15940 AUDIT nova.compute.manager [req-5d5e08dd-7e76-4891-98a3-48e3cb04a0f5 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Rebooting instance 2015-07-22 14:42:21.762 15940 AUDIT nova.compute.manager [req-602a0d93-37b4-49b3-930a-7b86c5f9824f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Rebooting instance 2015-07-22 14:42:21.858 15940 AUDIT nova.compute.manager [req-48f878c6-65aa-4500-a01b-89dcfaf996d8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Rebooting instance 2015-07-22 14:42:22.064 15940 AUDIT nova.compute.manager [req-c3942a2b-b7e5-4e55-907f-0b436b0d925c 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Rebooting instance 2015-07-22 14:42:22.139 15940 AUDIT nova.compute.manager [req-c5cbf3a6-43db-45b1-b8a1-16c1368507cb 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Rebooting instance 2015-07-22 14:42:22.230 15940 AUDIT nova.compute.manager [req-9e774fca-88e8-45fb-8475-458a7535965f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Rebooting instance 2015-07-22 14:42:40.677 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:42:41.730 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334 2015-07-22 14:42:41.731 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85 2015-07-22 14:42:41.731 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6 2015-07-22 14:42:41.769 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13 2015-07-22 14:43:01.649 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.650 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.650 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.650 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.651 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.652 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:01.652 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:43:40.679 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:43:41.718 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334 2015-07-22 14:43:41.718 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85 2015-07-22 14:43:41.718 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6 2015-07-22 14:43:41.762 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13 2015-07-22 14:44:01.631 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.631 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.631 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.632 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.633 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.633 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:01.633 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:22.722 15940 WARNING nova.virt.libvirt.driver [req-daf138b2-9715-4efe-afbb-82c254e1e744 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:22.930 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Stopped (Lifecycle Event) 2015-07-22 14:44:22.935 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance destroyed successfully. 2015-07-22 14:44:23.013 15940 WARNING nova.virt.libvirt.driver [req-6c1382a6-a826-4aef-9fed-e7b49c4e75a6 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:23.046 15940 INFO nova.virt.libvirt.driver [req-daf138b2-9715-4efe-afbb-82c254e1e744 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:23.101 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:23.231 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Stopped (Lifecycle Event) 2015-07-22 14:44:23.245 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance destroyed successfully. 2015-07-22 14:44:23.349 15940 INFO nova.virt.libvirt.driver [req-6c1382a6-a826-4aef-9fed-e7b49c4e75a6 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:23.502 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:23.539 15940 WARNING nova.virt.libvirt.driver [req-40c01fce-999b-4b4c-aeb3-b9136ca15746 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:23.703 15940 WARNING nova.virt.libvirt.driver [req-602a0d93-37b4-49b3-930a-7b86c5f9824f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:23.705 15940 WARNING nova.virt.libvirt.driver [req-ecf1d7a3-dd4d-492a-85ad-b859ca034e47 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:23.706 15940 WARNING nova.virt.libvirt.driver [req-5d5e08dd-7e76-4891-98a3-48e3cb04a0f5 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:23.779 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Stopped (Lifecycle Event) 2015-07-22 14:44:23.933 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:24.000 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Stopped (Lifecycle Event) 2015-07-22 14:44:24.008 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance destroyed successfully. 2015-07-22 14:44:24.105 15940 INFO nova.virt.libvirt.driver [req-40c01fce-999b-4b4c-aeb3-b9136ca15746 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:24.157 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:24.157 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Stopped (Lifecycle Event) 2015-07-22 14:44:24.215 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance destroyed successfully. 2015-07-22 14:44:24.232 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance destroyed successfully. 2015-07-22 14:44:24.236 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance destroyed successfully. 2015-07-22 14:44:24.237 15940 WARNING nova.virt.libvirt.driver [req-c3942a2b-b7e5-4e55-907f-0b436b0d925c 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:24.242 15940 WARNING nova.virt.libvirt.driver [req-48f878c6-65aa-4500-a01b-89dcfaf996d8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:24.245 15940 WARNING nova.virt.libvirt.driver [req-9e774fca-88e8-45fb-8475-458a7535965f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:24.250 15940 WARNING nova.virt.libvirt.driver [req-c5cbf3a6-43db-45b1-b8a1-16c1368507cb 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:44:24.307 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:24.308 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Stopped (Lifecycle Event) 2015-07-22 14:44:24.403 15940 INFO nova.virt.libvirt.driver [req-ecf1d7a3-dd4d-492a-85ad-b859ca034e47 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:24.428 15940 INFO nova.virt.libvirt.driver [req-5d5e08dd-7e76-4891-98a3-48e3cb04a0f5 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:24.438 15940 INFO nova.virt.libvirt.driver [req-602a0d93-37b4-49b3-930a-7b86c5f9824f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:24.483 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:24.484 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Stopped (Lifecycle Event) 2015-07-22 14:44:24.661 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:24.661 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Stopped (Lifecycle Event) 2015-07-22 14:44:24.805 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:24.806 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Stopped (Lifecycle Event) 2015-07-22 14:44:24.870 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance destroyed successfully. 2015-07-22 14:44:24.891 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance destroyed successfully. 2015-07-22 14:44:24.963 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance destroyed successfully. 2015-07-22 14:44:25.014 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:25.015 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Stopped (Lifecycle Event) 2015-07-22 14:44:25.025 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance destroyed successfully. 2015-07-22 14:44:25.028 15940 INFO nova.virt.libvirt.driver [req-c3942a2b-b7e5-4e55-907f-0b436b0d925c 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:25.071 15940 INFO nova.virt.libvirt.driver [req-9e774fca-88e8-45fb-8475-458a7535965f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:25.115 15940 INFO nova.virt.libvirt.driver [req-48f878c6-65aa-4500-a01b-89dcfaf996d8 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:25.184 15940 INFO nova.virt.libvirt.driver [req-c5cbf3a6-43db-45b1-b8a1-16c1368507cb 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] get numainfo (None, None) 2015-07-22 14:44:25.216 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance rebooted successfully. 2015-07-22 14:44:25.222 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance rebooted successfully. 2015-07-22 14:44:25.235 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:25.236 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Resumed (Lifecycle Event) 2015-07-22 14:44:25.452 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:25.453 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] VM Started (Lifecycle Event) 2015-07-22 14:44:25.710 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Resumed (Lifecycle Event) 2015-07-22 14:44:25.900 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] VM Started (Lifecycle Event) 2015-07-22 14:44:26.064 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Resumed (Lifecycle Event) 2015-07-22 14:44:26.219 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:44:26.220 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] VM Started (Lifecycle Event) 2015-07-22 14:44:26.295 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance rebooted successfully. 2015-07-22 14:44:26.309 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance rebooted successfully. 2015-07-22 14:44:26.330 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance rebooted successfully. 2015-07-22 14:44:26.346 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance rebooted successfully. 2015-07-22 14:44:26.546 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Resumed (Lifecycle Event) 2015-07-22 14:44:26.597 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance rebooted successfully. 2015-07-22 14:44:26.808 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance rebooted successfully. 2015-07-22 14:44:26.851 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance rebooted successfully. 2015-07-22 14:44:26.880 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] VM Started (Lifecycle Event) 2015-07-22 14:44:27.076 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Resumed (Lifecycle Event) 2015-07-22 14:44:27.198 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance rebooted successfully. 2015-07-22 14:44:27.246 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] VM Started (Lifecycle Event) 2015-07-22 14:44:27.397 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Resumed (Lifecycle Event) 2015-07-22 14:44:27.569 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Started (Lifecycle Event) 2015-07-22 14:44:27.771 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Resumed (Lifecycle Event) 2015-07-22 14:44:27.922 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] VM Started (Lifecycle Event) 2015-07-22 14:44:28.077 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Resumed (Lifecycle Event) 2015-07-22 14:44:28.269 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] VM Started (Lifecycle Event) 2015-07-22 14:44:28.546 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Resumed (Lifecycle Event) 2015-07-22 14:44:28.701 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] VM Started (Lifecycle Event) 2015-07-22 14:44:28.862 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Resumed (Lifecycle Event) 2015-07-22 14:44:29.033 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] VM Started (Lifecycle Event) 2015-07-22 14:44:40.680 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:44:41.732 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334 2015-07-22 14:44:41.733 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85 2015-07-22 14:44:41.733 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6 2015-07-22 14:44:41.772 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13 2015-07-22 14:45:28.435 15940 AUDIT nova.compute.manager [req-ad9a7e60-8dce-45ee-b572-eddaa7837680 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Rebooting instance 2015-07-22 14:45:28.531 15940 AUDIT nova.compute.manager [req-cc39a956-c951-4c50-8d62-c8a9ef680d41 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Rebooting instance 2015-07-22 14:45:29.227 15940 AUDIT nova.compute.manager [req-50f891f1-eccf-45ff-9283-dccc85f0d87f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Rebooting instance 2015-07-22 14:45:29.414 15940 AUDIT nova.compute.manager [req-e8299f7f-0e3e-40ab-ba59-71a615eb8a7a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Rebooting instance 2015-07-22 14:45:29.454 15940 AUDIT nova.compute.manager [req-f122ba36-fe74-4966-b593-c1f20f8ed1ed 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Rebooting instance 2015-07-22 14:45:29.663 15940 AUDIT nova.compute.manager [req-785f9111-ffa4-45c2-9904-c32459c18e40 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Rebooting instance 2015-07-22 14:45:29.777 15940 AUDIT nova.compute.manager [req-6e4f3e1a-4d16-49c8-949b-4c8dbf78d2da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Rebooting instance 2015-07-22 14:45:29.934 15940 AUDIT nova.compute.manager [req-ae8a97ee-f9cd-41bf-9e6c-1c7020583039 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Rebooting instance 2015-07-22 14:45:29.959 15940 AUDIT nova.compute.manager [req-fc60e213-fe51-475c-9546-4e3a14344591 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Rebooting instance 2015-07-22 14:45:30.112 15940 AUDIT nova.compute.manager [req-0494b0fe-f0c4-4b33-b596-f919429932da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Rebooting instance 2015-07-22 14:45:40.682 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:45:41.810 15940 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 18334 2015-07-22 14:45:41.811 15940 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 85 2015-07-22 14:45:41.811 15940 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 6 2015-07-22 14:45:41.854 15940 INFO nova.compute.resource_tracker [-] Compute_service record updated for slot13:slot13 2015-07-22 14:46:01.643 15940 INFO nova.compute.manager [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.643 15940 INFO nova.compute.manager [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.644 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.645 15940 INFO nova.compute.manager [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.645 15940 INFO nova.compute.manager [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:01.645 15940 INFO nova.compute.manager [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] During sync_power_state the instance has a pending task. Skip. 2015-07-22 14:46:40.684 15940 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-07-22 14:47:30.484 15940 WARNING nova.virt.libvirt.driver [req-cc39a956-c951-4c50-8d62-c8a9ef680d41 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:30.695 15940 INFO nova.compute.manager [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] VM Stopped (Lifecycle Event) 2015-07-22 14:47:30.701 15940 INFO nova.virt.libvirt.driver [-] [instance: f380346a-bc24-4d63-a6e0-e36b5a508a59] Instance destroyed successfully. 2015-07-22 14:47:30.928 15940 WARNING nova.virt.libvirt.driver [req-ad9a7e60-8dce-45ee-b572-eddaa7837680 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:30.963 15940 WARNING nova.virt.libvirt.driver [req-50f891f1-eccf-45ff-9283-dccc85f0d87f 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:31.125 15940 WARNING nova.virt.libvirt.driver [req-f122ba36-fe74-4966-b593-c1f20f8ed1ed 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:31.348 15940 INFO nova.virt.libvirt.driver [-] [instance: a3d9448d-25d8-4c01-925c-825a19164970] Instance destroyed successfully. 2015-07-22 14:47:31.558 15940 INFO nova.virt.libvirt.driver [-] [instance: fc396066-7ec6-4fe9-b7bf-70f162e293fe] Instance destroyed successfully. 2015-07-22 14:47:31.562 15940 INFO nova.virt.libvirt.driver [-] [instance: 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68] Instance destroyed successfully. 2015-07-22 14:47:31.699 15940 WARNING nova.virt.libvirt.driver [req-e8299f7f-0e3e-40ab-ba59-71a615eb8a7a 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:31.895 15940 WARNING nova.virt.libvirt.driver [req-785f9111-ffa4-45c2-9904-c32459c18e40 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:31.911 15940 INFO nova.virt.libvirt.driver [-] [instance: e5515b31-8edb-4558-bb9f-f6f8b1142db2] Instance destroyed successfully. 2015-07-22 14:47:32.118 15940 INFO nova.virt.libvirt.driver [-] [instance: 26d74bba-2e9a-4594-b6a1-1b27f337759b] Instance destroyed successfully. 2015-07-22 14:47:32.138 15940 WARNING nova.virt.libvirt.driver [req-6e4f3e1a-4d16-49c8-949b-4c8dbf78d2da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:32.350 15940 INFO nova.virt.libvirt.driver [-] [instance: ace0736a-0371-4a3d-8a7f-525e299a924b] Instance destroyed successfully. 2015-07-22 14:47:32.427 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672 2015-07-22 14:47:32.556 15940 WARNING nova.virt.libvirt.driver [req-ae8a97ee-f9cd-41bf-9e6c-1c7020583039 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:32.561 15940 WARNING nova.virt.libvirt.driver [req-fc60e213-fe51-475c-9546-4e3a14344591 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:32.702 15940 WARNING nova.virt.libvirt.driver [req-0494b0fe-f0c4-4b33-b596-f919429932da 92c4efc35a0a431b963e10f4552f668f 7ca2c6a04b45496cab9a41849af0263c] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Failed to soft reboot instance. Trying hard reboot. 2015-07-22 14:47:33.179 15940 INFO nova.virt.libvirt.driver [-] [instance: 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0] Instance destroyed successfully. 2015-07-22 14:47:33.184 15940 INFO nova.virt.libvirt.driver [-] [instance: a2d264f0-c1d0-449c-9ba3-32041247490c] Instance destroyed successfully. 2015-07-22 14:47:33.187 15940 INFO nova.virt.libvirt.driver [-] [instance: a993222f-4e41-483e-8e91-300be0d525a4] Instance destroyed successfully. 2015-07-22 14:47:33.320 15940 WARNING oslo.messaging._drivers.amqpdriver [-] Number of call queues is greater than warning threshold: 10. There could be a leak. 2015-07-22 14:47:33.369 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672 2015-07-22 14:47:33.369 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672 2015-07-22 14:47:33.370 15940 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672 2015-07-22 14:51:43.808 15940 WARNING nova.openstack.common.deadlock_monitor [-] service function nova.servicegroup.drivers.db._report_state is deadlocked! 2015-07-22 14:51:43.984 15940 WARNING nova.openstack.common.deadlock_monitor [-] deadlock traceback is [' File "/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner\n self.f(*self.args, **self.kw)\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/deadlock_monitor.py", line 58, in __deco\n ret = func(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 116, in _report_state\n service.service_ref, state_catalog)\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 218, in service_update\n return self._manager.service_update(context, service, values)\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 330, in service_update\n service=service_p, values=values)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call\n wait_for_reply=True, timeout=timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send\n timeout=timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 400, in _send\n conn.topic_send(topic, msg, timeout=timeout)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py", line 147, in __exit__\n self._done()\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py", line 136, in _done\n self.connection.reset()\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 656, in reset\n self.channel.close()\n', ' File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 163, in close\n (20, 41), # Channel.close_ok\n', ' File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait\n self.channel_id, allowed_methods)\n', ' File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 204, in _wait_method\n self.method_reader.read_method()\n', ' File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method\n self._next_method()\n', ' File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 112, in _next_method\n frame_type, channel, payload = read_frame()\n', ' File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 147, in read_frame\n frame_type, channel, size = unpack(\'>BHI\', read(7, True))\n', ' File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 245, in _read\n s = recv(131072)\n', ' File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 264, in recv\n timeout_exc=socket.timeout("timed out"))\n', ' File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 155, in trampoline\n return hub.switch()\n', ' File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch\n return self.greenlet.switch()\n'] 2015-07-22 14:51:44.007 15940 WARNING nova.openstack.common.deadlock_monitor [-] nova compute exit! 2015-07-22 23:22:08.095 28664 INFO nova.openstack.common.periodic_task [-] Skipping periodic task update_host_ssh_pubkey because its interval is negative 2015-07-22 23:22:08.124 28664 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2015-07-22 23:22:08.196 28664 INFO nova.virt.driver [-] Loading compute driver 'nova.virt.libvirt.LibvirtDriver' 2015-07-22 23:22:08.223 28664 INFO nova.vcpu_affinity.policy [-] no vcpupin policy is configed 2015-07-22 23:22:08.242 28664 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672 2015-07-22 23:22:08.257 28664 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.43.203.215:5672 2015-07-23 21:27:11.687 7924 INFO nova.openstack.common.periodic_task [-] Skipping periodic task update_host_ssh_pubkey because its interval is negative 2015-07-23 21:27:11.814 7924 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2015-07-23 21:27:11.884 7924 INFO nova.virt.driver [-] Loading compute driver 'nova.virt.libvirt.LibvirtDriver' 2015-07-23 21:27:12.051 7924 INFO nova.vcpu_affinity.policy [-] no vcpupin policy is configed 2015-07-23 21:27:12.067 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-07-23 21:27:13.069 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:27:13.069 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-07-23 21:27:14.078 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds. 2015-07-23 21:27:17.079 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:27:17.079 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-07-23 21:27:18.088 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds. 2015-07-23 21:27:23.092 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:27:23.092 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-07-23 21:27:24.101 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds. 2015-07-23 21:27:31.104 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:27:31.104 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-07-23 21:27:32.113 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds. 2015-07-23 21:27:41.117 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:27:41.118 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-07-23 21:27:42.127 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 2015-07-23 21:43:36.141 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:43:36.142 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-07-23 21:43:37.154 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. 2015-07-23 21:44:07.165 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:44:07.165 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-07-23 21:44:08.174 7924 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.43.203.215:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. 2015-07-23 21:44:38.182 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.43.203.215:5672 2015-07-23 21:44:38.182 7924 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... def deadlock_check(): while True: LOG.debug(_("service deadlock_check!, time is %f"), timeutils.monotonic_time()) now = timeutils.monotonic_time() for func in deadlock_dict.keys(): if deadlock_dict[func] < now: LOG.warn(_("service function %s is deadlocked!"),func) _print_deadlock_greenthreads(func) LOG.warn(_("nova compute exit!")) cmd = "ps -e|grep nova-compute | awk '{print $1}'|xargs kill -9" os.system(cmd) sys.exit(0) greenthread.sleep(10)