Can't destroy libvirt domain on TE host

[Derek Higgins]

From http://logs.openstack.org/99/85099/12/check-tripleo/check-tripleo-overcloud-f20/f32ba7e/console.html

2014-06-30 15:57:16.025 | | 262751a3-3ac6-4047-8c04-dd70ca1f8de9 | overcloud-controller0-bd7j3qctck3o | ERROR | spawning | NOSTATE | |

The virtual power driver is having trouble killing a particular VM on the TE host, This is causing about 7% of overcloud jobs to fail

[req-41c8745e-8c87-475d-bae3-6d2b96fbe6af None] Error running command: /usr/bin/virsh destroy "baremetalbrbm2_2"
Traceback (most recent call last):
 File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/virt/baremetal/virtual_power_driver.py", line 228, in _run_command
self._connection, cmd, check_exit_code=check_exit_code)
File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 271, in ssh_execute
    cmd=cmd)
ProcessExecutionError: Unexpected error while running command.
Command: /usr/bin/virsh destroy "baremetalbrbm2_2"
Exit code: 1
Stdout: '\n'
Stderr: 'Calling /usr/bin/virsh destroy "baremetalbrbm2_2" \nerror: Failed to destroy domain baremetalbrbm2_2\nerror: Failed to terminate process 8389 with SIGKILL: Device or resource b

Also confirmed on the TE host
[root@testenv2-testenv1-iqjvpciq5dzj ~]# virsh destroy baremetalbrbm2_2
error: Failed to destroy domain baremetalbrbm2_2
error: Failed to terminate process 8389 with SIGKILL: Device or resource busy

[Derek Higgins (derekh)]

Will rebuild the TE host in a few hours when its doing nothing

[Robert Collins (lifeless)]

This is happening again at the moment.

[Brent Eagles (beagles)]

Derek, is this still relevant? The full logs are long gone so I'm going to mark as incomplete. Let me know if we need to re-open.

[Derek Higgins (derekh)]

I don't think this is still happening (at least with any frequency), I think we resolved the problem with we introduced night restarts of ovs on the TE hosts. I'm closing this for now.