lxd Virtual Machines are not deployed sometimes.

[Diego Mascialino]

We are running our system tests using lxd VMs.

Several (but not all) executions fails on deploying step.
The machine times out with state:

 'status_name': 'Deploying',
 'status_message': 'Rebooting',
 'power_state': 'off'

[Alexsander de Souza (alexsander-souza)]

Please, can you attach the maas logs of one failed execution?

[Diego Mascialino (dmascialino)]

Well.. we have several fialed executions.
Please find attached the SOS report of maas-system-tests/1341/

[Diego Mascialino (dmascialino)]

I found these lxd logs in our jenkins server:

```
141 ubuntu@jenkins-slave-2:~/diego/system-tests$ journalctl -u snap.lxd.daemon.service --since "7 days ago" | grep virtual-machines
Jun 20 23:05:28 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-20T23:05:28Z" level=error msg="Failed to cleanly stop instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm2': device or resource busy" instance=vm2 instanceType=virtual-machine project=default
Jun 20 23:05:28 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-20T23:05:28Z" level=error msg="Failed to restart instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm2': device or resource busy" instance=vm2 instanceType=virtual-machine project=default
Jun 21 01:59:01 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-21T01:59:01Z" level=error msg="Failed to cleanly stop instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1': device or resource busy" instance=vm1 instanceType=virtual-machine project=default
Jun 21 01:59:01 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-21T01:59:01Z" level=error msg="Failed to restart instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1': device or resource busy" instance=vm1 instanceType=virtual-machine project=default
Jun 21 15:38:54 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-21T15:38:54Z" level=error msg="Failed to cleanly stop instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1': device or resource busy" instance=vm1 instanceType=virtual-machine project=default
Jun 21 15:38:54 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-21T15:38:54Z" level=error msg="Failed to restart instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1': device or resource busy" instance=vm1 instanceType=virtual-machine project=default
Jun 22 23:56:42 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-22T23:56:42Z" level=error msg="Failed to cleanly stop instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1': device or resource busy" instance=vm1 instanceType=virtual-machine project=default
Jun 22 23:56:42 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-22T23:56:42Z" level=error msg="Failed to restart instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1': device or resource busy" instance=vm1 instanceType=virtual-machine project=default
Jun 22 23:56:52 jenkins-slave-2 lxd.daemon[3200651]: time="2022-06-22T23:56:52Z" level=error msg="Failed to restart instance" err="Failed unmounting instance: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1': device or resource busy" instance=vm1 instanceType=virtual-machine project=default
Jun 23 23:02:28 j...

[Jerzy Husakowski (jhusakowski)]

We've seen something like this recently - the VM was stuck in EFI shell and LXD was unable to stop the VM using normal methods. Adding `--force` parameter kills the stuck VM. The system test has been updated.
Is the issue reproducible after this update?

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]