Disk erasure doesn't work, with KVM-backed node at least

[Peter Matulis]

With KVM-backed nodes, any type of disk erasure attempted failed. The node (node1) status would spin indefinitely on 'Disk erasing' until Aborted 5 minutes later, to arrive at 'Failed disk erasing'.

/var/log/maas/maas.log contains:

Nov 27 02:49:51 localhost maas.node: [info] node1: Status transition from DEPLOYED to DISK_ERASING
Nov 27 02:49:51 localhost maas.power: [info] Changing power state (on) of node: node1 (yspag4)
Nov 27 02:49:51 localhost maas.node: [info] node1: Disk erasure started
Nov 27 02:49:54 localhost maas.power: [info] Changed power state (on) of node: node1 (yspag4)
Nov 27 02:54:51 localhost maas.power: [info] Changing power state (off) of node: node1 (yspag4)
Nov 27 02:54:51 localhost maas.node: [info] node1: Status transition from DISK_ERASING to FAILED_DISK_ERASING
Nov 27 02:54:51 localhost maas.node: [info] node1: Disk erasing aborted
Nov 27 02:54:54 localhost maas.power: [info] Changed power state (off) of node: node1 (yspag4)

/var/log/syslog on the KVM host contains:

Nov 27 02:49:52 localhost systemd[1]: Started Session 761 of user ubuntu.
Nov 27 02:49:52 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:49:53 localhost systemd[1]: Started Session 762 of user ubuntu.
Nov 27 02:49:53 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:49:54 localhost systemd[1]: Started Session 763 of user ubuntu.
Nov 27 02:49:54 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:49:58 localhost kernel: [301426.466567] kvm [17676]: vcpu0 unhandled rdmsr: 0x611
Nov 27 02:49:58 localhost kernel: [301426.470139] kvm [17676]: vcpu0 unhandled rdmsr: 0x639
Nov 27 02:49:58 localhost kernel: [301426.472261] kvm [17676]: vcpu0 unhandled rdmsr: 0x641
Nov 27 02:49:58 localhost kernel: [301426.474095] kvm [17676]: vcpu0 unhandled rdmsr: 0x619
Nov 27 02:50:11 localhost systemd[1]: Started Session 764 of user ubuntu.
Nov 27 02:50:11 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:50:21 localhost systemd[1]: Started Session 765 of user ubuntu.
Nov 27 02:50:21 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:52:23 localhost systemd[1]: Started Session 766 of user ubuntu.
Nov 27 02:52:23 localhost systemd[1]: Started Session 767 of user ubuntu.
Nov 27 02:52:23 localhost systemd[1]: Started Session 768 of user ubuntu.
Nov 27 02:52:23 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:52:23 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:52:23 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:53:53 localhost systemd[1]: Started Session 769 of user ubuntu.
Nov 27 02:53:53 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:54:51 localhost systemd[1]: Started Session 770 of user ubuntu.
Nov 27 02:54:51 localhost kernel: [301719.480555] br0: port 4(vnet2) entered disabled state
Nov 27 02:54:51 localhost kernel: [301719.483444] device vnet2 left promiscuous mode
Nov 27 02:54:51 localhost kernel: [301719.483450] br0: port 4(vnet2) entered disabled state
Nov 27 02:54:51 localhost kernel: [301719.524283] br0: port 5(vnet3) entered disabled state
Nov 27 02:54:51 localhost kernel: [301719.526252] device vnet3 left promiscuous mode
Nov 27 02:54:51 localhost kernel: [301719.526255] br0: port 5(vnet3) entered disabled state
Nov 27 02:54:52 localhost virtlogd[15406]: End of file while reading data: Input/output error
Nov 27 02:54:52 localhost kernel: [301719.977783] audit: type=1400 audit(1480215292.384:271): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-6861a944-a682-4adc-9c36-c283e3b75d11" pid=19819 comm="apparmor_parser"
Nov 27 02:54:52 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:54:53 localhost systemd[1]: Started Session 771 of user ubuntu.
Nov 27 02:54:53 localhost libvirtd[15535]: End of file while reading data: Input/output error
Nov 27 02:54:54 localhost systemd[1]: Started Session 772 of user ubuntu.
Nov 27 02:54:54 localhost libvirtd[15535]: End of file while reading data: Input/output error

If it matters, the node has 2 disks.

[Peter Matulis (petermatulis)]

I (libvirt) detached the second disk for another node but I got the same results when I tried to erase its disk upon release.

[Blake Rouse (blake-rouse)]

I think the problem is that power cycle is not occuring. Try this to see if it works, then we can do the correct fix.

With a Deployed Machine:

1. Action "Power Off".
2. Wait for it to power off.
3. Action "Release w/ disk erasing."
4. Wait for power to come on and erase and release.

If that fixes this issue then we need to change the power action on disk erasing to power cycle.

[Peter Matulis (petermatulis)]

Yes, works using your procedure.