Compose machine failure: Start tag expected, '<' not found, line 1, column 1
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Expired
|
Medium
|
Unassigned |
Bug Description
Attempting to compose a machine on a virsh pod failed for me with an error like below:
$ maas maas1 pod compose 2
Unable to compose machine because: Failed talking to pod: Start tag expected, '<' not found, line 1, column 1
Full traceback from regiond.log is at http://
See the bottom of the bug for full reproduction steps. The error goes away after reboot if your virsh pod is properly set up, so I don't think this is a high priority. Workaround (other than reboot) is to restart libvirt-bin service.
Looking at what get_domain_
virsh domcapabilities --virttype kvm
This was a nested VM instance where it failed with:
ubuntu@maas1:~$ virsh domcapabilities --virttype kvm
error: failed to get emulator capabilities
error: invalid argument: unable to find any emulator to serve 'x86_64' architecture
so blindly attempting to parse that as XML failed.
Calling out to "virsh domcapabilities --virttype kvm /usr/bin/
It turns out the problem was that after installing qemu-kvm package, one needs to restart libvirt-bin service to get it to realize x86_64 emulator is present:
sudo systemctl restart libvirt-bin.service
After that, everything worked fine too.
We should:
1. Fix get_domain_
2. Surface an error from virsh call instead of the XML parsing error
3. Perhaps suggest a workaround to the user, even though they might not be an administrator of the POD (install qemu-kvm package and restart libvirt-bin service)
4. Maybe even file a bug against eg. qemu-kvm package (to restart libvirt-bin itself) or libvirtd (to watch for appearing emulators with inotify or whatever the latest FS monitoring solution is)
Alternatively, we could just use the detected architecture ourselves and try to find the emulator to "teach" libvirt-bin service about its presence if the package is already installed, though I am not sure if this would work well with a remote virsh connection.
To reproduce (probably possible to reproduce with just qemu-system-x86 instead of qemu-kvm package):
sudo apt remove -y qemu-kvm; sudo apt autoremove -y; sudo apt install -y qemu-kvm; sudo systemctl restart libvirt-bin.service
virsh domcapabilities --virttype kvm
maas maas-connection pod compose POD-ID (get it using "maas maas-connection pods read")
All the other places in virsh pod driver seem to at least check for xml output being None (though in this case, I do get an "\n" and not an empty string on stdout) before attempting to parse as XML with etree.XML().
Related branches
- Данило Шеган (community): Approve
- MAAS Maintainers: Pending requested
-
Diff: 35 lines (+14/-0)2 files modifiedsrc/provisioningserver/drivers/pod/tests/test_virsh.py (+4/-0)
src/provisioningserver/drivers/pod/virsh.py (+10/-0)
- Blake Rouse (community): Approve
-
Diff: 37 lines (+6/-6)2 files modifiedsrc/provisioningserver/drivers/pod/tests/test_virsh.py (+1/-1)
src/provisioningserver/drivers/pod/virsh.py (+5/-5)
Changed in maas: | |
importance: | Undecided → Medium |
milestone: | none → 2.2.0rc5 |
status: | New → Triaged |
Changed in maas: | |
assignee: | nobody → Newell Jensen (newell-jensen) |
Changed in maas: | |
status: | Triaged → In Progress |
Changed in maas: | |
milestone: | 2.2.0rc5 → 2.2.1 |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas: | |
status: | Fix Committed → Fix Released |
Changed in maas: | |
status: | Fix Released → Fix Committed |
Changed in maas: | |
assignee: | Newell Jensen (newell-jensen) → nobody |
Changed in maas: | |
milestone: | 2.2.1 → none |
Changed in maas: | |
status: | New → Triaged |
We encountered this bug (or at least looks like it) three times while testing 2.8.5-deb
After we redeployed the machine, it works again.
Here is one of the runs
https:/ /oil-jenkins. canonical. com/artifacts/ a1657646- db57-49fd- 9d8d-df28c777f6 53/index. html