No arm64 KVM container support

[Craig Magina]

I tried the workaround from https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1317680, but that did not solve the issue.

The steps I am performing to get here are:
juju bootstrap
uvt-simplestreams-libvirt purge
uvt-simplestreams-libvirt sync release=trusty arch=arm64
juju add-machine

Here are some outputs:
juju status
environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.20.1.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
    state-server-member-status: has-vote
  "1":
    agent-state-info: 'kvm container creation failed: exit status 1'
    instance-id: pending
    series: trusty
services: {}

root 1615 0.1 0.9 2029096 322460 ? Ssl Jul28 1:14 /home/ubuntu/.juju/local/tools/machine-0/jujud machine --data-dir /home/ubuntu/.juju/local --machine-id 0 --debug
root 1635 0.1 0.1 2013300 40752 ? Ssl Jul28 1:22 /usr/lib/juju/bin/mongod --auth --dbpath=/home/ubuntu/.juju/local/db --sslOnNormalPorts --sslPEMKeyFile /home/ubuntu/.juju/local/server.pem --sslPEMKeyPassword xxxxxxx --bind_ip 0.0.0.0 --port 37017 --noprealloc --syslog --smallfiles --journal --keyFile /home/ubuntu/.juju/local/shared-secret --replSet juju --oplogSize 1
ubuntu 29772 0.0 0.0 5136 824 pts/1 S+ 10:08 0:00 grep --color=auto juju

uvt-simplestreams-libvirt query
release=trusty arch=arm64 label=release (20140724)

[Craig Magina (craig.magina)]

Was able to reproduce this issue with the newer version of juju found here: http://juju-ci.vapour.ws:8080/job/publish-revision/711/

[Craig Magina (craig.magina)]

Using the tools from here: https://juju-dist.s3.amazonaws.com/rc-testing/tools

[Robie Basak (racb)]

What does "uvt-kvm create foo release=trusty arch=arm64 label=release" give you after the failure, please? Is uvt-kvm known to work correctly on arm64?

[Craig Magina (craig.magina)]

uvt-kvm create foo release=trusty arch=arm64 label=release
uvt-kvm: error: libvirt: XML error: No PCI buses available

I am not sure.

[Robie Basak (racb)]

OK, so we need to fix uvt-kvm in the first instance; Juju cannot be expected to support KVM containers until uvtool works.

Take a look at /usr/share/uvtool/libvirt/template.xml. This is what is supplied to libvirt by default (after some automatic manipulation for other command line arguments and defaults such as a disk image to use, memory size, etc). You can override this baseline definition with "uvt-kvm create --template=... release=trusty arch=arm64 label=release".

The XML schema here is defined by libvirt (specified in http://libvirt.org/formatdomain.html). What we need is either a corrected XML that uvtool should use for all architectures, or per-architecture XML definitions and a feature in uvtool to use the correct architecture's XML definitions by default (I'm curious as to how it should detect this; should detection be based on the architecture on the host, on the guest, or both?).

This may involve pushing patches to libvirt if support is not already there. I don't see anything PCI-specific in the default XML, so why does libvirt fail with that error? Do we know what the status of arm64 is in libvirt?

[Robie Basak (racb)]

Throwing this to upstream for now, since it's not specific to Ubuntu's Juju packaging. I suppose this is a headline Wishlist item for Juju. It looks like it'll require tasks in libvirt and/or uvtool for implementation; I'm not sure which.

[Robie Basak (racb)]

Marking the uvtool task Incomplete for now, as I can't do anything in uvtool until I know that libvirt works correctly with arm64 and how to call libvirt on arm64.

[Ali (asaidi)]

I have libvirt working on arm64 with the following command line and template.

There are a couple of issues.
1) The QEMU_EFI.fd in the wily qemu-efi package doesn't work so I had to use one from linaro:
http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/458/QEMU-AARCH64/RELEASE_GCC49/QEMU_EFI.fd

2) The images that are being used to boot the machine don't have an EFI partition, so the boot fails. Specifying the cloud uefi image solves the problem.

I can also use default the boot to network and it is successfully netbooted by MAAS.

#uvt-kvm create --template=template-arm-net2.xml test2 release=wily --backing-image-file=/root/wily-server-cloudimg-arm64-uefi1.img

# virsh console test2
error: no suitable video mode found.
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.2.0-23-generic (buildd@bos01-arm64-027) (gcc version 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) ) #28-Ubuntu SMP Sun Dec 27 17:47:01 UTC 2015 (Ubuntu 4.2.0-23.28-generic 4.2.6)
[ 0.000000] CPU: AArch64 Processor [411fd072] revision 2
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] alternatives: enabling workaround for ARM erratum 832075
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] EFI v2.50 by EDK II
[ 0.000000] efi:

It looks like the template in MAAS (https://github.com/juju/juju/blob/master/container/kvm/template.go) will also need some changes to support arm64.

[Ali (asaidi)]

I've been able to convince juju to start a KVM container on an arm64 platform using the information above. No changes were required to juju, however i did discover a bug in juju not properly detecting the correct KVM host architecture and providing the wrong juju tools. This can be worked around by specifying a constraint for the architecture --constraints arch=arm64.

Otherwise to get the juju to deploy a KVM container I had to replace the /usr/share/uvtool/libvirt/template.xml with the xml attached to the previous comment. That XML file had to used the QEMU_EFI.fd I linked above, not the one from the qemu-efi package, and I had to modify simplestreams to request the uefi1 image. It looks like http://bazaar.launchpad.net/~utlemming/simplestreams/uefi1/revision/239#simplestreams/mirrors/glance.py does something similar.

[Robie Basak (racb)]

Thanks Ali, this is very useful.

[Ali (asaidi)]

No problem Robie. I'm pretty interested in getting these issues addressed, so if I can be more help please let me know.

For the template issue I think uvtool/libvirt/kvm.py could be changed to request different xml files for aarch64 vs x86_64. It's not very different, but I'm not sure it can be unified.

The QEMU_EFI.fd needs to be addressed pulling in a new upstream version of the code. There already appears to be a bug in Debian to do that and I assume it will make it's way into Ubuntu after that happens.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=810495

Lastly, I don't completely understand how the filters should be changed is simple stream.py to choose uefi1 disks vs disk1. Should that be an aarch64 only change or for x86 as well assuming that the version of ubuntu requested is new enough that uefi1 images are available?

Thanks
Ali

[Ali (asaidi)]

An updated qemu-efi package is now in Xenial.

[Ali (asaidi)]

Hi Robie,

Anything we can do to move this forward? Xenial has the correct packages but uvtool still doesn't work out of the box. There are still two issues:
(1) /usr/shareuvtool/libvirt/template.xml doesn't needs some changes for aarch64. The following works although it might be able to be simplified:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <os>
    <type arch='aarch64' machine='virt'>hvm</type>
    <loader readonly='yes' type='rom'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <boot dev='hd' />
  </os>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>host</model>
  </cpu>
  <clock offset='utc'/>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <console type='pty'/>
      <console type='pty'>
      <target type='virtio' port='0'/>
    </console>
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
   </devices>
</domain>

(2) for aarch64 simple streams needs to get the uefi1 version of the image for aarch64

I'm not sure what how simple-streams should be changed, but replacing disk1 with uefi1 does the job e.g.: sed -i 's/disk1.img/uefi1.img/g /usr/lib/python2.7/dist-packages/uvtool/libvirt/simplestreams.py

[Robie Basak (racb)]

Could someone explain the significance of the disk1.img -> uefi1.img change please? I see that on both amd64 and arm64 both image types are shipped. What does disk1.img for arm64 mean? Is it usable at all?

[Ali (asaidi)]

The difference between disk1 and uefi1 is that uefi1 is GPT partitioned and has the UEFI boot loader in it. Since the arm64 system will boot via UEFI I believe it's necessary to use that image.

[Robie Basak (racb)]

On Mon, Oct 17, 2016 at 02:29:28PM -0000, Ali wrote:
> The difference between disk1 and uefi1 is that uefi1 is GPT partitioned
> and has the UEFI boot loader in it. Since the arm64 system will boot via
> UEFI I believe it's necessary to use that image.

Thanks. In Yakkety, we're shipping only one image, which is disk1, but
I'm told that this is dual GPT and MBR (so on Intel architectures, both
MBR-based and UEFI boot should work). In theory then, on arm64 disk1 on
Yakkety will work. Please could you confirm?

This makes incorporating proper support in uvtool quite painful, because
the correct ftype then becomes different depending on release and
simplestreams doesn't support a complex enough query for uvtool to dtrt.

However, if the ftype=disk1.img arm64 image in Xenial and earlier is
entirely useless to everyone, then we could just replace it with an
alias to uefi1.img. Then uvtool would sync a working image for all
releases without needing any code changes.

However, if somebody is using ftype=disk1.img for arch=arm64 in Xenial
or earlier, then this may cause a regression.

Do you know anything that might help either way?

Thanks,

Robie

[Ali (asaidi)]

Hi Robie,

I don't know if anyone is using the disk1 image, but I'll do a few experiments with the Yakkety image and see if it works as expected.
Ali

[dann frazier (dannf)]

fwiw, the yakkety disk1 image boots fine in UEFI mode, so it is appropriately partitioned.

[Reed O'Brien (reedobrien)]

The following changes are in develop, on target for 2.2 release:
 - uvtools dependency has been removed,
 - arm64 is now works on xenial guests/hosts
 - trusty needs an HWE kernel and qemu-efi packaged and installed.