MAAS KVM (Pods) on Power wasn't able to spawn VMs

[Frank Heimes]

While giving MAAS with controller and KVM on Power
the VMs didn#t came up and ended up in Pause mode.

Trying to start them manually (using virsh) led to the same situation.

The installation and Pod configuration looked good.

There are some errors in the logs, but I'm not sure if they are really related to this.
Anyway, all logs are attached ...

[Newell Jensen (newell-jensen)]

I suspect it has to do with:

https://paste.ubuntu.com/p/r5KG9MMHTf/

I am going to mark incomplete but if you can fix your keys and you are still seeing this issue please re-open.

[Andrew Cloke (andrew-cloke)]

Thanks Newell. As MAAS is spawning the VMs, doesn't MAAS manage the keys - through cloud-init?

[Newell Jensen (newell-jensen)]

Hello Andy,

Here is the sequence of steps if I am following correctly. Frank, please correct me if I am wrong:

1. Ubuntu is installed
2. Some network setup
3. Copy over ssh keys so can ssh into machine
4. Install libvirt
5. Add maas ppa
6. Install maas from ppa
7. ssh keys created for maas user
8. ssh keys added via the MAAS UI's prefs screen
9. logs are checked and regiond.log is showing stacktraces

So we are seeing issues as soon as the ssh keys were added to MAAS.

The first stacktraces from what I can see are related to the ssh keys and is the paste that I have in the previous comment.

Frank, can you please confirm that you addressed the ssh keys stacktrace that was being reported here? Since MAAS uses ssh to communicate with virsh I want to make sure that this is not causing an issue, which it seems to be.

Thanks

[Frank Heimes (fheimes)]

I used a slightly different sequence:

1) installed the host OS (with initial base nw setup)
2) ssh-copy-id the credentials from my workstation over
3) kvm/libvirt installation
4) maas installation ppa:maas/stable
5) network adjustment - adding bridge for kvm on 2nd interface
6) adjusting kvm/libvirt network config
7) iptables changes
8) start and initialize maas
9) added my LP key to maas
10) configured dynamic range and dhcp in maas
11) started to enable Pods - generated ssh keys and added them to maas
    key of default user at host and maas user
12) added a pod in the maas UI
13) finally composed a vm

In the past I successully used such an 'all-in-one' test setup on s390x (except LP 1859656), hence I followed that approach (https://ubuntu-on-big-iron.blogspot.com/2019/08/maas-kvm-on-s390x-part1.html) which is btw. again largely based on the maas tutorial: https://tutorials.ubuntu.com/tutorial/create-kvm-pods-with-maas, but a bit modified.

But the steps are all listed in the txt file attached in comment #2.

If I open that and for example search for traceback, it wasn't obvious to me that these issues could have been something to do with the ssh keys ...

Well, I was a bit in hurry when I quickly that, but up to the composing of a machine I did not faced a problem (at least everything worked like expected) and I didn't got any error msgs in the UI.
While trying to compose I saw that the states changed quickly, but the VM didn't come up.

Unfortunately I cannot double check the ssh keys, since I don't have the setup anymore, I even do not have access to the system at all (since I needed/wanted to use one that itself was not already managed by maas - so I borrowed one from bladernr).

[Andrew Cloke (andrew-cloke)]

As MAAS 2.8 adopts LXD to create VMs and LXC containers, support for ppc64el should be fixed.

[Patricia Domingues (patriciasd)]

Frank, I don't see in the logs that you've turned off SMT. As you were testing a POWER8 system.
SMT feature is visible only inside the guests and needs to be disabled in the host.
You are able to reproduce that, creating a VM on a Power8 host with SMT enabled it goes to paused state.
```
$ sudo ppc64_cpu --smt
SMT=8

# virsh list --all
 Id Name State
----------------------------
 1 smt8 paused
```

Disabling SMT and creating a new VM (called bionic) I'm able to ssh into it:
```
$ ppc64_cpu --smt
SMT is off

$ sudo virsh list --all
 Id Name State
----------------------------
 1 smt8 paused
 2 bionic running
```

[Patricia Domingues (patriciasd)]

I've tested deb-based MAAS 2.8.1 (2.8.1-8567-g.c4825ca06-0ubuntu1~18.04.1) on a Power9 host with `
Ubuntu 18.04.4 LTS (GNU/Linux 5.4.0-41-generic ppc64le)` and it works fine.

I've tested both KVM host types - Virsh and LXD.
As I was using Ubuntu Bionic I did remove deb-based LXD and installed snap LXD.

[Andrew Cloke (andrew-cloke)]

As this works fine with MAAS 2.8.x, closing out as "Fix Released".