juju behaviour in multi-hypervisor OpenStack clouds

[James Page]

Hi

I'm working on enabling deployment of OpenStack Clouds using Juju with mixed hypervisors; specifically LXD and KVM.

This means that the cloud can support instances of two different types, which has some impact on Juju and we have a few options on how this might get setup so that Juju can consume different types of resources.

a) Glance image properties

Right now, we don't by default set the 'hypervisor_type' property on a glance image; in a multi-hypervisor cloud, we'll have to maintainer different images for KVM and LXD, and this property would be set to indicate which type of hypervisor and instance using the image must be scheduled on.

The images are query-able using simplestreams data published in the cloud - Juju could use this to pick the right image for LXD or KVM instance, and then Nova would take care of the rest:

     juju deploy mysql --constraints="virt-type=lxd,mem=8G,cpu=4"

     juju deploy mysql --constraints="virt-type=kvm,mem=8G,cpu=4"

This provides flexibility as constraints are not tied to specific instances types (see b).

b) Extra specifications on flavors

Alternatively (or as well), we can setup a default set of flavors in the OpenStack cloud with extra specifications that set the hypervisor type using extra specs (a feature of nova's scheduler that ensures that a specific flavor gets scheduled on the right hypervisor) - for example (type naming TBD - these are just for example).

    juju deploy mysql --constraints="instance-type=l1.small" (LXD instance)

    juju deploy mysql --constraints="instance-type=k1.small" (KVM instance)

this obviously depends on the cloud being setup with the right flavor mappings, and it makes constraints potentially not portable across openstack clouds.

The above describes the two mechanisms by which we can configure the OpenStack cloud to support multiple hypervisors; I think that how juju exposes this to end users is still up for discussion.

[Cheryl Jennings (cherylj)]

Adding in an extra constraint type for openstack shouldn't be too hard. Is "virt-type" something that exists in simplestreams already?

What's the timeframe you were looking at for this?

[James Page (james-page)]

Right now, simplestreams does not populate a virt-type attribute - see below:

{
 "datatype": "image-ids",
 "updated": "Mon, 14 Dec 2015 09:28:06 +0000",
 "content_id": "localcloud.RegionOne:partners",
 "products": {
  "com.ubuntu.cloud:server:14.04:amd64": {
   "pubname": "ubuntu-trusty-14.04-amd64-server-20151117",
   "versions": {
    "20151117": {
     "items": {
      "root.tar.xz": {
       "ftype": "root.tar.xz",
       "endpoint": "http://10.5.36.137:5000/v2.0",
       "name": "ubuntu/ubuntu-trusty-14.04-amd64-server-20151117-root.tar.xz",
       "region": "RegionOne",
       "sha256": "468c16db671da401325659e3ec51100e3728e71a867f4dcdef7b104d6b5d8559",
       "size": "123917268",
       "id": "38f2cc3b-2461-4ce8-a4ea-25086beec757",
       "md5": "65fbd5a7e97643881975493f775e4907"
      },
      "disk1.img": {
       "ftype": "disk1.img",
       "endpoint": "http://10.5.36.137:5000/v2.0",
       "name": "ubuntu/ubuntu-trusty-14.04-amd64-server-20151117-disk1.img",
       "region": "RegionOne",
       "sha256": "e49d43f60f32590c9dbd0dbb40b71707ec7ba2239531556bb4fe4097e6d1bac4",
       "size": "258802176",
       "id": "f527d208-5293-4eb5-8a42-d73c68f3f6a8",
       "md5": "4977f2e809492537b409d669feb5db58"
      }
     }
    }
   },
   "os": "ubuntu",
   "supported": "True",
   "release_title": "14.04 LTS",
   "label": "release",
   "release_codename": "Trusty Tahr",
   "version": "14.04",
   "release": "trusty",
   "owner_id": "80f44871ee1744c583ffb2f21c3dd690",
   "arch": "amd64",
   "support_eol": "2019-04-17"
  }
 },
 "format": "products:1.0"
}

Adding cloud specific data such as this should be possible - I'll take a look.

[James Page (james-page)]

Timescale: multi-hypervisor regions are a 16.04 objective for the openstack team.

[Cheryl Jennings (cherylj)]

Added this to the list of bugs that should be considered for 2.0.

[Richard Harding (rharding)]

We need to look at using a) because the second path requires too much manual setup and 'getting it right' instead of it being more automatic for end users and hurts portability.

[Cheryl Jennings (cherylj)]

@james-page - do you have an proposed simplestreams with virt-type I could look at? And, do you have an openstack environment I could use to test while adding this functionality?

[Antonio Rosales (arosales)]

@cherylj,

Apologies for being dense here. Is there design work we need to do first before looking at the solution? Perhaps that is already complete, and I just need to be educated. Thus, (from a Juju user perspective deploying OpenStack with multi-hypervisor support) will the user specify virt type at the juju command line for a given service?

-thanks,
Antonio

[Anastasia (anastasia-macmood)]

@arosales,

The design has been done. Supporting data models exist and are being used for ec2. We just need to bring our juju openstack-related implementation to the same level of sophistication :D

[Anastasia (anastasia-macmood)]

On further inspection, using virt-type in Openstack images metadata will work out of the box on juju side. Simplestream metadata needs to be provided.

However, what is needed to support using desired hypervisor on deploy is to extend constraints to contain virt-type as a value.

I'll extend constraints and will open this option up for openstack only at this stage: openstack constraints validator will be modified accordingly. This behaviour will be similar to instance-type.

[Anastasia (anastasia-macmood)]

First step: adding virt-type to constraints (juju side): https://github.com/juju/juju/pull/4752

[Anastasia (anastasia-macmood)]

Second step: black list virt-type as a constraint for all providers: https://github.com/juju/juju/pull/4753

Individual providers will need to be dealt with separately when multi-visor support is enabled.

[Anastasia (anastasia-macmood)]

Third step, updating goose: https://github.com/go-goose/goose/pull/20

[Anastasia (anastasia-macmood)]

Third step above, modification of the goose, is not needed.

The last change that is needed is proposed against master: https://github.com/juju/juju/pull/4835.

For openstack clouds seeded with images where virtualisation type is defined, this PR allows to specify "--constraints="virt-type=...". Currently supported types are lxd and qemu.

For deployments with no constraints specified, where an image with virtualisation type is used, instance type virtualisation type will reflect it.

[Anastasia (anastasia-macmood)]

With the above proposal landing in juju master, juju now supports multi-hypervisor OpenStack clouds.

The cloud needs to be populated with streams data that has the "virt" set to 'qemu' and 'lxc' for KVM and LXD respectively.

For example,

  "com.ubuntu.cloud.daily:server:12.04:i386": {
   "aliases": "12.04,p,precise",
   "arch": "i386",
   "endpoint": "http://10.245.161.156:5000/v2.0",
   "virt": "qemu",
    ...
   "versions": {
    "20160311": {
     "items": {
        ...
      }
     },
     "pubname": "ubuntu-precise-daily-i386-server-20160311"
    },

Originally offered cloud was not populated with the images in format that Juju recognized. James is working on correcting format. I'll re-test manually as soon as it is available.

[James Page (james-page)]

Anastasia

I've updated the format of the streams data in the test cloud; I also stood back a bit and decided that 'qemu' and 'lxc' are the underlying mechanics rather than the 'virt' type - so supported virt-type flags should really be 'lxd' and 'kvm' as I think that's what end-users really talk about.

[Anastasia (anastasia-macmood)]

James \o/

Thank you for clarification! This is brilliant.

Juju proposal against master: https://github.com/juju/juju/pull/4875