lxc nested within kvm machines do not use lxc-clone

[Adam Stokes]

[Impact]
When deploying services within an LXC nested in a KVM machine the lxc-clone support is not used.

[Test Case]

status output:

environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.18.1.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
  "1":
    agent-state: started
    agent-version: 1.18.1.1
    dns-name: 10.0.3.129
    instance-id: poe-local-machine-1
    series: trusty
    containers:
      1/lxc/0:
        instance-id: pending
        series: precise
    hardware: arch=amd64 cpu-cores=1 mem=512M root-disk=8192M
services:
  wordpress:
    charm: cs:precise/wordpress-21
    exposed: false
    relations:
      loadbalancer:
      - wordpress
    units:
      wordpress/0:
        agent-state: pending
        machine: 1/lxc/0

lxc-ls output from the kvm machine

ubuntu@ubuntu:~$ sudo lxc-ls --fancy
NAME STATE IPV4 I PV6 AUTOSTART
---------------------------------------------------------------------
juju-machine-1-lxc-0 RUNNING 10.0.3.105 - NO

No juju-trusty-template is ever defined for re-use.

[Regression Potentional]
This only applies to new bootstrapped environments with lxc-use-clone applied. Existing environments will continue unaffected.

Thanks
Adam

[Adam Stokes (adam-stokes)]

Curtis,

Could we get the importance set higher? Without lxc cloning support in kvm instances deploy times can take a significantly longer amount of time (upwards of 30minutes to deploy openstack)

Thanks
Adam

[Ian Booth (wallyworld)]

Cloning support was only intended as a local provider feature due to potential issues with placing a large blob (the lxc template) into the kvm container. I'll retriage as high and see if we can mitigate any potential issues that may affect the viability of a solution.

[Adam Stokes (adam-stokes)]

> Cloning support was only intended as a local provider feature due to potential issues with placing a large blob (the lxc template) into the kvm container.

sorry, this statement is a little confusing, to clarify are you talking about local provider and LXC? As local provider supports both kvm/lxc and also we have the ability to deploy both kvm/lxc in the same environment. Since we can not run lxc 'machines' along side kvm 'machines' we have to nest those containers within a kvm machine. with that said should juju treat a kvm machine any differently than a bare metal machine via maas? the cloning features intended for a local provider deploy should not be limited to the machine creation but also available to the containers within a machine.

[John A Meinel (jameinel)]

You can use the "juju deploy --to lxc:0" or "juju deploy --to kvm:0" to
create a container on the same machine-0 as the other ones you are
creating. I'm not sure how that compares to what you are doing right now.

It certainly wasn't expect that the normal behavior would be to start a KVM
instance and then run 10 LXC instances inside of it. I certainly could see
a case for having a flag for using templates when deploying containers
inside of regular instances, but it does cause a large increase in initial
consumption if your target filesystem doesn't support cheap clones.

John
=:->

On Tue, May 6, 2014 at 8:09 PM, Adam Stokes <email address hidden>wrote:

> > Cloning support was only intended as a local provider feature due to
> potential issues with placing a large blob (the lxc template) into the
> kvm container.
>
> sorry, this statement is a little confusing, to clarify are you talking
> about local provider and LXC? As local provider supports both kvm/lxc
> and also we have the ability to deploy both kvm/lxc in the same
> environment. Since we can not run lxc 'machines' along side kvm
> 'machines' we have to nest those containers within a kvm machine. with
> that said should juju treat a kvm machine any differently than a bare
> metal machine via maas? the cloning features intended for a local
> provider deploy should not be limited to the machine creation but also
> available to the containers within a machine.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1315216
>
> Title:
> lxc nested within kvm machines do not use lxc-clone
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1315216/+subscriptions
>

[Adam Stokes (adam-stokes)]

The issue with deploying containers/kvms to machine 0 is:

lxc's aren't reliably destroyed if destroy-environment is excuted
kvm's on machine 0 do not honor --constraints so it defaults to 512M and 8G disk sizes

So our approach is to bootstrap machine 0 with the local provider configured to use kvm as the default. Add machines with our defined constraints (usually enough to managed several lxc containers).

[Daniel Westervelt (danwest)]

sorry, I am a little confused here. Why is this so complicated? Why does the machine being a kvm make this any different?

[Adam Stokes (adam-stokes)]

Also in my testing I've noticed that with MaaS providers do not use the lxc-clone capabilities either. I realize this is intentional as stated in comment #2 but this would be a really great feature to have for any provider that support nested lxc's

[Ian Booth (wallyworld)]

The machine being kvm makes a big difference because the fast lxc feature was developed by Tim *only* for the local provider creating non-nested lxc containers. The requirement for it to work inside kvm is new to us. This is because of the technical requirements to actually make lxc cloning using btrfs possible. It's easy enough to set up on a local machine but there's a bunch more work required to make it work elsewhere. Here's a summary of what I needed to do:

I've done some investigating and have implemented a solution which can be
experimented with. The tl;dr; is:

1. New config option for all providers: "fast-lxc-clone"
Set to true to enable fast lxc cloning

eg
    hpcloud:
        type: openstack
        fast-lxc-clone: true
        default-series: precise

2. Machine cloud init for images on metal or kvm/lxc containers will install
btrfs-tools and qemu-utils. If fast lxc cloning is required, a btrfs image is
created and mounted at /var/lib/lxc. It is hard coded to 2G currently, which
should be enough to fit a lxc template and a few images.

ie

"qemu-img create -f raw /var/lib/juju/lxc 2G"
"/sbin/mkfs.btrfs /var/lib/juju/lxc"
"mkdir -p /var/lib/lxc"
"mount -t btrfs -o loop /var/lib/juju/lxc /var/lib/lxc"

3. When lxc containers are created, if fast clone is enabled, the functionality
works as for the local provider since now the necessary set up is done for all
machines and host containers.

I've pushed a branch to lp:~wallyworld/juju-core/fast-lxc-everywhere
It's a coding spike so there's no tests, but I've tested on HP Cloud to ensure
creating an LXC container on an instance works as expected. I can't create KVM
containers or don't currently have access to MAAS to check things there.

[Ian Booth (wallyworld)]

Some more work has been done on this issue by Andrew. If you want to try with MAAS, there's a new branch to pull from. The instructions are basically the same:

1. Pull from lp:~axwalk/juju-core/lp1315216-lxc-use-clone <---- note new branch
2. Set config attr in environment.yaml : lxc-use-clone: true <----- note changed config attr name
3. Use --upload-tools with bootstrap

This new code does not set up /var/lib/lxc to be explicitly mounted on a btrfs file system but will take advantage of it if is.

[Adam Stokes (adam-stokes)]

Added Trusty so we can track SRU when filed