lxc rootfs and cache location should be configurable

[Shu Shen]

This bug requests support of configurable lxc-cache and container rootfs folders.

LXC rootfs uses significant amount of disk space, about 800MB per container upon creation. Currently there is no way in the playbooks to specify an alternative location of the rootfs other than the lxc-create default that located under /var/lib/lxc folder.

On hosts where the partition holds /var folder is relatively small and there are storage capacity else where, being able to specify the rootfs folder is the only way to deploy on these hosts.

The request for configurable lxc-cache comes as a consequence of configurable rootfs because of how the lxc-cache works - the lxc-cache is hard-linked to container rootfs which assumes both are on the same file system. lxc-create will fail should the rootfs be located on a different partition from the cache.

Note on the implementation:
The underlying lxc-create template lxc-ubuntu as used by OSAD hard-code the lxc-cache to /var/cache/lxc. Arguably a change shall be requested to lxc upstream for lxc-ubuntu to allow configurable cache location. But until then we need patch lxc-ubuntu in the lxc-host playbook.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/190048
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=de2dc69641d41f1a1e42dec4374f708634af55a9
Submitter: Jenkins
Branch: master

commit de2dc69641d41f1a1e42dec4374f708634af55a9
Author: Shu Shen <email address hidden>
Date: Wed Jun 10 05:40:56 2015 +0000

    Configurable lxc cache and rootfs path

    The rootfs of the containers use large disk space and is defaulted to
    /var/lib/lxc. It's desirable to configure an alternative rootfs directory on a
    partition with large enough free space to hold the rootfs.

    The rootfs of a container is hard-linked to the lxc cache when created by lxc
    templates. Therefore both have to reside on the same partition and the lxc
    cache shall be configurable as well as the rootfs directory.

    The lxc-ubuntu template is patched by lxc-host role to use the configured lxc
    cache folder. This patch should be removed once the updated upstream lxc with
    support of configuring the cache path goes mainstream.

    To confgiure these parameters, set the following variables in global_overrides
    of openstack_user_config.yml:
        - lxc_container_directory, defaults to /var/lib/lxc
        - lxc_container_cache_path, defaults to /var/cache/lxc

    Closes-Bug: #1463905
    Change-Id: I9ccf0c45e0a1875644750e0d70ce613265040176

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/192165

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/192165
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=e355dd3e99a4f0e51507565b201a1c2a26c816ae
Submitter: Jenkins
Branch: kilo

commit e355dd3e99a4f0e51507565b201a1c2a26c816ae
Author: Shu Shen <email address hidden>
Date: Wed Jun 10 05:40:56 2015 +0000

    Configurable lxc cache and rootfs path

    The rootfs of the containers use large disk space and is defaulted to
    /var/lib/lxc. It's desirable to configure an alternative rootfs directory on a
    partition with large enough free space to hold the rootfs.

    The rootfs of a container is hard-linked to the lxc cache when created by lxc
    templates. Therefore both have to reside on the same partition and the lxc
    cache shall be configurable as well as the rootfs directory.

    The lxc-ubuntu template is patched by lxc-host role to use the configured lxc
    cache folder. This patch should be removed once the updated upstream lxc with
    support of configuring the cache path goes mainstream.

    To confgiure these parameters, set the following variables in global_overrides
    of openstack_user_config.yml:
        - lxc_container_directory, defaults to /var/lib/lxc
        - lxc_container_cache_path, defaults to /var/cache/lxc

    Closes-Bug: #1463905
    Change-Id: I9ccf0c45e0a1875644750e0d70ce613265040176
    (cherry picked from commit de2dc69641d41f1a1e42dec4374f708634af55a9)

[Matt Thompson (mattt416)]

I've noticed something odd and believe it's a result of this commit. I'm not sure it's a bug, but I preferred the previous behaviour. Currently, a container's rootfs is in /var/lib/lxc/<container>. For example:

root@jenk-heat-350-node1:/var/lib/lxc# ls -al jenk-heat-350-node1_cinder_api_container-32e9101e
total 104
drwxr-xr-x 22 root root 4096 Jun 29 14:57 .
drwx------ 27 root root 4096 Jun 29 14:17 ..
drwxr-xr-x 2 root root 4096 Jun 29 14:22 bin
drwxr-xr-x 2 root root 4096 Apr 10 2014 boot
-rw-r--r-- 1 root root 1370 Jun 29 14:57 config
-rw-r--r-- 1 root root 1219 Jun 29 14:18 config.2015-06-29@14:18:45~
drwxr-xr-x 5 root root 4096 Jun 29 15:00 dev
drwxr-xr-x 77 root root 4096 Jun 29 15:00 etc
-rw-r--r-- 1 root root 512 Jun 29 14:18 eth1.ini
-rw-r--r-- 1 root root 515 Jun 29 14:18 eth2.ini
-rw-r--r-- 1 root root 0 Jun 29 14:17 fstab
drwxr-xr-x 3 root root 4096 Jun 29 14:17 home
drwxr-xr-x 14 root root 4096 Jun 29 14:58 lib
drwxr-xr-x 2 root root 4096 May 31 20:07 lib64
drwxr-xr-x 2 root root 4096 Jul 29 2014 media
drwxr-xr-x 2 root root 4096 Apr 10 2014 mnt
drwxr-xr-x 2 root root 4096 Jun 29 14:57 opt
drwxr-xr-x 2 root root 4096 Apr 10 2014 proc
drwx--S--- 6 root root 4096 Jun 29 14:58 root
drwxr-xr-x 2 root root 4096 Jun 29 14:16 rootfs
drwxr-xr-x 9 root root 4096 Jun 29 14:17 run
drwxr-xr-x 2 root root 4096 Jun 29 14:58 sbin
drwxr-xr-x 2 root root 4096 Jul 29 2014 srv
drwxr-xr-x 2 root root 4096 Mar 13 2014 sys
drwxrwxrwt 2 root root 4096 Jun 30 09:17 tmp
drwxr-xr-x 10 root root 4096 Jul 29 2014 usr
drwxr-xr-x 12 root root 4096 Jun 15 12:38 var
root@jenk-heat-350-node1:/var/lib/lxc#

Previously, the rootfs would have been in /var/lib/lxc/<container>/rootfs.

This new behaviour is quite messy as it's going to become difficult to determine what files are LXC configuration versus files within the rootfs.

Assuming this is a bug, I'll go ahead and create a new bug for it.

Comments? Thoughts?

[Matt Thompson (mattt416)]

Appears to be a bug, created https://bugs.launchpad.net/openstack-ansible/+bug/1470101 for this.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.11 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.12 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.