Bindmount for Glance

[Kevin Stevens]

Related to : https://bugs.launchpad.net/openstack-ansible/+bug/1403584

According to http://docs.openstack.org/developer/glance/cache.html, "when images are successfully returned from a call to GET /images/<IMAGE_ID>, the image cache automatically writes the image file to its cache, regardless of whether the resulting write would make the image cache’s size exceed the value of image_cache_max_size."

Because of this, we can still easily fill up the filesystem for the Glance containers at least until the next run of glance-cache-pruner or glance-cache-cleaner.

Instead of making the glance container root LV bigger, we should probably configure a bind mount for /var/lib/glance or something similar.

Also, if playbooks will not automatically resize containers, we should probably make note of this in the release notes on launchpad here? https://launchpad.net/openstack-ansible/+milestone/10.1.2

[Bjoern (bjoern-t)]

I think this is a generally good idea but we need to think of how that works with glance and NFS/Netapp as backend which already bind mounts /var/lib/glance/images onto a NFS shared directory

[Andy McCrae (andrew-mccrae)]

I'm not a fan of bind mounting for glance here, for the following reason.

The problem we're solving for here is "glance container runs out of space" - the proposed solution is essentially to push that up to the host, based on a premise that the host "won't run out of space ever". Since we're dealing with arbitrary sizes of disks here that are always going to be finite, it doesn't make sense to me as a solution over a planned solution of setting the disk size for the glance container to be larger (its configurable).

So that is to say, moving the issue from the container to the host creates larger potential issues, but still the same potential issue, and if it doesn't exist on the host, at what point is the disk large enough for it to not exist? We should set the container size to that and it won't exist for the container either.

If the host fills up the consequences are far worse than if a glance container fails - that is one of the benefits of containers.

[Ian Cordasco (icordasc)]

I think the *actual* bug here is not that we need a bindmount but instead that we need to set up a cron job to manage and prune the cache using glance's tooling.

[Ian Cordasco (icordasc)]

So it looks like we do have a glance-cache-manage cron job and we do run the role to install it in Juno.

I wonder if the solution is to actually shorten the period of time between jobs in cron. It's clear we don't want to make a bindmount, so we need to ensure the cache is pruned more often it seems.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/176365

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/176368

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/176370

[Kevin Stevens (kstev)]

Should we then advise users to never have an image larger than 12GB?

[Kevin Stevens (kstev)]

Or maybe we disable glance image cache by default?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: juno
Review: https://review.openstack.org/176464

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176464
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=1e27d6735b47107d81d3ef8bf5e7db25bf4f9259
Submitter: Jenkins
Branch: juno

commit 1e27d6735b47107d81d3ef8bf5e7db25bf4f9259
Author: Ian Cordasco <email address hidden>
Date: Wed Apr 22 15:18:32 2015 -0500

    Allow users to configure the Glance flavor in use

    Some users may wish to completely disable the Glance cache. More recent
    versions of os-ansible-deployment allow for this with the glance_flavor
    variable. We backport this to provide the same flexibility to users on
    Juno.

    Related-bug: 1439892
    Change-Id: Ibd4c7852ed79422bf3de3d3ca2a0818604363a02

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176365
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=83f0d59191dc2afe9cbfbb8bf255edd4134446a7
Submitter: Jenkins
Branch: master

commit 83f0d59191dc2afe9cbfbb8bf255edd4134446a7
Author: Ian Cordasco <email address hidden>
Date: Wed Apr 22 10:25:38 2015 -0500

    Run the glance cache cron jobs more frequently

    Currently, running every 24 hours is long enough to allow the cache to
    completely fill up the container's disk. By running the jobs every 6
    hours instead (4 times per day), we hope to eliminate the problem where
    the container will fill up.

    Closes-bug: 1439892
    Change-Id: Ie7311f70656447382c2403e79453fe88401efca1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176368
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=48f5398cf774c482001da2631abcfcb8cebbd211
Submitter: Jenkins
Branch: juno

commit 48f5398cf774c482001da2631abcfcb8cebbd211
Author: Ian Cordasco <email address hidden>
Date: Wed Apr 22 10:25:38 2015 -0500

    Run the glance cache cron jobs more frequently

    Currently, running every 24 hours is long enough to allow the cache to
    completely fill up the container's disk. By running the jobs every 6
    hours instead (4 times per day), we hope to eliminate the problem where
    the container will fill up.

    Conflicts:
     playbooks/roles/os_glance/tasks/glance_post_install.yml

    Closes-bug: 1439892
    Change-Id: Ie7311f70656447382c2403e79453fe88401efca1
    (cherry picked from commit 83f0d59191dc2afe9cbfbb8bf255edd4134446a7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176370
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=fe68c500561fe392723668c63db6433d6c4679bd
Submitter: Jenkins
Branch: kilo

commit fe68c500561fe392723668c63db6433d6c4679bd
Author: Ian Cordasco <email address hidden>
Date: Wed Apr 22 10:25:38 2015 -0500

    Run the glance cache cron jobs more frequently

    Currently, running every 24 hours is long enough to allow the cache to
    completely fill up the container's disk. By running the jobs every 6
    hours instead (4 times per day), we hope to eliminate the problem where
    the container will fill up.

    Closes-bug: 1439892
    Change-Id: Ie7311f70656447382c2403e79453fe88401efca1
    (cherry picked from commit 83f0d59191dc2afe9cbfbb8bf255edd4134446a7)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.