Plays fail to mount NFS glance store inside glance containers if local_path is set to /var/lib/glance/images

[Jacob Wagner]

When using glance_nfs_client config, the nfs store will fail to mount to the local_path variable if it is equal to "/var/lib/glance/images".

TASK: [os_glance | Create nfs shares local path] ******************************
ok: [568575-infra01_glance_container-abedf963] => (item={'local_path': '/var/lib/glance/images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})
ok: [569038-infra03_glance_container-b53f1ebd] => (item={'local_path': '/var/lib/glance/images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})
ok: [569037-infra02_glance_container-a7d423c5] => (item={'local_path': '/var/lib/glance/images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})

TASK: [os_glance | Glance mount nfs] ******************************************
failed: [568575-infra01_glance_container-abedf963] => (item={'local_path': '/var/lib/glance/images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'}) => {"failed": true, "item": {"local_path": "/var/lib/glance/images", "options": "_netdev,auto", "remote_path": "/glancefs0/", "server": "172.29.244.24", "type": "nfs"}}
msg: Error mounting /var/lib/glance/images:
failed: [569037-infra02_glance_container-a7d423c5] => (item={'local_path': '/var/lib/glance/images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'}) => {"failed": true, "item": {"local_path": "/var/lib/glance/images", "options": "_netdev,auto", "remote_path": "/glancefs0/", "server": "172.29.244.24", "type": "nfs"}}
msg: Error mounting /var/lib/glance/images:
failed: [569038-infra03_glance_container-b53f1ebd] => (item={'local_path': '/var/lib/glance/images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'}) => {"failed": true, "item": {"local_path": "/var/lib/glance/images", "options": "_netdev,auto", "remote_path": "/glancefs0/", "server": "172.29.244.24", "type": "nfs"}}
msg: Error mounting /var/lib/glance/images:

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/os-glance-install.retry

568575-infra01_glance_container-abedf963 : ok=40 changed=29 unreachable=0 failed=1
569037-infra02_glance_container-a7d423c5 : ok=40 changed=29 unreachable=0 failed=1
569038-infra03_glance_container-b53f1ebd : ok=40 changed=28 unreachable=0 failed=1

I believe this is due to the fact that when the glance_store user_variable is set to file, "/var/lib/glance/images" is bind mounted to the physical host before the nfs task is run, causing the mount issue.

lxc.mount.entry = /openstack/568575-infra01_glance_container-abedf963 var/lib/glance/images none bind 0 0

If you set the local_path variable to any other point, the plays pass, but the cinder-api.conf still has /var/lib/glance/images as the datadir for glance images.

glance_nfs_client:
  - server: "172.29.244.24" ## Hostname or IP address of NFS Server
    remote_path: "/glancefs0/" ## Remote path from the NFS server's export
    local_path: "/var/lib/glance/nfs_images" ## Local path on machine
    type: "nfs" ## This can be nfs or nfs4
    options: "_netdev,auto" ## Mount options

TASK: [os_glance | Create nfs shares local path] ******************************
changed: [568575-infra01_glance_container-abedf963] => (item={'local_path': '/var/lib/glance/nfs_images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})
changed: [569038-infra03_glance_container-b53f1ebd] => (item={'local_path': '/var/lib/glance/nfs_images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})
changed: [569037-infra02_glance_container-a7d423c5] => (item={'local_path': '/var/lib/glance/nfs_images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})

TASK: [os_glance | Glance mount nfs] ******************************************
changed: [568575-infra01_glance_container-abedf963] => (item={'local_path': '/var/lib/glance/nfs_images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})
changed: [569037-infra02_glance_container-a7d423c5] => (item={'local_path': '/var/lib/glance/nfs_images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})
changed: [569038-infra03_glance_container-b53f1ebd] => (item={'local_path': '/var/lib/glance/nfs_images', 'type': 'nfs', 'options': '_netdev,auto', 'remote_path': '/glancefs0/', 'server': '172.29.244.24'})

[glance_store]
default_store = file
stores = glance.store.filesystem.Store,glance.store.http.Store,glance.store.cinder.Store
filesystem_store_datadir = /var/lib/glance/images/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/237058

[Andy McCrae (andrew-mccrae)]

@swagner1104 - can you clarify on the cinder part?

Where is the cinder var for the glance image dir set that you referenced?

[Andy McCrae (andrew-mccrae)]

I've put a patch in so that the dir won't be bind mounted if you are not using file store, or if you have specified nfs clients.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237058
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=537912778d8c79ffe54930612077e2bea72de737
Submitter: Jenkins
Branch: master

commit 537912778d8c79ffe54930612077e2bea72de737
Author: Andy McCrae <email address hidden>
Date: Mon Oct 19 16:50:43 2015 +0100

    Only bind mount images dir /w local Glance storage

    /var/lib/images should not be bind mounted unless:
    The glance server is not on metal
    Is using local file storage
    Isn't using glance_nfs_client settings

    This isn't required for non-file backed glance servers, and will cause
    failures if /var/lib/images is bind mounted when using nfs_clients
    for glance.

    Closes-bug: #1503411
    Change-Id: Ie90f75652a960af664fcf8f29d554045f0599944

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237145
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=d3a8fc9cdb235ea0f7f1da9996007d0d945d7fc8
Submitter: Jenkins
Branch: kilo

commit d3a8fc9cdb235ea0f7f1da9996007d0d945d7fc8
Author: Andy McCrae <email address hidden>
Date: Mon Oct 19 16:50:43 2015 +0100

    Only bind mount images dir /w local Glance storage

    /var/lib/images should not be bind mounted unless:
    The glance server is not on metal
    Is using local file storage
    Isn't using glance_nfs_client settings

    This isn't required for non-file backed glance servers, and will cause
    failures if /var/lib/images is bind mounted when using nfs_clients
    for glance.

    Closes-bug: #1503411
    Change-Id: Ie90f75652a960af664fcf8f29d554045f0599944
    (cherry picked from commit 537912778d8c79ffe54930612077e2bea72de737)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.15 release.