extra kernel module checks for kernel version only done on the deploy node

[Javeria Khan]

On the liberty branch in roles/openstack_hosts/defaults/main.yml the kernel version check for host kernel modules for newer modules such as br_netfilter is only done on the deploy node. This fails when the targets are still on older kernel versions and the playbooks try to load these modules.

The check needs to be extended to the target hosts.

[Jimmy McCrory (jimmy-mccrory)]

Could you check that all hosts' cached ansible facts have the correct ansible_kernel?
/etc/openstack_deploy/ansible_facts

[Jesse Pretorius (jesse-pretorius)]

ref: https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L31

[Jimmy McCrory (jimmy-mccrory)]

After further testing, I'm able to confirm that the kernel version check is done on each host and was able to reproduce this problem with an out of date cache.

###
Only 3.19 kernel on host3
###
TASK: [openstack_hosts | Ensure kernel module(s)] *****************************
<127.0.0.1> REMOTE_MODULE modprobe name=''
<192.168.250.21> REMOTE_MODULE modprobe name=''
<192.168.250.22> REMOTE_MODULE modprobe name=br_netfilter
changed: [localhost] => {"changed": true, "failed": false, "item": "", "name": "", "params": "", "state": "present"}
changed: [host3] => (item=br_netfilter) => {"changed": true, "failed": false, "item": "br_netfilter", "name": "br_netfilter", "params": "", "state": "present"}
changed: [host2] => {"changed": true, "failed": false, "item": "", "name": "", "params": "", "state": "present"}

###
After upgrading kernel to 3.19 on and rebooting host2
###
TASK: [openstack_hosts | Ensure kernel module(s)] *****************************
<127.0.0.1> REMOTE_MODULE modprobe name=''
<192.168.250.21> REMOTE_MODULE modprobe name=''
<192.168.250.22> REMOTE_MODULE modprobe name=br_netfilter
changed: [localhost] => {"changed": true, "failed": false, "item": "", "name": "", "params": "", "state": "present"}
changed: [host2] => {"changed": true, "failed": false, "item": "", "name": "", "params": "", "state": "present"}
ok: [host3] => (item=br_netfilter) => {"changed": false, "failed": false, "item": "br_netfilter", "name": "br_netfilter", "params": "", "state": "present"}

###
Removed cached facts and reran playbook
###
TASK: [openstack_hosts | Ensure kernel module(s)] *****************************
<127.0.0.1> REMOTE_MODULE modprobe name=''
<192.168.250.21> REMOTE_MODULE modprobe name=br_netfilter
<192.168.250.22> REMOTE_MODULE modprobe name=br_netfilter
ok: [host3] => (item=br_netfilter) => {"changed": false, "failed": false, "item": "br_netfilter", "name": "br_netfilter", "params": "", "state": "present"}
changed: [host2] => (item=br_netfilter) => {"changed": true, "failed": false, "item": "br_netfilter", "name": "br_netfilter", "params": "", "state": "present"}
changed: [localhost] => {"changed": true, "failed": false, "item": "", "name": "", "params": "", "state": "present"}

[Jesse Pretorius (jesse-pretorius)]

The fact cache can be cleared, if need be, by executing:

rm -rf /etc/openstack_deploy/ansible_facts/*

This bug has been marked as invalid as this isn't really a bug.

[Jesse Pretorius (jesse-pretorius)]

Switching to triaged for the purpose of adding a documentation entry in the Operations Guide.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275354

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/276034

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276034
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=1536ba1e9efb05f7664e3c2c284808488d5c453f
Submitter: Jenkins
Branch: master

commit 1536ba1e9efb05f7664e3c2c284808488d5c453f
Author: Jimmy McCrory <email address hidden>
Date: Wed Feb 3 20:16:19 2016 -0800

    Docs: Troubleshooting cached Ansible facts issues

    Add troubleshooting page describing cached facts and providing a command
    to remove the current cached facts in the event they become invalid.

    The ansible-playbook option `--flush-cache` is broken for a jsonfile
    based cache in Ansible 1.9.x. When OpenStack-Ansible moves to Ansible 2,
    that should replace the rm command documented here.

    Change-Id: I268a7e7c18cc7ed8986418d2a3232beb214341aa
    Related-Bug: #1535075

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275354
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=c83b6a20108a912379843d3e07c630fb16a6c3d4
Submitter: Jenkins
Branch: master

commit c83b6a20108a912379843d3e07c630fb16a6c3d4
Author: Michael Gugino <email address hidden>
Date: Tue Feb 2 12:52:35 2016 -0500

    Docs: encourage users to run apt-get dist-upgrade on target hosts

    Ubuntu 14.04 is currently provided in several point releases, most
    currently 14.04.3. It is possible that users may initially deploy
    roles on a previous point release of Ubuntu, and then later upgrade
    system packages. Most notably, br_netfilter is implemented in it's
    own kernel module in kernel versions >= 3.18. This results in ansible
    fact caches being out of date on subsequent runs, and has caused a few
    bugs to be filed along the way.

    This patch is to hopefully ensure that users are using the most recent
    version of packages when beginning with openstack-ansible.

    Change-Id: I1fd4f11fc2e4be4fa4335f63031f7b2bdb312d34
    Closes-Bug: #1535075

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.