kolla-ansible reconfigure fails when manila is enabled

[Bartłomiej Daca]

In multinode environment with manila component enabled when running "kolla-ansible reconfigure" the command will fail with message:
error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'

in step:
TASK: [manila | Remove the containers]

The manila-related inventory configuration:
[control]
gs0104
gs0105
gs0106
[storage]
ssd05
ssd06
[manila:children]
control
[manila-api:children]
manila
[manila-scheduler:children]
manila
[manila-share:children]
storage

The important ansible steps log:

TASK: [manila | Check the configs] ********************************************
skipping: [ssd06] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd06] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [ssd05] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd05] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0105] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0106] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0104] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [ssd06] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [ssd05] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0105] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0106] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0105] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0104] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0106] => (item={'group': 'manila-share', 'name': 'manila_share'})
skipping: [gs0104] => (item={'group': 'manila-share', 'name': 'manila_share'})

TASK: [manila | Containers config strategy] ***********************************
skipping: [ssd05] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd06] => (item={'group': 'manila-api', 'name': 'manila_api'})
skipping: [ssd05] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [ssd06] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0106] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0104] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [gs0105] => (item={'group': 'manila-api', 'name': 'manila_api'})
ok: [ssd06] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [ssd05] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0106] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
ok: [gs0104] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0106] => (item={'group': 'manila-share', 'name': 'manila_share'})
ok: [gs0105] => (item={'group': 'manila-scheduler', 'name': 'manila_scheduler'})
skipping: [gs0104] => (item={'group': 'manila-share', 'name': 'manila_share'})
skipping: [gs0105] => (item={'group': 'manila-share', 'name': 'manila_share'})

TASK: [manila | Remove the containers] ****************************************
skipping: [gs0104] => (item=[{'group': 'manila-api', 'name': 'manila_api'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_api'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-api', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_api', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.572993', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072470', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_api /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.500523'}])
skipping: [gs0105] => (item=[{'group': 'manila-api', 'name': 'manila_api'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_api'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-api', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_api', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.574359', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.070076', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_api /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.504283'}])
skipping: [gs0106] => (item=[{'group': 'manila-api', 'name': 'manila_api'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_api'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-api', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_api', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.574669', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-api', 'name': 'manila_api'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072411', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_api /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.502258'}])
fatal: [ssd05] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
fatal: [ssd06] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
skipping: [gs0104] => (item=[{'group': 'manila-scheduler', 'name': 'manila_scheduler'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_scheduler'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-scheduler', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_scheduler', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.793828', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072669', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_scheduler /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.721159'}])
skipping: [gs0105] => (item=[{'group': 'manila-scheduler', 'name': 'manila_scheduler'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_scheduler'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-scheduler', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_scheduler', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.794886', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.073903', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_scheduler /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.720983'}])
skipping: [gs0106] => (item=[{'group': 'manila-scheduler', 'name': 'manila_scheduler'}, {'KOLLA_BASE_DISTRO': 'centos', 'PS1': '$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ', 'KOLLA_INSTALL_TYPE': 'source', 'changed': False, 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'PATH': '/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'invocation': {'module_name': u'kolla_docker', 'module_complex_args': {'action': 'get_container_env', 'name': u'manila_scheduler'}, 'module_args': ''}, 'KOLLA_SERVICE_NAME': 'manila-scheduler', 'KOLLA_INSTALL_METATYPE': 'mixed'}, {'cmd': ['docker', 'exec', 'manila_scheduler', '/usr/local/bin/kolla_set_configs', '--check'], 'end': '2016-07-15 11:41:41.801423', 'stderr': 'INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:The config files are in the expected state', 'stdout': u'', 'item': {'group': 'manila-scheduler', 'name': 'manila_scheduler'}, 'changed': False, 'rc': 0, 'failed': False, 'warnings': [], 'delta': '0:00:00.072810', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'docker exec manila_scheduler /usr/local/bin/kolla_set_configs --check'}, 'stdout_lines': [], 'failed_when_result': False, 'start': '2016-07-15 11:41:41.728613'}])
fatal: [gs0104] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
fatal: [gs0105] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'
fatal: [gs0106] => error while evaluating conditional: config_strategy == "COPY_ONCE" or item[1]['KOLLA_CONFIG_STRATEGY'] == 'COPY_ONCE'

It looks like it fails because some items are skipped in steps "Check the configs" and "Containers config strategy". Because of this, the item[1] (container_envs.results) cannot have the 'KOLLA_CONFIG_STRATEGY' defined.
Why these items are skipped? Because part of the manila components are deployed on the controller ansible group, and the other part on the storage ansible group, which is ok I guess.

From the code analysis it seems that this is not only manila-related problem. The same problem may occur with other components, when someone will try to customize placement of the component "subelements". For example:
# Murano
[murano-api:children]
host001
[murano-engine:children]
host002

Affected components list:
* ceilometer
* glance
* heat
* ironic
* magnum
* mistral
* murano

Solution: change the order of "when" statements and bring "inventory_hostname in groups[item[0]['group']]" to the top.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/343604
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=e61f13d880efd95de8002854ad9e64526d34ab4b
Submitter: Jenkins
Branch: master

commit e61f13d880efd95de8002854ad9e64526d34ab4b
Author: Bartłomiej Daca <email address hidden>
Date: Mon Jul 18 13:51:11 2016 +0200

    Fix 'reconfigure' command of kolla

    Changed the order of the 'when' statements in "remove/restart
    containers" tasks. It will fix the reconfiguration problem when
    deploying different components on different hosts.

    Change-Id: Ibee9dd56b6128b664144deb1d9eb7ec32e39fd5c
    Closes-Bug: #1603943

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/344400
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=00d2856caf1b266301b0b9b978038c129cf53cc0
Submitter: Jenkins
Branch: stable/mitaka

commit 00d2856caf1b266301b0b9b978038c129cf53cc0
Author: Bartłomiej Daca <email address hidden>
Date: Mon Jul 18 13:51:11 2016 +0200

    Fix 'reconfigure' command of kolla

    Changed the order of the 'when' statements in "remove/restart
    containers" tasks. It will fix the reconfiguration problem when
    deploying different components on different hosts.

    Change-Id: Ibee9dd56b6128b664144deb1d9eb7ec32e39fd5c
    Closes-Bug: #1603943

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/348120

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/348120
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=7abd34284204bbbc10ac304c122280cfdea9b2e5
Submitter: Jenkins
Branch: stable/liberty

commit 7abd34284204bbbc10ac304c122280cfdea9b2e5
Author: Bartłomiej Daca <email address hidden>
Date: Mon Jul 18 13:51:11 2016 +0200

    Fix 'reconfigure' command of kolla

    Changed the order of the 'when' statements in "remove/restart
    containers" tasks. It will fix the reconfiguration problem when
    deploying different components on different hosts.

    Change-Id: Ibee9dd56b6128b664144deb1d9eb7ec32e39fd5c
    Closes-Bug: #1603943
    (cherry picked from commit 00d2856caf1b266301b0b9b978038c129cf53cc0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 3.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 2.0.3 release.