hostvars breaks on Queens

[Logan V]

$ sudo ansible lsn-mc1023 -m debug -a msg="{{ hostvars['lsn-mc1023'] }}"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
lsn-mc1023 | FAILED! => {
    "msg": "the field 'args' has an invalid value (False), and could not be converted to an dict.The error was: 'lsn-mc1023' is not in list\nexception type: <class 'ValueError'>\nexception: 'lsn-mc1023' is not in list"
}

This can be reproduced by:
git clone --recursive https://github.com/continuouspw/openstack-ci-cloud
cd openstack-ci-cloud
cp -R openstack_deploy /etc
cd openstack-ansible
scripts/bootstrap-ansible
cd playbooks
ansible lsn-mc1023 -m debug -a msg="{{ hostvars['lsn-mc1023'] }}"

[Logan V (loganv)]

root@9e4755cadd13:~/openstack-ci-cloud/openstack-ansible/playbooks# ansible lsnmc1023 -m debug -a var=hostvars -vvvv
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml "
ansible 2.4.1.0
  config file = None
  configured module search path = ['/etc/ansible/roles/plugins/library']
  ansible python module location = /opt/ansible-runtime/lib/python3.5/site-packages/ansible
  executable location = /opt/ansible-runtime/bin/ansible
  python version = 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
No config file found; using defaults
setting up inventory plugins
Parsed /home/ubuntu/openstack-ci-cloud/openstack-ansible/inventory/dynamic_inventory.py inventory source with script plugin
 [WARNING]: Unable to parse /etc/openstack_deploy/inventory.ini as an inventory source

Loading callback plugin minimal of type stdout, v2.0 from /opt/ansible-runtime/lib/python3.5/site-packages/ansible/plugins/callback/__init__.py
META: ran handlers
container_name: "lsnmc1023"
physical_host: "lsnmc1023"
The full traceback is:
Traceback (most recent call last):
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 521, in _execute
    result = self._handler.run(task_vars=variables)
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/plugins/action/debug.py", line 57, in run
    results = self._templar.template(self._task.args['var'], convert_bare=True, fail_on_undefined=True, bare_deprecated=False)
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/template/__init__.py", line 450, in template
    disable_lookups=disable_lookups,
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/template/__init__.py", line 672, in do_template
    res = j2_concat(rf)
  File "<template>", line 11, in root
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/vars/hostvars.py", line 117, in __repr__
    out[host] = self.get(host)
  File "/opt/ansible-runtime/lib/python3.5/_collections_abc.py", line 595, in get
    return self[key]
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/vars/hostvars.py", line 91, in __getitem__
    self._cached_result[sha1_hash] = templar.template(data, fail_on_undefined=False, static_vars=STATIC_VARS)
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/template/__init__.py", line 494, in template
    disable_lookups=disable_lookups,
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/template/__init__.py", line 494, in template
    disable_lookups=disable_lookups,
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/template/__init__.py", line 494, in template
    disable_lookups=disable_lookups,
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/template/__init__.py", line 450, in template
    disable_lookups=disable_lookups,
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/template/__init__.py", line 672, in do_template
    res = j2_concat(rf)
  File "<template>", line 13, in root
  File "/opt/ansible-runtime/lib/python3.5/site-p...

[Jean-Philippe Evrard (jean-philippe-evrard)]

I can't seem to test it on my machine, because of credentials (no secrets, nor access to the remote machine). Running bootstrap aio and then copying the user variables, env.d, and group vars fail on the proxy part of the user_variables.yml, which is expected, so i removed that part of your user_variables file.

Anyway... With all that set, it proves the user_variables, env.d, groups_vars, are not causing this.
It's definitely something that must appear from the inventory/remote node.

I'd be curious to know if ansible localhost -m debug -a msg="{{ hostvars['lsn-mc1023'] }}" -vvvv has the same problem.

[Jean-Philippe Evrard (jean-philippe-evrard)]

Also it would be interesting to know why we're using python3 on the node.

[Jean-Philippe Evrard (jean-philippe-evrard)]

Could you also have a dump of your bootstrap-ansible script?

I have the impression we are doing something wrong with python3, and installing stuff that don't matter (with different requirements) using pip instead of pip3.

[Jean-Philippe Evrard (jean-philippe-evrard)]

And if possible the whole thing with an ANSIBLE_DEBUG=yes , please :)

[Jean-Philippe Evrard (jean-philippe-evrard)]

And also knowing ansible --version would be nice. I know, I am asking a lot!

[Logan V (loganv)]

>> I can't seem to test it on my machine, because of credentials (no secrets, nor access to the remote machine). Running bootstrap aio and then copying the user variables, env.d, and group vars fail on the proxy part of the user_variables.yml, which is expected, so i removed that part of your user_variables file.

ahould be able to clone repo, cp -R openstack_deploy /etc, rm /etc/openstack_deploy/user_secrets.yml... user_secrets is not needed to repro this and its the only vaulted file.

>> Anyway... With all that set, it proves the user_variables, env.d, groups_vars, are not causing this.
It's definitely something that must appear from the inventory/remote node.

yep.. this is reproducible without any of the user_* files present

>> I'd be curious to know if ansible localhost -m debug -a msg="{{ hostvars['lsn-mc1023'] }}" -vvvv has the same problem.

it does.
py3 queens env: http://paste.openstack.org/raw/700925/
py2 queens env: http://paste.openstack.org/raw/700927/

>> Also it would be interesting to know why we're using python3 on the node.

We've been using py3 for the ansible-runtime venv for most of the last 2 (pike and queens) cycles: https://github.com/openstack/openstack-ansible/commit/5b9d96e75a0b290bb6c5e42a4c419cca78090ee9#diff-30c3701b5527d757251851f9253bbb3c

>> Could you also have a dump of your bootstrap-ansible script?
>> I have the impression we are doing something wrong with python3, and installing stuff that don't matter (with different requirements) using pip instead of pip3.

Full py3 run: http://paste.openstack.org/show/700932/
Full py2 run: http://paste.openstack.org/show/700935/

[Logan V (loganv)]

Since the pastes at the end of my last reply got truncated.. here is the full output:

Full py3 run: https://paste.ubuntu.com/p/VwrRjH8M87/
Full py2 run: https://paste.ubuntu.com/p/mcsy92ZXcv/

[Damian Cikowski (dcdamien)]

I have a similar issue:

ansible infra1 -m debug -a msg="{{ hostvars['infra1'] }}" -vvvv
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables_.yml -e @/etc/openstack_deploy/user_variables.yml "
ansible 2.4.3.0
  config file = None
  configured module search path = ['/etc/ansible/roles/plugins/library']
  ansible python module location = /opt/ansible-runtime/lib/python3.5/site-packages/ansible
  executable location = /opt/ansible-runtime/bin/ansible
  python version = 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
No config file found; using defaults
setting up inventory plugins
Parsed /opt/openstack-ansible/inventory/dynamic_inventory.py inventory source with script plugin
Parsed /etc/openstack_deploy/inventory.ini inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /opt/ansible-runtime/lib/python3.5/site-packages/ansible/plugins/callback/minimal.py
META: ran handlers
infra1 | FAILED! => {
    "msg": "the field 'args' has an invalid value (15), and could not be converted to an dict.The error was: 'infra1' is not in list\nexception type: <class 'ValueError'>\nexception: 'infra1' is not in list"
}

Another try:

infra1 | FAILED! => {
    "msg": "the field 'args' has an invalid value ([]), and could not be converted to an dict.The error was: 'infra1' is not in list\nexception type: <class 'ValueError'>\nexception: 'infra1' is not in list"
}

[Logan V (loganv)]

@dcdamien: Do you have any environment override files in /etc/openstack_deploy/env.d/* ? Could you share them if you have any?

[Damian Cikowski (dcdamien)]

Yes, I have.

cinder.yml: http://paste.openstack.org/show/708044/
haproxy.yml: http://paste.openstack.org/show/708047/
neutron.yml: http://paste.openstack.org/show/708050/
santricity-proxy.yml: http://paste.openstack.org/show/708051/

[Damian Cikowski (dcdamien)]

@loganv I have disabled overrides in env.d but it still fail in the same way.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556588
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=ca8e8fc90abc08e9080a16d5b9fa3fc5a93c29bb
Submitter: Zuul
Branch: master

commit ca8e8fc90abc08e9080a16d5b9fa3fc5a93c29bb
Author: Jean-Philippe Evrard <email address hidden>
Date: Mon Mar 26 16:56:11 2018 +0000

    Apply haproxy vars to haproxy group instead of haproxy_all

    Any ansible command loads the group vars/host vars.
    If haproxy group isn't properly loaded with hosts, the whole
    group_vars/haproxy_all/keepalived.yml will fail to get an index
    for any host belonging to the group.

    Instead the index should search in the group in which the variable
    applies to, to ensure it's always the same kind of group.

    In a later commit, we should make sure the index is always done when
    group size is bigger than 0.

    Change-Id: Id489b3f337ea0db0be0008477c9e3afde3872004
    Closes-Bug: #1752932

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/556822

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556822
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=85714acedb50ea65d7e7684c127984c8dc56afe4
Submitter: Zuul
Branch: stable/queens

commit 85714acedb50ea65d7e7684c127984c8dc56afe4
Author: Jean-Philippe Evrard <email address hidden>
Date: Mon Mar 26 16:56:11 2018 +0000

    Apply haproxy vars to haproxy group instead of haproxy_all

    Any ansible command loads the group vars/host vars.
    If haproxy group isn't properly loaded with hosts, the whole
    group_vars/haproxy_all/keepalived.yml will fail to get an index
    for any host belonging to the group.

    Instead the index should search in the group in which the variable
    applies to, to ensure it's always the same kind of group.

    In a later commit, we should make sure the index is always done when
    group size is bigger than 0.

    Change-Id: Id489b3f337ea0db0be0008477c9e3afde3872004
    Closes-Bug: #1752932
    (cherry picked from commit ca8e8fc90abc08e9080a16d5b9fa3fc5a93c29bb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 17.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 18.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 18.0.0.0b2 development milestone.