unbound-client.yml failure depending on TERMCAP envvar content

[Francois Deppierraz]

It looks like ansible is trying to evaluate the content of environment variables (such as TERMCAP) as jinja template. This leads to failure when an environment variable contains special characters such as "|".

This issue happens in practice when openstack-ansible is run from inside a GNU Screen session.

TASK [Apply resolver sorting] *************************************************************************************************************************************************************************************
fatal: [infra1_keystone_container-44f47275]: FAILED! => {"msg": "The conditional check 'physical_host in hostvars['localhost']['unbound_physical_hosts']' failed. The error was: template error while templating string: unexpected '|'. String: SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:\\\n\t:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:\\\n\t:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:\\\n\t:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:\\\n\t:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:\\\n\t:li#56:co#211:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:\\\n\t:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:\\\n\t:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:\\\n\t:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:\\\n\t:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:\\\n\t:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:\\\n\t:me=\\E[m:ms:\\\n\t:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:\\\n\t:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:\\\n\t:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\\\n\t:po=\\E[5i:pf=\\E[4i:Km=\\E[M:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:\\\n\t:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:\\\n\t:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:\\\n\t:F3=\\E[1;2P:F4=\\E[1;2Q:F5=\\E[1;2R:F6=\\E[1;2S:\\\n\t:F7=\\E[15;2~:F8=\\E[17;2~:F9=\\E[18;2~:FA=\\E[19;2~:kb=:\\\n\t:K2=\\EOE:kB=\\E[Z:kF=\\E[1;2B:kR=\\E[1;2A:*4=\\E[3;2~:\\\n\t:*7=\\E[1;2F:#2=\\E[1;2H:#3=\\E[2;2~:#4=\\E[1;2D:%c=\\E[6;2~:\\\n\t:%e=\\E[5;2~:%i=\\E[1;2C:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:\\\n\t:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:\\\n\t:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:\n\nThe error appears to have been in '/opt/openstack-ansible/playbooks/common-tasks/unbound-clients.yml': line 18, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n#on the same physical_host\n- name: Apply resolver sorting\n ^ here\n"}

PLAY RECAP ********************************************************************************************************************************************************************************************************
infra1_keystone_container-44f47275 : ok=6 changed=0 unreachable=0 failed=1
localhost : ok=3 changed=0 unreachable=0 failed=0

[Mohammed Naser (mnaser)]

Suggested workaround by Francois:

"an easy workaround is simply deleting this variable from /etc/openstack_deploy/ansible_facts/localhost"

[Francois Deppierraz (francois-ctrlaltdel)]

In https://github.com/openstack/openstack-ansible/blob/1ab4e908f2cfd537171deff5dc3e237c3cf37dd7/inventory/host_vars/localhost/unbound.yml#L17, the unbound_physical_hosts is defined this way:

unbound_physical_hosts: >-
  {{ groups[resolvconf_resolver_group] |
    map('extract', hostvars, 'physical_host') |
    select('defined') |
    list
  }}

Because ansible seems to do lazy evaluation of jinja2 in variables, this one actually contains all the data from groups[resolvconf_resolver_group], including environment variables.

Not sure yet if there's an easy way to set unbound_physical_hosts without running into this issue.

[Francois Deppierraz (francois-ctrlaltdel)]

Oops, please disregard the previous comment.

Here's how this issue can be reproduced on an AIO deployment running on Ubuntu 16.04.

root@francois-osa-aio:/opt/openstack-ansible# git branch
* (HEAD detached at 18.1.0)
  master
  stable/queens
root@francois-osa-aio:/opt/openstack-ansible# openstack-ansible --version
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
ansible-playbook 2.5.10
  config file = None
  configured module search path = [u'/etc/ansible/roles/plugins/library', u'/etc/ansible/roles/ceph-ansible/library']
  ansible python module location = /opt/ansible-runtime/local/lib/python2.7/site-packages/ansible
  executable location = /opt/ansible-runtime/bin/ansible-playbook
  python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]

EXIT NOTICE [Playbook execution success] **************************************
===============================================================================
root@francois-osa-aio:/opt/openstack-ansible# cat /tmp/test.yaml
---
- hosts: localhost
  tasks:
    - debug:
        msg: "{{ hostvars['localhost'] }}"
root@francois-osa-aio:/opt/openstack-ansible# openstack-ansible /tmp/test.yaml
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
 [WARNING]: Unable to parse /etc/openstack_deploy/inventory.ini as an inventory source

PLAY [localhost] ****************************************************************************************

TASK [debug] ********************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: unexpected '|'. String: SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:\\\n\t:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:\\\n\t:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:\\\n\t:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:\\\n\t:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:\\\n\t:li#54:co#211:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:\\\n\t:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:\\\n\t:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:\\\n\t:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:\\\n\t:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:\\\n\t:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:\\\n\t:me=\\E[m:ms:\\\n\t:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:\\\n\t:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:\\\n\t:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\\\n\t:po=\\E[5i:pf=\\E[4i:Km=\\E[M:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:\\\n\t:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:\\\n\t:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:\\\n\t:F3=\\E[1;2P:F4=\\E[1;2Q:F5=\\E[1;2R:F6=\\E[1;2S:\\\n\t:F7=\\E[15;2~:F8=\\E[17;2~:F9=\\E[18;2~:FA=\\E[19;2~:kb=:\\\n\t:K2=\\EOE:kB=\\E[Z:kF=\\E[1;2B:kR=\\E[1;2A:*4=\\E[3;2~:\\\n\t:*7=\\E[1;2F:#2=\\E[1;2H:#3=\\E[2;2~:#4=\\E[1;2D:%c=\\E[6;2~:\\\n\t:%e=\\E[5;2~:%i=\\E[1;2C:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:\\\n\t:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:\\\n\t:kd=\\EOB:kr=\\EOC:k...

[Francois Deppierraz (francois-ctrlaltdel)]

Downgrading to ansible 2.4.6 also fixed this issue while keeping the cache activated, it looks like an upstream ansible bug.

[Francois Deppierraz (francois-ctrlaltdel)]

Upgrading to ansible version 2.6.0 also fixes this issue.

[Francois Deppierraz (francois-ctrlaltdel)]

Here's the upstream ansible bug: https://github.com/ansible/ansible/issues/33259.

Manually applying the patch from https://github.com/ansible/ansible/commit/dae737c8b714f2c6d28663ef0afeab10e6d3a667 worked for fixing this issue and I'm pretty sure that it could speed deployment up because of less unused jinja2 interpretation.

We discussed during this weekly meeting and the conclusion is that the safest way would be to get it patched in the next 2.5 point release and then have OSA use it.

[Francois Deppierraz (francois-ctrlaltdel)]

Here's how to patch ansible in an easy way:

root@os-deploy:~# (cd /opt/ansible-runtime/lib/python2.7/site-packages; curl -s https://github.com/ansible/ansible/commit/dae737c8b714f2c6d28663ef0afeab10e6d3a667.patch | patch -p2)
patching file ansible/parsing/yaml/dumper.py
patching file ansible/plugins/filter/core.py
patching file ansible/template/__init__.py
Hunk #2 succeeded at 358 (offset 2 lines).
Hunk #3 succeeded at 511 (offset 2 lines).
patching file ansible/vars/hostvars.py
root@os-deploy:~#

Unfortunately, getting this fixed in the next ansible 2.5 point release doesn't seem possible, see https://github.com/ansible/ansible/pull/50101

[Jonathan Rosser (jrosser)]

This is now fixed in the ansible version used by all currently supported releases of openstack-ansible