rhel-8, ansible-playbook-3 fails to import python3-jinja2 in tripleo deployment

[wes hayutin]

2019-07-18 20:44:22Z [standalone.AllNodesDeploySteps]: CREATE_COMPLETE Stack CREATE completed successfully
2019-07-18 20:44:22Z [standalone.AllNodesDeploySteps]: CREATE_COMPLETE state changed
2019-07-18 20:44:22Z [standalone]: CREATE_COMPLETE Stack CREATE completed successfully
Generating default ansible config file /home/zuul/ansible.cfg
** Running ansible deploy tasks **
Traceback (most recent call last):
  File "/bin/ansible-playbook-3", line 60, in <module>
    import ansible.constants as C
  File "/usr/lib/python3.6/site-packages/ansible/constants.py", line 12, in <module>
    from jinja2 import Template
ModuleNotFoundError: No module named 'jinja2'
Exception: Deployment failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1287, in _standalone_deploy
    raise exceptions.DeploymentError('Deployment failed')
tripleoclient.exceptions.DeploymentError: Deployment failed
None
Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
Not cleaning ansible directory /home/zuul/undercloud-ansible-wjjfv4ne
Install artifact is located at /home/zuul/undercloud-install-20190718204447.tar.bzip2

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment Failed!

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-fdyh5wgv

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment failed.
[zuul@dhcp-9-114-101 ~]$ python3
Python 3.6.8 (default, Jan 11 2019, 02:17:16)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jinja2 import Template
>>> dir(Template)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_from_namespace', '_get_default_module', '_get_default_module_async', 'debug_info', 'from_code', 'from_module_dict', 'generate', 'generate_async', 'get_corresponding_lineno', 'is_up_to_date', 'make_module', 'make_module_async', 'module', 'new_context', 'render', 'render_async', 'stream']
>>>

[zuul@dhcp-9-114-101 ~]$ rpm -qa | grep jinja2
python3-jinja2-2.10.1-2.el8_0.noarch
[zuul@dhcp-9-114-101 ~]$ rpm -qa | grep ansible
ansible-role-container-registry-1.0.1-0.20190625233542.54526cc.el8.noarch
ansible-role-chrony-0.0.1-0.20190510140854.494f3c4.el8.noarch
ansible-role-openstack-operations-0.0.1-0.20190531000407.b5debee.el8.noarch
tripleo-ansible-0.1.1-0.20190702230730.1f2a301.el8.noarch
ansible-role-tripleo-modify-image-1.0.1-0.20190621160641.1415b04.el8.noarch
ansible-2.8.2-1.el8ae.noarch
ansible-pacemaker-1.0.4-0.20190510134605.0e4d7c0.el8.noarch
ansible-role-atos-hsm-0.1.1-0.20190510140636.90bb6f6.el8.noarch
python3-heat-agent-ansible-1.9.0-0.20190625010502.a6b1984.el8.noarch
ceph-ansible-4.0.0-0.1.rc9.el8cp.noarch
ansible-tripleo-ipsec-9.1.1-0.20190514112459.ffe104c.el8.noarch
ansible-role-thales-hsm-0.2.1-0.20190510141742.8fe9600.el8.noarch

[wes hayutin (weshayutin)]

attempted w/ some internal updated rpms..

019-07-18 21:14:56Z [standalone.AllNodesDeploySteps.StandaloneExtraConfigPost]: CREATE_COMPLETE Stack CREATE completed successfully** Downloading standalone ansible.. **

2019-07-18 21:14:57Z [standalone.AllNodesDeploySteps.StandaloneExtraConfigPost]: CREATE_COMPLETE state changed
2019-07-18 21:14:57Z [standalone.AllNodesDeploySteps.StandalonePostConfig]: CREATE_IN_PROGRESS state changed
2019-07-18 21:14:57Z [standalone.AllNodesDeploySteps.StandalonePostConfig]: CREATE_COMPLETE state changed
2019-07-18 21:14:57Z [standalone.AllNodesDeploySteps]: CREATE_COMPLETE Stack CREATE completed successfully
2019-07-18 21:14:57Z [standalone.AllNodesDeploySteps]: CREATE_COMPLETE state changed
2019-07-18 21:14:57Z [standalone]: CREATE_COMPLETE Stack CREATE completed successfully
Generating default ansible config file /home/zuul/ansible.cfg
** Running ansible deploy tasks **
Traceback (most recent call last):
  File "/bin/ansible-playbook-3", line 60, in <module>
    import ansible.constants as C
  File "/usr/lib/python3.6/site-packages/ansible/constants.py", line 12, in <module>
    from jinja2 import Template
  File "/usr/lib/python3.6/site-packages/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 15, in <module>
    from jinja2 import nodes
  File "/usr/lib/python3.6/site-packages/jinja2/nodes.py", line 19, in <module>
    from jinja2.utils import Markup
  File "/usr/lib/python3.6/site-packages/jinja2/utils.py", line 647, in <module>
    from markupsafe import Markup, escape, soft_unicode
ModuleNotFoundError: No module named 'markupsafe'
Exception: Deployment failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1287, in _standalone_deploy
    raise exceptions.DeploymentError('Deployment failed')
tripleoclient.exceptions.DeploymentError: Deployment failed
None
Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
Not cleaning ansible directory /home/zuul/undercloud-ansible-je_v5hfh
Install artifact is located at /home/zuul/undercloud-install-20190718211523.tar.bzip2

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment Failed!

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-_kvfax0v

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment failed.
[zuul@dhcp-9-114-101 ~]$ rpm -qa | grep python3-jinja2
python3-jinja2-2.10-9.el8ost.noarch
[zuul@dhcp-9-114-101 ~]$ rpm -qa | grep ansible
ansible-role-container-registry-1.0.1-0.20190625233542.54526cc.el8.noarch
ansible-role-chrony-0.0.1-0.20190510140854.494f3c4.el8.noarch
ansible-role-openstack-operations-0.0.1-0.20190531000407.b5debee.el8.noarch
tripleo-ansible-0.1.1-0.20190702230730.1f2a301.el8.noarch
ansible-role-tripleo-modify-image-1.0.1-0.20190621160641.1415b04.el8.noarch
ansible-2.8.2-1.el8ae.noarch
ansible-pacemaker-1.0.4-0.20190510134605.0e4d7c0.el8.noarch
ansible-role-atos-hsm-0.1.1-0.20190510140636.90bb6f6.el8.noarch
python3-heat-agent-ansible-1.9.0-0.20190625010502.a6b1984.el8.noarch
ceph-ans...

[wes hayutin (weshayutin)]

[root@dhcp-9-114-101 ~]# python3
Python 3.6.8 (default, Jan 11 2019, 02:17:16)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from markupsafe import Markup, escape, soft_unicode
>>>
>>> dir(Markup)
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__html__', '__html_format__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__module__', '__mul__', '__ne__', '__new__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'escape', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'striptags', 'swapcase', 'title', 'translate', 'unescape', 'upper', 'zfill']
>>>

[wes hayutin (weshayutin)]

ansible-playbook-3 is using the following python interp

[root@dhcp-9-114-101 ~]# head -n 3 /usr/bin/ansible-playbook-3
#!/usr/libexec/platform-python -s

[wes hayutin (weshayutin)]

-s : don't add user site directory to sys.path; also PYTHONNOUSERSITE

[root@dhcp-9-114-101 ~]# /usr/libexec/platform-python -s
Python 3.6.8 (default, Jan 11 2019, 02:17:16)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jinja2 import Template
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 15, in <module>
    from jinja2 import nodes
  File "/usr/lib/python3.6/site-packages/jinja2/nodes.py", line 19, in <module>
    from jinja2.utils import Markup
  File "/usr/lib/python3.6/site-packages/jinja2/utils.py", line 647, in <module>
    from markupsafe import Markup, escape, soft_unicode
ModuleNotFoundError: No module named 'markupsafe'

[wes hayutin (weshayutin)]

So the issue is the -s flag in ansible-playbook-3

#!/usr/libexec/platform-python -s FAILS

#!/usr/libexec/platform-python WORKS

[Sorin Sbarnea (ssbarnea)]

I am afraid that the -s was added on purpose for preventing ansible from picking user site-packages instead of system ones. Is not the thing that happens accidentally.

I guess that was their way of assuring that ansible runs with what is shipped, but obviously this breaks any code that may install packages in user-dir, like quickstart if I remember well.

I think that this decision forces us to adopt the route of implicit-virtualenv. This is in fact the Pythoun foundation recommended way of deploying python applications, where each application is installed inside a virtualenv.

As I said last week, the public `ubi8/python-36` image is such an example, where the default user is not root (app=root) and has an implicit virtualenv.

Try it yourself:
```
+ docker run -u root -it registry.redhat.io/ubi8/python-36 /bin/bash
(app-root) python -c "import sys; print(sys.path)"
['', '/opt/app-root/lib64/python36.zip', '/opt/app-root/lib64/python3.6', '/opt/app-root/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6', '/usr/lib/python3.6', '/opt/app-root/lib/python3.6/site-packages']
(app-root) which python
/opt/app-root/bin/python
```

While I do like the approach, I found one potential issue with it: the app-root environmnet was install without using system site packages, meaning that even if you install some other rpm python packages, your application will not be able to find them. From a container distribution point of view, this seems bit weird.

PS. There is also a python-27 one.

[Sorin Sbarnea (ssbarnea)]

Also reported by another user at https://github.com/ansible/molecule/issues/2173

[Cédric Jeanneret (cjeanner)]

Isn't it "just" a missing package?
on a rhel8 with osp-15/Stein, python3:
[stack@undercloud ~]$ /usr/libexec/platform-python -s
Python 3.6.8 (default, Apr 3 2019, 17:26:03)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jinja2 import Template
>>>
[stack@undercloud ~]$ rpm -qa | grep jinja2
python3-jinja2-2.10.1-2.el8_0.noarch
[stack@undercloud ~]$ rpm -qa | grep markupsafe
python3-markupsafe-1.1.0-2.el8ost.x86_64
[stack@undercloud ~]$

Care to check/ensure you have that python3-markupsafe thingy?

[Cédric Jeanneret (cjeanner)]

Probably found out: I'm using platform-python-3.6.8-2.el8_0.x86_64 on my box, while the one with the fault is using platform-python-3.6.8-1.el8.x86_64

Else packages are identical.

Testing with updating.

[Cédric Jeanneret (cjeanner)]

python3-markupsafe was listed as "installed", but no file were present on disk:
ls: cannot access '/usr/lib64/python3.6/site-packages/markupsafe': No such file or directory

Running `yum reinstall python3-markupsafe' provided the missing files, and we can import things again, as expected.

I suspect some kind of virtual-env or something - fun part is "it's working without the -s".... ?

[wes hayutin (weshayutin)]

Got it working, should have started from a clean box

2019-07-19 14:28:21Z [standalone]: CREATE_COMPLETE Stack CREATE completed successfully
Generating default ansible config file /root/ansible.cfg
** Running ansible deploy tasks **

PLAY [Gather facts from undercloud] *********************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************
ok: [undercloud]