tox-linters job failing with yaml.representer.RepresenterError

[Rabi Mishra]

When using to_yaml/to_nice_yaml filters in tripleo-ansible playbooks, ansible-lint fails with yaml.representer.RepresenterError[2]

Downgrading pyyaml to 3.13 fixes the issue when tested locally. On looking further it seems like a regression from https://github.com/yaml/pyyaml/commit/ef744d8609f47e8b70ce007ecfe6595d3c367174#diff-7187d694ecf380d9eca1386c4da05f6a

RepresenterError is just of type Exeption[1] and there is nothing to convert objects to string.

I've submitted a PR to revert that patch https://github.com/yaml/pyyaml/pull/429

[1] https://github.com/yaml/pyyaml/blob/master/lib3/yaml/error.py#L45-L46
[2]
2020-08-17 10:34:29.999552 | centos-8 | Traceback (most recent call last):
2020-08-17 10:34:29.999566 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/bin/ansible-lint", line 8, in <module>
2020-08-17 10:34:29.999580 | centos-8 | sys.exit(main())
2020-08-17 10:34:29.999594 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansiblelint/__main__.py", line 186, in main
2020-08-17 10:34:29.999609 | centos-8 | matches.extend(runner.run())
2020-08-17 10:34:29.999623 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansiblelint/__init__.py", line 267, in run
2020-08-17 10:34:29.999638 | centos-8 | for child in ansiblelint.utils.find_children(arg, self.playbook_dir):
2020-08-17 10:34:29.999664 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansiblelint/utils.py", line 172, in find_children
2020-08-17 10:34:29.999687 | centos-8 | for child in play_children(basedir, item, playbook[1], playbook_dir):
2020-08-17 10:34:29.999703 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansiblelint/utils.py", line 223, in play_children
2020-08-17 10:34:29.999718 | centos-8 | fail_on_undefined=False)
2020-08-17 10:34:29.999732 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansiblelint/utils.py", line 191, in template
2020-08-17 10:34:29.999746 | centos-8 | **dict(kwargs, fail_on_undefined=fail_on_undefined))
2020-08-17 10:34:29.999760 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansiblelint/utils.py", line 79, in ansible_template
2020-08-17 10:34:29.999775 | centos-8 | return templar.template(varname, **kwargs)
2020-08-17 10:34:29.999789 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansible/template/__init__.py", line 614, in template
2020-08-17 10:34:29.999808 | centos-8 | ) for v in variable]
2020-08-17 10:34:29.999822 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansible/template/__init__.py", line 614, in <listcomp>
2020-08-17 10:34:29.999837 | centos-8 | ) for v in variable]
2020-08-17 10:34:29.999864 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansible/template/__init__.py", line 626, in template
2020-08-17 10:34:29.999896 | centos-8 | disable_lookups=disable_lookups,
2020-08-17 10:34:29.999925 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansible/template/__init__.py", line 626, in template
2020-08-17 10:34:29.999942 | centos-8 | disable_lookups=disable_lookups,
2020-08-17 10:34:29.999965 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansible/template/__init__.py", line 581, in template
2020-08-17 10:34:29.999983 | centos-8 | disable_lookups=disable_lookups,
2020-08-17 10:34:29.999998 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansible/template/__init__.py", line 840, in do_template
2020-08-17 10:34:30.000014 | centos-8 | res = j2_concat(rf)
2020-08-17 10:34:30.000032 | centos-8 | File "<template>", line 13, in root
2020-08-17 10:34:30.000048 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/ansible/plugins/filter/core.py", line 72, in to_nice_yaml
2020-08-17 10:34:30.000064 | centos-8 | transformed = yaml.dump(a, Dumper=AnsibleDumper, indent=indent, allow_unicode=True, default_flow_style=False, **kw)
2020-08-17 10:34:30.000079 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/yaml/__init__.py", line 290, in dump
2020-08-17 10:34:30.000094 | centos-8 | return dump_all([data], stream, Dumper=Dumper, **kwds)
2020-08-17 10:34:30.000109 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all
2020-08-17 10:34:30.000125 | centos-8 | dumper.represent(data)
2020-08-17 10:34:30.000139 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent
2020-08-17 10:34:30.000155 | centos-8 | node = self.represent_data(data)
2020-08-17 10:34:30.000170 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data
2020-08-17 10:34:30.000185 | centos-8 | node = self.yaml_representers[None](self, data)
2020-08-17 10:34:30.000200 | centos-8 | File "/home/zuul/.cache/pre-commit/repo2p26tojf/py_env-python3.6/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined
2020-08-17 10:34:30.000216 | centos-8 | raise RepresenterError("cannot represent an object", data)
2020-08-17 10:34:30.000243 | centos-8 | yaml.representer.RepresenterError: ('cannot represent an object', AnsibleUndefined)
2020-08-17 10:34:30.000257 | centos-8 | [WARNING]: Falling back to Ansible unique filter as Jinja2 one failed: 'pool'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/747139

[Sorin Sbarnea (ssbarnea)]

Can someone show me an example of code that generates this error? This bug report is missing a simple reproducing case: running <command> on <file> produces this error.

Somehow the bug appears to be an exception from ansible-lint but it links to code from pyyaml, which is weird.

[Rabi Mishra (rabi)]

Sorin, please check https://review.opendev.org/#/c/746476/1/tripleo_ansible/playbooks/cli-container-image-prepare.yaml for a sample (it uses to_nice_yaml) and the failure of the tox-linters job in that patch.

https://f93e5d55100282bca79c-a1eedb38eefc88bab97a9b5aab34511e.ssl.cf2.rackcdn.com/746476/1/check/openstack-tox-linters/2bdbe86/job-output.txt

[Sorin Sbarnea (ssbarnea)]

I tried to build a reproducing file based on that but ansible fails to parse the file, so you can no longer blame ansible-lint or pyyaml for that.

```
- hosts: localhost
  vars:
    x: 123
  tasks:
    - name: one
      debug:
        msg: "{{ x | b64decode | from_yaml }}"
    - name: two
      debug:
        msg: "{{ x | to_nice_yaml }}"

```

Output:
```
Thursday 20 August 2020 15:39:48 +0100 (0:00:00.799) 0:00:00.830 *******
fatal: [localhost]: FAILED! => {"msg": "the field 'args' has an invalid value ({'msg': '{{ x | b64decode | from_yaml }}'}), and could not be converted to an dict.The error was: Incorrect paddingnnThe error appears to be in '/Users/ssbarnea/c/os/ansible-lint/foo.yml': line 5, column 7, but maynbe elsewhere in the file depending on the exact syntax problem.nnThe offending line appears to be:nn tasks:n - name: onen ^ heren"}
```

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747139
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=9404c783b497a2fd36480ab6f5abf5cdf889ef0d
Submitter: Zuul
Branch: master

commit 9404c783b497a2fd36480ab6f5abf5cdf889ef0d
Author: Rabi Mishra <email address hidden>
Date: Thu Aug 20 15:49:04 2020 +0530

    Use latest ansible-lint

    we need fix[1] to bypass RepresenterError when using to_yaml filter.

    This patch also migrates to pycqa/flake8 hooks repo for missing flake8.

    Adds missing flake8 and ignores some newly added flake8 rules that would
    be fixed in subsequent patches.

    [1] https://github.com/ansible/ansible-lint/pull/518

    Closes-Bug: #1892056
    Change-Id: I40b5ef703a19f7b839ff6f263140ca3cf5c4dfd8