python-wcmatch not compatible to python3.12

[Alexander Sitnik]

[ Impact ]

 * The code in wcmatch uses inner details of libraries hidden in python 3.12
 * That breaks several use cases like ansible-lint and many self tests

[racb] ansible-lint doesn't work at all. Calling it with no arguments, calling it with --help, and calling it with --version all fail with the same stack trace.

[ Test Plan ]

1. For the original report, run ansible-lint
   - without arguments you expect it to fail to load any file (that is ok, a total breakage failing to load wcmatch is not)

   - with --help you want to see the help to be shown

   - One can run ansible yaml file, with ansible-lint <file>, here are two one can use for that

a) With mistakes (1x fqcn[action-core] 1x yaml[truthy])
Result is
...
fqcn[action-core]: Use FQCN for builtin module actions (apt).
yaml[truthy]: Truthy value should be one of [false, true]
...

$ cat test.yaml
- name: Install apache2
  hosts: servers
  tasks:
    - name: Ensure apache2 is installed
      apt:
        name: apache2
        state: present
      become: yes

b) A file that makes lint happy, result is
Passed: 0 failure(s), 0 warning(s) on 1 files. Last profile that met the validation criteria was 'production'.

$ cat test2.yaml
- name: Install apache2
  hosts: servers
  tasks:
    - name: Ensure apache2 is installed
      ansible.builtin.apt:
        name: apache2
        state: present
      become: true

2. For a wider coverage I recommend running the full self-tests

$ apt install dpkg-dev
# add deb-src to entries in /etc/apt/sources.list.d/ubuntu.sources
$ apt update
$ apt source python-wcmatch
$ cd python-wcmatch-8.*
$ pytestdetailed instructions how to reproduce the bug

[ Where problems could occur ]

 * wcmatch has multiple subelements, only pathlib use cases got changed
 * There are untested changes to the windows code paths, but even WSL will run
   the linux paths, so that should be a no-op

[ Other Info ]

 * while in universe, this is just an easy fix and an oversight in
   the python transition hence I think we should fix it

---
original report

Running ansible-lint in Ubuntu 24.04 on my role ends up with a stack trace:

Traceback (most recent call last):
  File "/usr/bin/ansible-lint", line 5, in <module>
    from ansiblelint.__main__ import _run_cli_entrypoint
  File "/usr/lib/python3/dist-packages/ansiblelint/__main__.py", line 39, in <module>
    from ansiblelint import cli
  File "/usr/lib/python3/dist-packages/ansiblelint/cli.py", line 20, in <module>
    from ansiblelint.file_utils import (
  File "/usr/lib/python3/dist-packages/ansiblelint/file_utils.py", line 15, in <module>
    import wcmatch.pathlib
  File "/usr/lib/python3/dist-packages/wcmatch/pathlib.py", line 218, in <module>
    class PurePosixPath(PurePath):
  File "/usr/lib/python3/dist-packages/wcmatch/pathlib.py", line 221, in PurePosixPath
    _flavour = pathlib._posix_flavour # type: ignore[attr-defined]
               ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pathlib' has no attribute '_posix_flavour'

This is due to python3-wcmatch=8.4-2 being incompatible with python3.12. Upstream fixed this in the next version https://github.com/facelessuser/wcmatch/releases/tag/8.5

I think the following portion of changes can be added to library package for Noble release:
https://github.com/facelessuser/wcmatch/pull/203/files#diff-081b2096c270ab6b17a6c70f95ea07d24dd0d343522e135dbdeb609e59867186L83

Versions of installed packages:

ansible-lint:
  Installed: 6.17.2-1
  Candidate: 6.17.2-1
  Version table:
 *** 6.17.2-1 500
        500 http://ru.archive.ubuntu.com/ubuntu noble/universe amd64 Packages
        500 http://ru.archive.ubuntu.com/ubuntu noble/universe i386 Packages
        100 /var/lib/dpkg/status

python3-wcmatch:
  Installed: 8.4-2
  Candidate: 8.4-2
  Version table:
 *** 8.4-2 500
        500 http://ru.archive.ubuntu.com/ubuntu noble/universe amd64 Packages
        500 http://ru.archive.ubuntu.com/ubuntu noble/universe i386 Packages
        100 /var/lib/dpkg/status

[Matthias Lüscher (m-luescher)]

The Python compatibility is actually fixed in the upstream wcmatch package:
https://github.com/facelessuser/wcmatch/releases/tag/8.5

(Update: Sorry - I did not see that this already got reported.)

[Alexander Sitnik (alex-sitnik)]

Matthias, yes I know. I've mentioned this in bug report.

Is there any chance that newer version of python3-wcmatch will reach Noble release?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sorin Sbarnea (ssbarnea)]

We are going to add https://github.com/ansible/ansible-lint/pull/4130 upstream to prevent accidental installation of incompatible dependencies. Feel free to switch to the newer version to avoid this issue.

Note that linter switched to CalVer, so don't be scared about the major version change (is not really breaking one).

There are no plans for backporting as we currently support only latest version (no maintenance branches).

[Sorin Sbarnea (ssbarnea)]

Basically the correct description for this bug is: "ubuntu 22.04 ships a broken python3-wcmatch" because it is known that the version shipped is incompatible with python 3.12, which happens to be the system python.

Once wcmatch is update to 8.5.0 or newer the linter will start working.

[Robie Basak (racb)]

Looks like the upstream changes to wcmatch/pathlib.py can be minimally backported. Something like:

_flavour = posixpath

and

_flavour = ntpath

could be used in PurePosixPath and PureWindowsPath respectively, together with their imports.

This needs testing though.

[Christian Ehrhardt  (paelzer)]

As reported this was just the pre python 3.12 version and not rebuilt since April 2023.
Thereby I agree to Sorin's suggestion of this not being ansible related at all.

Running the packages self-tests would have been enough to spot this in the first place,.

But this only had autodep8-python3 defined for autopkgtests.
Which oddly only ran ran import wcmatch; print(wcmatch)
And those are too trivial, they worked.
Due to that there was no signal of that fail when python3.12 migrated.
See https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/p/python-wcmatch/20240303_232528_d4ebb@/log.gz

One should mid term try to look at ensuring the real testsuite runs at autopkgtest time to avoid that from happening again (which it likely will given that the project explicitly enables new versions). There is already https://github.com/facelessuser/wcmatch/commit/09675ed9

But at least thereby manual testing this can be outlined well for the SRU at hand here:

$ apt install dpkg-dev
# add deb-src to entries in /etc/apt/sources.list.d/ubuntu.sources
$ apt update
$ apt source python-wcmatch
$ cd python-wcmatch-8.*
$ pytest

With that we also see that the helpful suggestion of a more minimal change in comment #6 will only lead to overcome this but fail in 57 other cases (down from 2 during collection and not even starting in depth).

According to git there was no later change to the python 3.12 compat changes in 223f9cab "Update to support Python 3.12 alpha (#203)". So a subset (no git workflow, no tox, no metadata) seems to be the best tradeoff for an SRU.

Noble with fix:
1165 passed, 150 skipped in 2.55s
Compared to Oracular (full 8.5.x)
1171 passed, 151 skipped, 54 warnings in 2.71s

I've done the backport for some minimal fuzz and a lot of release/workflow changes that are only noise in regard to the SRU. Applying that right to the source makes it work through all tests in noble.

Preparing a MP, the SRU template and PPA with that ...

[Christian Ehrhardt  (paelzer)]

MR: https://code.launchpad.net/~paelzer/ubuntu/+source/python-wcmatch/+git/python-wcmatch/+merge/469193
PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp-2060649-support-python-312

[Robie Basak (racb)]

> [Test Plan]
> 1. For the original report, run ansible-lint

Could someone expand on this please, to ensure that when we have a package available for testing, we are able to verify that ansible-lint works as expected in at least one normal case?

[Adric Norris (landstander668)]

After upgrading python3-wcmatch to version 8.4-2ubuntu0.1~nobleppa1 (from the lp-2060649-support-python-312 PPA referenced above), I'm now able to run ansible-lint successfully.

[Adric Norris (landstander668)]

To expand upon my previous comment, I tested the following scenarios:

* no command line arguments
* the `--help` option only
* linting of several playbooks (`ansible-lint playbook_name.yml`)

All of the above appear to be working normally at this point.

[Daniel von Obernitz (itzonban)]

I also tested the ppa version of python3-wcmatch and can confirm, that ansible-lint is now working as expected.

[Christian Ehrhardt  (paelzer)]

Thank you Adric and Daniel.
I'll upload this to Noble unapproved where it should hopefully be quickly accepted given the great pre-checks. I'll use your hints to extend on the test steps in the description ...

[Christian Ehrhardt  (paelzer)]

Done: Description updated and fix uploaded

[Robie Basak (racb)]

Hello Alexander, or anyone else affected,

Accepted python-wcmatch into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-wcmatch/8.4-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

Upgrade

root@n:~/python-wcmatch# apt install python3-wcmatch -t noble-proposed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libaio1 libsoup2.4-1 libsoup2.4-common libvpx8 python3-boto
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
  python3-wcmatch
1 upgraded, 0 newly installed, 0 to remove and 57 not upgraded.
Need to get 32.8 kB of archives.
After this operation, 1024 B disk space will be freed.
Get:1 http://archive.ubuntu.com/ubuntu noble-proposed/universe amd64 python3-wcmatch all 8.4-2ubuntu0.1 [32.8 kB]
Fetched 32.8 kB in 0s (87.1 kB/s)
(Reading database ... 134318 files and directories currently installed.)
Preparing to unpack .../python3-wcmatch_8.4-2ubuntu0.1_all.deb ...
Unpacking python3-wcmatch (8.4-2ubuntu0.1) over (8.4-2) ...
Setting up python3-wcmatch (8.4-2ubuntu0.1) ...
Scanning processes...
Scanning candidates...

Restarting services...
 systemctl restart wsdd-server.service

Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart networkd-dispatcher.service
 systemctl restart systemd-logind.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

----

root@n:~/python-wcmatch# ansible-lint test.yaml
WARNING Listing 2 violation(s) that are fatal
fqcn[action-core]: Use FQCN for builtin module actions (apt).
test.yaml:4 Use `ansible.builtin.apt` or `ansible.legacy.apt` instead.

yaml[truthy]: Truthy value should be one of [false, true]
test.yaml:8

Read documentation for instructions on how to ignore specific rule violations.

                 Rule Violation Summary
 count tag profile rule associated tags
     1 yaml[truthy] basic formatting, yaml
     1 fqcn[action-core] production formatting

Failed: 2 failure(s), 0 warning(s) on 1 files. Last profile that met the validation criteria was 'min'.

----

root@n:~/python-wcmatch# ansible-lint test2.yaml

Passed: 0 failure(s), 0 warning(s) on 1 files. Last profile that met the validation criteria was 'production'.

----

root@n:~/test# apt source python-wcmatch
Reading package lists... Done
NOTICE: 'python-wcmatch' packaging is maintained in the 'Git' version control system at:
https://salsa.debian.org/python-team/packages/python-wcmatch.git
Please use:
git clone https://salsa.debian.org/python-team/packages/python-wcmatch.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 123 kB of source archives.
Get:1 http://archive.ubuntu.com/ubuntu noble-proposed/universe python-wcmatch 8.4-2...

[Adric Norris (landstander668)]

I've enabled proposed updates and installed its python3-wcmatch package, and can also confirm that this resolves the ansible-lint issue.

Thanx everyone!

[Adric Norris (landstander668)]

Since I neglected to explicitly state the version above...

$ apt policy python3-wcmatch
python3-wcmatch:
  Installed: 8.4-2ubuntu0.1
  Candidate: 8.4-2ubuntu0.1
  Version table:
 *** 8.4-2ubuntu0.1 100
        100 http://us.archive.ubuntu.com/ubuntu noble-proposed/universe amd64 Packages
        100 http://us.archive.ubuntu.com/ubuntu noble-proposed/universe i386 Packages
        100 /var/lib/dpkg/status
     8.4-2 500
        500 http://us.archive.ubuntu.com/ubuntu noble/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu noble/universe i386 Packages

[Daniel von Obernitz (itzonban)]

Same here, the issue is solved using the python3-wcmatch package from noble-proposed.

Thanks a lot!

[Christian Ehrhardt  (paelzer)]

Thank you for cross checking again Daniel and Adric.
Given all your support in verifying I think we are all good now.

This will stay the mandatory aging period in -proposed and then should be good to be released fixing this for everyone.