4.11.0 release tarball is missing tests

[Michał Górny]

The bs4/tests directory is missing in 4.11.0 release tarball. This prevents us from packaging the new version for Gentoo.

[Leonard Richardson (leonardr)]

Thanks for filing this ticket. This is an interesting problem: the setup.py file hasn't changed, but now that I've dropped Python 2 support I'm using a much more recent version of setuptools which may have some new logic I didn't know about. There's an obvious thing in setup.py that could have been causing this, but removing that bit doesn't help. I'll look into this.

FYI, the test suite is now designed to use pytest, which may require a change to your packaging script.

[Michał Górny (mgorny)]

To be honest, I never managed to grasp the sdist logic in setuptools. I *think* it sometimes takes all files tracked by the VCS but I never really knew why, and I don't know if it applies to bzr as well.

I wanted to try figuring it out but apparently launchpad's "download tarball" 500s for me.

[Leonard Richardson (leonardr)]

I've cleaned things up and released 4.11.1, which I think you'll find has the bs4/tests directory.

The most fundamental underlying problem is that my release process didn't remove any old beautifulsoup4.egg-info directory. I think this means an old manifest file was sticking around when it shouldn't have been, possibly ensuring that the 4.10.0 sdist included test files when it otherwise wouldn't have. Once I fixed that, the obvious issue in setup.py, which I mentioned earlier, turned out to be the thing I was looking for.

However, the "obvious issue" was that setup.py had code explicitly excluding the tests/ directory from inclusion in sdist or wheel packages. This code has a history in the version control, of course: it dates to 2015 (https://bazaar.launchpad.net/~leonardr/beautifulsoup/bs4/revision/386) and the commit message "Change setup.py to focus on creating wheels." But I don't know what I was thinking back then, I don't see any reason why wheels can't or shouldn't include the unit tests, and older releases of Beautiful Soup have the tests in both the sdist and the wheel.