Python 2 is retiring

[Darwin Wu]

Python 2 is retiring in thirty months. Calibre needs to convert to Python 3.

[Kovid Goyal (kovid)]

No, it doesn't. I am perfectly capable of maintaining python 2 myself.
Far less work than migrating the entire calibre codebase.

 status wontfix

[Sean (sean.brady)]

ROFL

I sincerely hope this is a joke.

[Andrew Rabert (nvllsvm)]

I strongly urge you to reconsider this. Unless others begin contributing to your fork of Python 2.7, you're probably going to miss a number of future security issues, make building and deployment more complex, and break compatibility with the Python packages Calibre depends upon.

I've started working on getting Calibre running under both Python 2 and 3. Are you open to patches provided they do not negatively impact anything?

[Eli Schwartz (eschwartz)]

Kovid has stated numerous times that any patches which work towards
python3 compatibility without hurting python2 functionality or
performance would be happily accepted. Oddly enough, no one has ever
taken him up on that, though a number of people have insisted it is
*very important* that he himself do that work.

See e.g. bug 1456642 or bug 1756458

No, past offers to donate the computing power to run the 2to3 tool for
low-quality, non-polyglot, python3-only results don't count as valid
contributions.

[Eli Schwartz (eschwartz)]

Also note that there is sure to be at least one of several different
organizations who will be more than happy to maintain Python 2 forks,
for example Red Hat and other distros whose commercial extended support
will continue to provide python2.

[Hellmark (spamtrap-hellmark)]

Eli, problem is, most of those commercial distros aren't really used on the desktop. Asking people to use RHEL isn't really feasible. Plus you'll have distros like Debian that don't accept bundled applications.

[Eli Schwartz (eschwartz)]

I have no clue what you're talking about. Why would Red Hat as an upstream source for python2 be any more objectionable than the Python Software Foundation?

You know, that Red Hat are the upstream developers for lots of software available on a wide variety of Linux distributions?

Anyway, this is quite beside the point. See https://github.com/kovidgoyal/calibre/pull/870 and followup Pull Requests, where someone has graciously put in a fair amount of effort towards supporting python3. You can now use python3 to try building calibre after exporting the environment variable "CALIBRE_PY3_PORT" to check out the progress -- so far, basic 2to3 tidying up has been done, and the compiled extensions should compile without errors on python3 -- anything more complex than that will need more work, but that's at least enough to play around with it and see what major fixes are needed.

[Thorsten Glaser (mirabilos)]

Debian is going to remove Python 2 after the buster release (which will happen within a couple of weeks), and that’s already stretching it (we had to ask several package maintainers to retain Python 2 support for the release). This means that all the libraries and then the interpreter itself will be removed peu à peu in a few weeks, and I’m sure *buntu will follow.

This means that software that’s not ported to py3k (admittedly a totally different programming language from Python 2, not just a new version) is basically dead, such as GNU mailman ☹

Bundling Python 2 with your software is also not going to be an option; distributions won’t ship it any more.

[darkom (darko-miletic)]

I do not understand you people. Author said he will not do it and you keep insisting. If you really care about project contribute to the project with real stuff not just whining.

[Eli Schwartz (eschwartz)]

Hi, mirabilos, I'm the Arch Linux distribution maintainer for calibre. I'm happy to continue packaging a system distribution of python2 if that is what it takes to get real, useful programs in the wild to continue working.

As I've mentioned in the comment immediately above yours, there are valid reasons to suspect that just because the PSF drops python2, does not mean python2 will not continue to be supported elsewhere. Python2 is already long since frozen except for security fixes and the odd bugfix, so as long as someone continues to work on security issues, it's not really technically correct to state that it cannot be used, like, at all.

RHEL 7 at least will support it until 2024, and it seems RHEL 8 will support it too: https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/ so that will give it another few years.

Ubuntu 18.04 supports python2.7 in main, also, and is not EOL until 2023; I'm not sure whether 20.04 intends to support it though.

I cannot find any plans or ETA for Gentoo administering last-rites to dev-lang/python:2.7.
I cannot find any plans or ETA for slackware the current version of slackware, which contains python2, nor plans to put it to pasture for the next stable release.

So I don't think I would go quite so far as to call calibre dead, based purely on what Debian does.

This is aside for the fact that it is not dead in Debian either, since Stretch will continue to support it until 2022, and if Debian Buster has python2 at all, which it seems it does, then it will be supported for some probably lengthy period of time there too. Buster missed the boat for removing python2 support.

[Eli Schwartz (eschwartz)]

On the topic of distributions not shipping calibre anymore, this will no doubt come as a relief to Kovid, since he has often complained that Debian breaks calibre by applying illegitimate patches to the source code, and his standard operating policy is to close bugs as invalid, tell the users that *only* his official prebuilt binaries are supported, and that if they wish for support they should first demonstrate that their issues can be reproduced using the official binaries.

So IMHO Debian does not really have a leg to stand on when it comes to calibre, as it is, historically, usually Debian users that have mysterious heisenbugs when using a distribution release of calibre rather than the recommended prebuilt binaries.

(As a distribution packager, I have submitted a number of bugs for distribution issues, which usually deal with the build process or PyQt5 compatibility issues, and, since I can demonstrate the causes, I also get fixes. So this is not a matter of refusing to support building from source, simply a matter of refusing to invest time in debugging why and how building from patched sources causes usability issues.)

(Also as a distribution packager, I've submitted a number of PRs to make calibre natively work with system python modules, rather than following the Debian route and unpredictably patching them, which is of course the leading cause of said heisenbugs. Currently on Debian's calibre, at *least* markdown's headerid extension will just raise stacktraces -- which a PR of mine fixed by adding markdown 3 support and devendoring it at the same time -- and mathjax is simply utterly broken on every level although that I will take the blame for, since it was my devendoring PR which caused the current Debian patches to no longer apply.)

[Eli Schwartz (eschwartz)]

On a completely different note, actions speak louder than words, which is why I've been spending not inconsiderable time incrementally porting calibre to polyglot code, in the hope of finally making that exact switch. Thank you for wasting both my time and Kovid's time by complaining about things which everyone knows about, and which as you correctly pointed out, requires a massive effort to port the entire (500K sloc) application over to a *new* programming language.

Have you, in fact, read the comment immediately above yours, where I point to the place where several people, me included, are working on this longterm effort? Or is it more important to use scare tactics that aren't even true, in order to pressure an upstream developer who doesn't care about having his software in your distro, to do something he has already repeatedly said he doesn't care about doing (but will accept patches for)?

I know what I'm doing about python2...

[Norbert Preining (preining)]

@Eli I just ready your comment from March about Debian Calibre packages
being dump. I consider this rather unfair a comment, based on experience
of the past, I guess. I have been maintaining Calibre in Debian since
August 2017, and the only changes to the Calibre packaging is that we use
msgpack and feedparser from the packaged versions. I have brought back
unrardll support and many more things.

I have also reported bugs that were fixed, and not because there were
bugs in only Debian.

I recommend rethinking your position concerning Debian.

Thanks

[Eli Schwartz (eschwartz)]

There are and have been numerous release-critical bugs in Debian's
calibre packaging. Your support for mathjax and markdown are both
*utterly* broken, due to downstream patching which is completely
incorrect. You include package dependencies that were dropped from
calibre *years* ago. You fail to include other package dependencies when
calibre's official list of module dependencies is updated. You don't run
the test suite, which would catch most or all of these errors. You
historically patched calibre to use incompatible system versions of
vendored code that was actually forked and extended with significant new
features that calibre relied on.

You disable major features of calibre, like the plugin system, claiming
that it is "unauthenticated and non-trusted" (the only conclusion I can
draw from here is that the Debian packagers consider anyone installing
*plugins* that haven't been pre-vetted by Debian, to be doing
untrustworthy things).

You apply pointless "privacy" patches to inert files which are only used
for maintenance of the https://calibre-ebook.com website, meaning your
packaging sources contain gunk that obscures what you are actually doing
and results in line noise every time you need to rebase a patch for
files that are not used. Your choice, I guess.

> we use msgpack and feedparser from the packaged versions

msgpack is a system dependency and always has been. I'm not even sure I
understand your meaning. Is there a non-packaged version that could have
otherwise been used?

feedparser is one of a number of things I personally worked to remove
from calibre upstream. It's now a system dependency for *any* distro
that builds calibre, without the need for some Debian-specific patch.
The reason is because I don't like vendored code either, but I refuse on
ethical grounds to devendor it myself, downstream, and therefore in
order to devendor it I must push those changes upstream and wait for
them to be merged and either released or backported. I've had a *lot* of
success.

(I guess I could crack the obvious openssl joke at this point.)