pytz only understands legacy 32bit transition times

[Paul Eggert]

When pytz/tzfile.py reads a TZif file, its build_tzinfo function parses only the file's version-1 data, which is limited to 32-bit timestamps that stop working after 2038. In 1995 the TZif file format was expanded to include 64-bit transition times and a POSIX-style TZ string for future timestamps, and tzfile.py needs to support that. Without that support, tzfile.py's build_tzinfo function mishandles some timestamps after 2038 on most platforms. Worse, build_tzinfo mishandles current timestamps on NetBSD 9.0 because NetBSD 9.0 uses the new 'zic -b slim' option that causes TZif files to omit 32-bit data deducible from the 64-bit data that code written since 1995 is supposed to be looking at instead.

[Paul Eggert (eggert-cs)]

I should have written that I see the same bug symptoms on Ubuntu 18.04.4.

[Brian Park (xparks)]

The stackoverflow question already has an answer for this. It's because pytz does not accurately handle the constructor for `datetime` with the`tzinfo` parameter. It will often give the wrong UTC offset. We need to use the `timezone.localize()` method instead.

This little wart in the pytz API confuses people frequently. I have to consult the pytz documentation every time I use it. The fact that it confuses Paul Eggert makes me feel a little better.

I don't fully understand why pytz has this limitation, I think it has something to do with the API exposed by the `datetime` class, and the way pytz caches the DST transitions. This article by the author of `dateutil` has a lot more info: https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html. The newer `dateutil` package does not have this problem.

By the way, both pytz and dateutil seems to have numerous inaccuracies with regards to the DST offset for various timezones (I recall that the negative -1:00 offset for Europe/Dublin is one of them). Both packages currently support only 32-bit transitions, until year 2038. In addition, dateutil has special problems for transitions in the year 2037, while pytz handles those properly. I have not filed bugs against either packages for the DST offset bugs: The pytz package seems to be in maintenance mode. The large backlog of bugs on the `dateutil` package indicates that the maintainer is too busy.

[Paul Eggert (eggert-cs)]

Part of the problem is that when pytz/tzfile.py reads a TZif file, its build_tzinfo function parses only the file's version-1 data, which is limited to 32-bit timestamps that stop working after 2038. In 1995 the TZif file format was expanded to include 64-bit transition times and a POSIX-style TZ string for future timestamps, and tzfile.py needs to support that. Without that support, tzfile.py's build_tzinfo function mishandles some timestamps after 2038 on most platforms. Worse, build_tzinfo mishandles current timestamps on NetBSD 9.0 because NetBSD 9.0 uses the new 'zic -b slim' option that causes TZif files to omit 32-bit data deducible from the 64-bit data that code written since 1995 is supposed to be looking at instead.

For more on this topic, please see the dateutil bug report here:

https://github.com/dateutil/dateutil/issues/1059

which dateutil's maintainer indicates is a bug that is reasonably high on his radar.

I suggest that the priority of this pytz bug be increased, as pytz is already broken for current timestamps on NetBSD 9.0 and this problem will start occurring on other distros as "zic -b slim" becomes more popular - see:

https://github.com/eggert/tz/commit/6ba6f2117b95eab345a7ed9159cef939e30c4cd3

[Stuart Bishop (stub)]

The (awful) API provided by pytz and the requirement for localize() is there for legacy reasons. At the time, it was the only way to get the Python datetime library to do what was needed to get cross-timezone, DST aware arithmetic working. The future is the new built-in support in Python 3.9, available as a backport to earlier Python versions at https://pypi.org/project/backports.zoneinfo/

[Stuart Bishop (stub)]

(I hope for pytz to become a compatibility layer for old code, wrapping the Python 3.9 zoneinfo code and allowing codebases to transition seamlessly)

[Stuart Bishop (stub)]

Further discussion at https://github.com/stub42/pytz/issues/48

(2020d needs to go out requiring 'fat' support, since it is already overdue)