pytz dst() incorrectly handles Pacific/Apia day leap

[Ilya Novoselov]

Request for SRU for Lucid, Maverick, Natty and Oneiric
======================================================

1. Impact
- Incorrect calculation of DST changes
- Landscape failure to accept new users or edit current users' setting
- "Django timezones app "timezones.zones" module unimportable" (reported on comment #11)

2. Development fix
The issue was fixed upstream in revision 268:
http://bazaar.launchpad.net/~stub/pytz/devel/revision/268

That is the diff grabbed for this SRU.

The same diff is also applied to the python-tz package in the current ubuntu development version (precise).

3. Stable fix
The development fix applies cleanly to the stable releases.

4. Test case
Run the script below. If you get a backtrace, you are affected. If not, the bug is fixed in your environment:
#!/usr/bin/python
import pytz
from datetime import datetime
tz = pytz.timezone("Pacific/Apia")
d2 = datetime(2012,1,1)
local = tz.localize(d2)
local.dst()
print "All is good, NOT hit by bug #885163!"

5. Regression potential
Not sure, timezones can be tricky. I welcome comments from upstream for this section since I basically packaged their fix. The patch is minimal, and it fixed the problems we were having in our production servers.

Original bug description follows:
=================================
>>> from datetime import datetime
>>> import pytz
>>> tz = pytz.timezone('Pacific/Apia')
>>> d1 = datetime(2011, 11, 1)
>>> d2 = datetime(2012, 1, 1)
>>> tz.dst(d1)
datetime.timedelta(0, 3600)
>>> tz.dst(d2)
datetime.timedelta(1, 3600)

while utcoffset is correct:

>>> tz.utcoffset(d1)
datetime.timedelta(-1, 50400)
>>> tz.utcoffset(d2)
datetime.timedelta(0, 50400)

[Ilya Novoselov (ilya-novoselov)]

It affects python's datetime.dst method:

>>> local = tz.localize(d2)
>>> local.dst()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
ValueError: tzinfo.dst() returned 1440; must be in -1439 .. 1439

and strftime:
>>> local.strftime('%H:%M')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
ValueError: tzinfo.dst() returned 1440; must be in -1439 .. 1439

[Ilya Novoselov (ilya-novoselov)]

Simple dst %= 1440 should work for this case, I'm not sure about other

[Ilya Novoselov (ilya-novoselov)]

Workaround with test attached. I can't really understand what dst calculation does, but this fixes issue and all other tests pass.

[StarSteve (info-astroworld)]

I'm confused !?
I think this error has nothing to do with dst (Daylight Saving Time) transitions/changes, instead:

in tzData2011k: australasia
*** 532,590 ****
# From David Zuelke (2011-05-09):
# Subject: Samoa to move timezone from east to west of international date line
# ...
# From Mark Sim-Smith (2011-08-17):
# ...
# the essence is that at midnight 29 Dec (UTC-11 I suppose), Samoa
# changes from UTC-11 to UTC+13:

comment:
UTC-11 to UTC+13 = 24hours = 1440Minutes
datetime.timedelta does not alow Values > 1439 => ValueError: ... must be in -1439 .. 1439

see: http://docs.python.org/library/datetime.html?highlight=timedelta#datetime.timedelta
tzinfo.utcoffset(self, dt)
   Return offset of local time from UTC, in minutes east of UTC. If local time is west of UTC, this should be negative. Note that this is intended to be the total offset from UTC; for example, if a tzinfo object represents both time zone and DST adjustments, utcoffset() should return their sum. If the UTC offset isn’t known, return None. Else the value returned must be a timedelta object specifying a whole number of minutes in the range -1439 to 1439 inclusive (1440 = 24*60; the magnitude of the offset must be less than one day).

# ...
# ...
# Dateline Change skip Friday 30th Dec 2011
# Thursday 29th December 2011 23:59:59 Hours
# Saturday 31st December 2011 00:00:00 Hours

-stefan

[Ilya Novoselov (ilya-novoselov)]

Timedelta does indeed allow value outside this range. But datetime doesn't allow dst delta of more than 24 hours, and for a reason. This is not dst change, this is timezone transition, and pytz computes utcoffset right. But dst is computed incorrectly, yelding value of 25 hours, while dst should really be same before and after transition (1 hour).

[StarSteve (info-astroworld)]

I go with you, but still confused, maybe I'm dense:
search in \python2.x\lib\test\test_datetime.py for 1439 and you may understand why...
e.g.

*** 2031,...
def test_utc_offset_out_of_bounds(self):
def test_tzinfo_classes(self):

*** 2746,2752 ****
        # dst() at the edge.
        self.assertEqual(cls(1,1,1, tzinfo=DST(1439)).timetuple().tm_isdst, 1)
        self.assertEqual(cls(1,1,1, tzinfo=DST(-1439)).timetuple().tm_isdst, 1)

        # dst() out of range.
        self.assertRaises(ValueError, cls(1,1,1, tzinfo=DST(1440)).timetuple)
        self.assertRaises(ValueError, cls(1,1,1, tzinfo=DST(-1440)).timetuple)

if u can - plz explain...

[Ilya Novoselov (ilya-novoselov)]

These are just tests of the limits.

[Ilya Novoselov (ilya-novoselov)]

We'd probably never know about this bug if this limits didn't exist.

[Stuart Bishop (stub)]

The code that loads the compiled timezone datafile was setting the dst offset incorrectly for this case.

[Forest Bond (forest-bond)]

I have prepared a patched version for Lucid. It is available in the ~rapidrollout/testing PPA:

  https://launchpad.net/~rapidrollout/+archive/testing

Here is the relevant changeset:

  http://bazaar.launchpad.net/~rapidrollout/ubuntu/lucid/python-tz/rr/revision/22

This issue became more prominent for me today due to a recent time zone change:

  http://www.abc.net.au/news/2011-12-29/samoa-time-zone-jump/3751254/?site=melbourne

I think this deserves an SRU for Lucid, but the fix first has to be applied in Precise, Oneiric, and Natty. I don't have time for all that this morning. Can anyone else take a look?

Thanks,
Forest

[Forest Bond (forest-bond)]

Just to clarify, the recent time zone change can cause failures on production sites. In my case, our production web site was severely impacted because this issue made the Django timezones app "timezones.zones" module unimportable.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "pacificapia.diff" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Stuart Bishop (stub)]

'pacificapia.diff' isn't the correct patch, but http://bazaar.launchpad.net/~rapidrollout/ubuntu/lucid/python-tz/rr/revision/22 linked earlier by Forest is.

[Andreas Hasenack (ahasenack)]

Is someone preparing an SRU for this? If not, I can do it.

[Andreas Hasenack (ahasenack)]

If someone could upload 2011n to precise, that would help.

[Forest Bond (forest-bond)]

No SRU is being prepared as far as I am aware.

[Andreas Hasenack (ahasenack)]

Here is a practical test case:

#!/usr/bin/python
import pytz
from datetime import datetime
tz = pytz.timezone("Pacific/Apia")
d2 = datetime(2012,1,1)
local = tz.localize(d2)
local.dst()
print "All is good, NOT hit by bug #885163!"

If you get a backtrace, you are hit by the bug. I will use this case for the SRU process.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2011k-0ubuntu3

---------------
python-tz (2011k-0ubuntu3) precise; urgency=low

  [ Forest Bond ]
  * Add patch samoa-idl (LP: #885163).
 -- Andreas Hasenack <email address hidden> Fri, 06 Jan 2012 10:29:22 -0800

[Colin Watson (cjwatson)]

Hello Ilya, or anyone else affected,

Accepted python-tz into lucid-proposed, maverick-proposed, natty-proposed, and oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Andreas Hasenack (ahasenack)]

Verified on Lucid:

root@ubuntu:~# cat /etc/lsb-release |grep RELEASE
DISTRIB_RELEASE=10.04

root@ubuntu:~# dpkg-query -W -f='${Package}-${Version}\n' python-tz
python-tz-2010b-1

root@ubuntu:~# python ./test.py
Traceback (most recent call last):
  File "./test.py", line 7, in <module>
    local.dst()
ValueError: tzinfo.dst() returned 1440; must be in -1439 .. 1439

root@ubuntu:~# apt-get install python-tz
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libx86-1 libpciaccess0 vbetool libffi5 libhal1 hal libhal-storage1 python-dbus pm-utils smartdimmer radeontool hal-info python-gobject
Use 'apt-get autoremove' to remove them.
The following packages will be upgraded:
  python-tz
1 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.
Need to get 44.0kB of archives.
After this operation, 0B of additional disk space will be used.
Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid-proposed/universe python-tz 2010b-1ubuntu0.10.04.1 [44.0kB]
Fetched 44.0kB in 0s (760kB/s)
(Reading database ... 41916 files and directories currently installed.)
Preparing to replace python-tz 2010b-1 (using .../python-tz_2010b-1ubuntu0.10.04.1_all.deb) ...
Unpacking replacement python-tz ...
Setting up python-tz (2010b-1ubuntu0.10.04.1) ...

Processing triggers for python-central ...

root@ubuntu:~# python ./test.py
All is good, NOT hit by bug #885163!

root@ubuntu:~# dpkg-query -W -f='${Package}-${Version}\n' python-tz
python-tz-2010b-1ubuntu0.10.04.1
root@ubuntu:~#

[Andreas Hasenack (ahasenack)]

Here is a cloud-init file to ease verification. It will run the test script, showing the bug, upgrade python-tz to the version in proposed and run the test again.

Use it like this:

ec2-run-instances <ami> -g ssh -k yourkey -f test-case.cloud-init

after a while, ssh into it and cat /tmp/log

[Andreas Hasenack (ahasenack)]

Confirmation for maverick, using that cloud-init file:

Wed Jan 18 18:50:02 UTC 2012
Codename: maverick
python-tz-2010b-1
Traceback (most recent call last):
  File "/tmp/bug.py", line 6, in <module>
    local.dst()
ValueError: tzinfo.dst() returned 1440; must be in -1439 .. 1439
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be upgraded:
  python-tz
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 42.6kB of archives.
After this operation, 4096B disk space will be freed.
Get:1 http://archive.ubuntu.com/ubuntu/ maverick-proposed/universe python-tz all 2010b-1ubuntu0.10.10.1 [42.6kB]
dpkg-preconfigure: unable to re-open stdin:
Fetched 42.6kB in 0s (94.1kB/s)
(Reading database ... 23516 files and directories currently installed.)
Preparing to replace python-tz 2010b-1 (using .../python-tz_2010b-1ubuntu0.10.10.1_all.deb) ...
Unpacking replacement python-tz ...
Setting up python-tz (2010b-1ubuntu0.10.10.1) ...
Processing triggers for python-central ...
python-tz-2010b-1ubuntu0.10.10.1
All is good, NOT hit by bug #885163!
Done.

[Andreas Hasenack (ahasenack)]

Confirmation for natty, using that cloud-init file:

Wed Jan 18 18:49:45 UTC 2012
Codename: natty
python-tz-2010b-1
Traceback (most recent call last):
  File "/tmp/bug.py", line 6, in <module>
    local.dst()
ValueError: tzinfo.dst() returned 1440; must be in -1439 .. 1439
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be upgraded:
  python-tz
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 42.5 kB of archives.
After this operation, 4096 B disk space will be freed.
Get:1 http://archive.ubuntu.com/ubuntu/ natty-proposed/universe python-tz all 2010b-1ubuntu0.11.04.1 [42.5 kB]
dpkg-preconfigure: unable to re-open stdin:
Fetched 42.5 kB in 0s (92.2 kB/s)
(Reading database ... 25192 files and directories currently installed.)
Preparing to replace python-tz 2010b-1 (using .../python-tz_2010b-1ubuntu0.11.04.1_all.deb) ...
Unpacking replacement python-tz ...
Setting up python-tz (2010b-1ubuntu0.11.04.1) ...
Processing triggers for python-central ...
python-tz-2010b-1ubuntu0.11.04.1
All is good, NOT hit by bug #885163!
Done.

[Andreas Hasenack (ahasenack)]

Confirmation for oneiric, using that cloud-init file:

Wed Jan 18 18:49:04 UTC 2012
Codename: oneiric
python-tz-2010b-1ubuntu1
Traceback (most recent call last):
  File "/tmp/bug.py", line 6, in <module>
    local.dst()
ValueError: tzinfo.dst() returned 1440; must be in -1439 .. 1439
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be upgraded:
  python-tz
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 41.5 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ oneiric-proposed/universe python-tz all 2010b-1ubuntu1.1 [41.5 kB]
dpkg-preconfigure: unable to re-open stdin:
Fetched 41.5 kB in 0s (84.5 kB/s)
(Reading database ... 26902 files and directories currently installed.)
Preparing to replace python-tz 2010b-1ubuntu1 (using .../python-tz_2010b-1ubuntu1.1_all.deb) ...
Unpacking replacement python-tz ...
Setting up python-tz (2010b-1ubuntu1.1) ...
python-tz-2010b-1ubuntu1.1
All is good, NOT hit by bug #885163!
Done.

[Andreas Hasenack (ahasenack)]

Just for the sake of completeness, here is the confirmation for lucid again, using the cloud-init file:

Wed Jan 18 18:50:46 UTC 2012
Codename: lucid
python-tz-2010b-1
Traceback (most recent call last):
  File "/tmp/bug.py", line 6, in <module>
    local.dst()
ValueError: tzinfo.dst() returned 1440; must be in -1439 .. 1439
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be upgraded:
  python-tz
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 44.0kB of archives.
After this operation, 0B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ lucid-proposed/universe python-tz 2010b-1ubuntu0.10.04.1 [44.0kB]
dpkg-preconfigure: unable to re-open stdin:
Fetched 44.0kB in 0s (91.0kB/s)
(Reading database ... 24530 files and directories currently installed.)
Preparing to replace python-tz 2010b-1 (using .../python-tz_2010b-1ubuntu0.10.04.1_all.deb) ...
Unpacking replacement python-tz ...
Setting up python-tz (2010b-1ubuntu0.10.04.1) ...

Processing triggers for python-central ...
python-tz-2010b-1ubuntu0.10.04.1
All is good, NOT hit by bug #885163!
Done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2010b-1ubuntu0.10.04.1

---------------
python-tz (2010b-1ubuntu0.10.04.1) lucid-proposed; urgency=low

  [ Forest Bond ]
  * Add patch samoa-idl (LP: #885163).
 -- Andreas Hasenack <email address hidden> Mon, 09 Jan 2012 21:45:33 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2010b-1ubuntu0.10.10.1

---------------
python-tz (2010b-1ubuntu0.10.10.1) maverick-proposed; urgency=low

  [ Forest Bond ]
  * Add patch samoa-idl (LP: #885163).
 -- Andreas Hasenack <email address hidden> Mon, 09 Jan 2012 22:02:50 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2010b-1ubuntu0.11.04.1

---------------
python-tz (2010b-1ubuntu0.11.04.1) natty-proposed; urgency=low

  * Add patch samoa-idl (LP: #885163).
 -- Forest Bond <email address hidden> Mon, 09 Jan 2012 22:05:30 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2010b-1ubuntu1.1

---------------
python-tz (2010b-1ubuntu1.1) oneiric-proposed; urgency=low

  [ Forest Bond ]
  * Add patch samoa-idl (LP: #885163).
 -- Andreas Hasenack <email address hidden> Mon, 09 Jan 2012 22:09:43 +0200