CountryNameDict function trying to parse UTF-8 iso3166.tab as US-ASCII

[Martin Jansa]

[Impact]

 * The latest tzdata update change the content type for iso3166.tab
   and zone.tab to UTF-8 which causes an exception in pytz

[Test Case]

$ apt-get install -y python-tz python3-tz

# These should produce exceptions for country_names. This test
# case also includes a test for country_timezones because the
# tzdata file for zones has changed to use utf-8 but has yet
# to include utf-8 characters. You can hand edit
# /usr/share/zoneinfo/zone.tab to include a UTF-8 character
# to force the exception and then test the proposed package.

$ python -c 'import pytz
for item in pytz.country_names.items():
  pass'

$ python -c 'import pytz
for item in pytz.country_timezones.items():
  pass'

$ python3 -c 'from pytz import country_timezones
for item in country_timezones.items():
  pass'

$ python3 -c 'from pytz import country_names
for item in country_names.items():
  pass'

# A recreate will raise an exception like this:
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/usr/lib/python3.4/_collections_abc.py", line 497, in __iter__
    for key in self._mapping:
  File "/usr/lib/python3/dist-packages/pytz/lazy.py", line 41, in __iter__
    self._fill()
  File "/usr/lib/python3/dist-packages/pytz/__init__.py", line 350, in _fill
    line = line.decode('US-ASCII')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

[Regression Potential]

 * Older tzdata releases should be okay (and this has been tested)

[Other Info]

 * None

---- Original Description ----

Since tzdata-2015e there are UTF-8 characters in iso3166.tab, see:
http://mm.icann.org/pipermail/tz/2015-May/022258.html
http://mm.icann.org/pipermail/tz/2015-June/022306.html

pytz/__init__.py:_CountryNameDict(LazyDict) is using:
        zone_tab = open_resource('iso3166.tab')
        try:
            for line in zone_tab.readlines():
                line = line.decode('US-ASCII')

to read it and fails on AX, CI, RE lines, using UTF-8 fixes the issues and should work OK even with older tzdata releases.

[Andrew Bird (ajb-spheresystems)]

I note that debian is also tracking this bug and they suggest that the latest version is fixed. Is it possible to get that fix into 12.04 LTS, 14.04 LTS and up please?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792192

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "fix.for.tzdata-2015e.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Sebastien Bacher (seb128)]

upstream commit on http://bazaar.launchpad.net/~stub/pytz/devel/revision/367

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2014.10~dfsg1-0ubuntu2

---------------
python-tz (2014.10~dfsg1-0ubuntu2) wily; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Sebastien Bacher <email address hidden> Tue, 29 Sep 2015 15:08:16 +0200

[Robert C Jennings (rcj)]

I have attached the debdiffs for the 3 SRUs and have them uploaded to ppa:rcj/test as well. I have tested with the packages from that PPA. Just need upload review.

[Robert C Jennings (rcj)]

Updated patches with dep-3 tags

[Chris J Arges (arges)]

Sponsored for T/U/V... not sure why I can't unsub sponsors, but somebody that can please do.

[Steve Langasek (vorlon)]

Hello Martin, or anyone else affected,

Accepted python-tz into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2012c-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Martin, or anyone else affected,

Accepted python-tz into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2011k-0ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Jansa (martin-jansa)]

I wasn't affected by this bug in Ubuntu environment, I'm using my own distribution built with OpenEmbedded, I've only reported it here as "upstream" python-tz.

The diff applied here is the same as what I'm using (only slightly extended to cover zone.tab as well), so I believe the bug is fixed in Ubuntu as well.

[Steve Langasek (vorlon)]

Hello Martin, or anyone else affected,

Accepted python-tz into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2014.10~dfsg1-0ubuntu2~15.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robert C Jennings (rcj)]

I have verified python-tz in precise, trusty, and vivid from -proposed using the testcase in the description.
I have verified python3-tz in trusty and vivid from -proposed using the testcase in the description.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2014.10~dfsg1-0ubuntu2~15.04.0

---------------
python-tz (2014.10~dfsg1-0ubuntu2~15.04.0) vivid; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Robert C Jennings <email address hidden> Fri, 02 Oct 2015 09:30:53 -0500

[Chris J Arges (arges)]

The verification of the Stable Release Update for python-tz has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2012c-1ubuntu0.1

---------------
python-tz (2012c-1ubuntu0.1) trusty; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Robert C Jennings <email address hidden> Fri, 02 Oct 2015 09:38:53 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tz - 2011k-0ubuntu5.1

---------------
python-tz (2011k-0ubuntu5.1) precise; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Robert C Jennings <email address hidden> Fri, 02 Oct 2015 09:43:15 -0500