CountryNameDict function trying to parse UTF-8 iso3166.tab as US-ASCII

Bug #1473533 reported by Martin Jansa on 2015-07-10
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
pytz
Undecided
Stuart Bishop
python-tz (Debian)
Fix Released
Unknown
python-tz (Ubuntu)
High
Unassigned
Precise
Critical
Robert C Jennings
Trusty
Critical
Robert C Jennings
Vivid
Critical
Robert C Jennings

Bug Description

[Impact]

 * The latest tzdata update change the content type for iso3166.tab
   and zone.tab to UTF-8 which causes an exception in pytz

[Test Case]

$ apt-get install -y python-tz python3-tz

# These should produce exceptions for country_names. This test
# case also includes a test for country_timezones because the
# tzdata file for zones has changed to use utf-8 but has yet
# to include utf-8 characters. You can hand edit
# /usr/share/zoneinfo/zone.tab to include a UTF-8 character
# to force the exception and then test the proposed package.

$ python -c 'import pytz
for item in pytz.country_names.items():
  pass'

$ python -c 'import pytz
for item in pytz.country_timezones.items():
  pass'

$ python3 -c 'from pytz import country_timezones
for item in country_timezones.items():
  pass'

$ python3 -c 'from pytz import country_names
for item in country_names.items():
  pass'

# A recreate will raise an exception like this:
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/usr/lib/python3.4/_collections_abc.py", line 497, in __iter__
    for key in self._mapping:
  File "/usr/lib/python3/dist-packages/pytz/lazy.py", line 41, in __iter__
    self._fill()
  File "/usr/lib/python3/dist-packages/pytz/__init__.py", line 350, in _fill
    line = line.decode('US-ASCII')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

[Regression Potential]

 * Older tzdata releases should be okay (and this has been tested)

[Other Info]

 * None

---- Original Description ----

Since tzdata-2015e there are UTF-8 characters in iso3166.tab, see:
http://mm.icann.org/pipermail/tz/2015-May/022258.html
http://mm.icann.org/pipermail/tz/2015-June/022306.html

pytz/__init__.py:_CountryNameDict(LazyDict) is using:
        zone_tab = open_resource('iso3166.tab')
        try:
            for line in zone_tab.readlines():
                line = line.decode('US-ASCII')

to read it and fails on AX, CI, RE lines, using UTF-8 fixes the issues and should work OK even with older tzdata releases.

Martin Jansa (martin-jansa) wrote :
Andrew Bird (ajb-spheresystems) wrote :

I note that debian is also tracking this bug and they suggest that the latest version is fixed. Is it possible to get that fix into 12.04 LTS, 14.04 LTS and up please?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792192

Changed in pytz:
status: New → Confirmed
Stuart Bishop (stub) on 2015-09-21
Changed in pytz:
status: Confirmed → In Progress
assignee: nobody → Stuart Bishop (stub)
Stuart Bishop (stub) on 2015-09-21
Changed in pytz:
status: In Progress → Fix Released

The attachment "fix.for.tzdata-2015e.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Changed in python-tz (Debian):
status: Unknown → New
Changed in python-tz (Ubuntu):
importance: Undecided → High
status: New → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-tz - 2014.10~dfsg1-0ubuntu2

---------------
python-tz (2014.10~dfsg1-0ubuntu2) wily; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Sebastien Bacher <email address hidden> Tue, 29 Sep 2015 15:08:16 +0200

Changed in python-tz (Ubuntu):
status: Fix Committed → Fix Released
Robert C Jennings (rcj) on 2015-10-02
description: updated
Changed in python-tz (Ubuntu Precise):
assignee: nobody → Robert C Jennings (rcj)
Changed in python-tz (Ubuntu Trusty):
assignee: nobody → Robert C Jennings (rcj)
Changed in python-tz (Ubuntu Vivid):
assignee: nobody → Robert C Jennings (rcj)
Robert C Jennings (rcj) wrote :

I have attached the debdiffs for the 3 SRUs and have them uploaded to ppa:rcj/test as well. I have tested with the packages from that PPA. Just need upload review.

Changed in python-tz (Ubuntu Precise):
status: New → In Progress
Changed in python-tz (Ubuntu Trusty):
status: New → In Progress
Changed in python-tz (Ubuntu Vivid):
status: New → In Progress
Changed in python-tz (Ubuntu Precise):
importance: Undecided → High
Changed in python-tz (Ubuntu Trusty):
importance: Undecided → High
Changed in python-tz (Ubuntu Vivid):
importance: Undecided → Critical
Changed in python-tz (Ubuntu Trusty):
importance: High → Critical
Changed in python-tz (Ubuntu Precise):
importance: High → Critical
Robert C Jennings (rcj) wrote :
Robert C Jennings (rcj) wrote :
Robert C Jennings (rcj) wrote :

Updated patches with dep-3 tags

Robert C Jennings (rcj) wrote :
Chris J Arges (arges) wrote :

Sponsored for T/U/V... not sure why I can't unsub sponsors, but somebody that can please do.

Hello Martin, or anyone else affected,

Accepted python-tz into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2012c-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-tz (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Steve Langasek (vorlon) wrote :

Hello Martin, or anyone else affected,

Accepted python-tz into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2011k-0ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-tz (Ubuntu Precise):
status: In Progress → Fix Committed
Martin Jansa (martin-jansa) wrote :

I wasn't affected by this bug in Ubuntu environment, I'm using my own distribution built with OpenEmbedded, I've only reported it here as "upstream" python-tz.

The diff applied here is the same as what I'm using (only slightly extended to cover zone.tab as well), so I believe the bug is fixed in Ubuntu as well.

Steve Langasek (vorlon) wrote :

Hello Martin, or anyone else affected,

Accepted python-tz into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2014.10~dfsg1-0ubuntu2~15.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-tz (Ubuntu Vivid):
status: In Progress → Fix Committed
Robert C Jennings (rcj) wrote :

I have verified python-tz in precise, trusty, and vivid from -proposed using the testcase in the description.
I have verified python3-tz in trusty and vivid from -proposed using the testcase in the description.

tags: added: verification-done-precise verification-done-trusty verification-done-vivid
removed: verification-needed
tags: added: verification-done
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-tz - 2014.10~dfsg1-0ubuntu2~15.04.0

---------------
python-tz (2014.10~dfsg1-0ubuntu2~15.04.0) vivid; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Robert C Jennings <email address hidden> Fri, 02 Oct 2015 09:30:53 -0500

Changed in python-tz (Ubuntu Vivid):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for python-tz has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-tz - 2012c-1ubuntu0.1

---------------
python-tz (2012c-1ubuntu0.1) trusty; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Robert C Jennings <email address hidden> Fri, 02 Oct 2015 09:38:53 -0500

Changed in python-tz (Ubuntu Trusty):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-tz - 2011k-0ubuntu5.1

---------------
python-tz (2011k-0ubuntu5.1) precise; urgency=medium

  * debian/patches/use_utf8_encoding.patch:
    - use utf8 encoding to be compatible with new tzdata versions
      (lp: #1473533)

 -- Robert C Jennings <email address hidden> Fri, 02 Oct 2015 09:43:15 -0500

Changed in python-tz (Ubuntu Precise):
status: Fix Committed → Fix Released
Changed in python-tz (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.