Timezone lookups failing for some installations

[Brent S Elmer]

I have version 2009g installed on AIX. US/Eastern is not in the timezone information. America/New_York is not either. The strange thing is that I have the same version installed on Linux and they are there. I installed both from downloading the tgz from the pytz web site. The python version is different on the two systems. I would like to be able to find something that works the same on both systems. Either US/Eastern or America/New_York.

Here is the AIX information:

$ python
Python 2.6 (r26:66714, Oct 28 2008, 16:33:58) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytz
>>> pytz.__version__
'2009g'
>>> 'US/Eastern' in pytz.common_timezones
False
>>> 'US/Eastern' in pytz.common_timezones_set
False
>>> 'US/Eastern' in pytz.all_timezones
False
>>> pytz.country_timezones('US')
['America/Kentucky/Louisville', 'America/Kentucky/Monticello', 'America/Indiana/Vincennes', 'America/Indiana/Winamac', 'America/Indiana/Marengo', 'America/Indiana/Petersburg', 'America/Indiana/Vevay', 'America/Indiana/Tell_City', 'America/Menominee', 'America/North_Dakota/Center', 'America/North_Dakota/New_Salem', 'America/Boise', 'America/Juneau', 'America/Yakutat', 'America/Nome']
>>> 'America/New_York' in pytz.all_timezones
False

Here is the Linux information:

brente@belmer:~$ python
Python 2.5.4 (r254:67916, Feb 17 2009, 20:16:45)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytz
>>> pytz.__version__
'2009g'
>>> 'US/Eastern' in pytz.common_timezones
True
>>> 'US/Eastern' in pytz.all_timezones
True
>>> pytz.country_timezones('US')
['America/New_York', 'America/Detroit', 'America/Kentucky/Louisville', 'America/Kentucky/Monticello', 'America/Indiana/Indianapolis', 'America/Indiana/Vincennes', 'America/Indiana/Winamac', 'America/Indiana/Marengo', 'America/Indiana/Petersburg', 'America/Indiana/Vevay', 'America/Chicago', 'America/Indiana/Tell_City', 'America/Indiana/Knox', 'America/Menominee', 'America/North_Dakota/Center', 'America/North_Dakota/New_Salem', 'America/Denver', 'America/Boise', 'America/Shiprock', 'America/Phoenix', 'America/Los_Angeles', 'America/Anchorage', 'America/Juneau', 'America/Yakutat', 'America/Nome', 'America/Adak', 'Pacific/Honolulu']
>>> 'America/New_York' in pytz.all_timezones
True

[Stuart Bishop (stub)]

The source timezone lists are hardcoded in pytz/__init__.py, and are identical on all installations. However, they are filtered at runtime to cope with missing timezone data files (a number of installations are modified to use the system installed timezone data rather than the files distributed with pytz). I think the difference you see could be caused by an incompatible version or a bug in the pkg_resources module, which is normally distributed as part of setuptools.

pkg_resources doesn't seem to contain a version identifier, so I'm not sure how to check what version is installed.

Upgrading pkg_resources to the latest version might solve the problem.

pkg_resources is needed if the timezone data files are inside a .egg. We can avoid using pkg_resources unless absolutely necessary by checking for the existence of the calculated data file path on disk first, and only attempting to load using pkg_resources.resource_stream if that file does not exist.

[Stuart Bishop (stub)]

If my guess is not correct, I'll need more information, in particular the actual setup.py line being used and if the zoneinfo/US/Eastern file exists on disk in the expected location.

[Brent S Elmer (webe3vt)]

On Wed, 2009-06-17 at 05:00 +0000, Stuart Bishop wrote:
> If my guess is not correct, I'll need more information, in particular
> the actual setup.py line being used and if the zoneinfo/US/Eastern file
> exists on disk in the expected location.
>
> ** Changed in: pytz
> Status: Confirmed => Fix Released
>

I did find out a difference between the Linux and AIX install. The
difference is that setuptools is not installed on AIX. I didn't see
anything in pytz that says it is required. I was trying to have setup
tools installed to see if that fixed the problem. I do not have an
admin authority on AIX so I have to beg to get things installed and
sometimes it takes a while. So, is setuptools required or will the new
version of pytz fix the problem by itself? And by "Fix Released", is
that the i version that is there now?

Thanks,
Brent

[Stuart Bishop (stub)]

On Wed, Jun 17, 2009 at 8:25 PM, Brent S Elmer<email address hidden> wrote:
> On Wed, 2009-06-17 at 05:00 +0000, Stuart Bishop wrote:
>> If my guess is not correct, I'll need more information, in particular
>> the actual setup.py line being used and if the zoneinfo/US/Eastern file
>> exists on disk in the expected location.
>>
>> ** Changed in: pytz
>>        Status: Confirmed => Fix Released
>>
>
> I did find out a difference between the Linux and AIX install.  The
> difference is that setuptools is not installed on AIX.  I didn't see
> anything in pytz that says it is required.  I was trying to have setup
> tools installed to see if that fixed the problem.  I do not have an
> admin authority on AIX so I have to beg to get things installed and
> sometimes it takes a while.  So, is setuptools required or will the new
> version of pytz fix the problem by itself?  And by "Fix Released", is
> that the i version that is there now?

setuptools should not be required. pytz 2009i (released today)
contains what I hope is the fix. I also happened to see something
similar today when using buildout to assemble a work environment, so
there may be something lurking still. Perhaps a distutils issue that
is avoided when setuptools is installed?

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Brent S Elmer (webe3vt)]

>
> setuptools should not be required. pytz 2009i (released today)
> contains what I hope is the fix. I also happened to see something
> similar today when using buildout to assemble a work environment, so
> there may be something lurking still. Perhaps a distutils issue that
> is avoided when setuptools is installed?
>
>
> --
> Stuart Bishop <email address hidden>
> http://www.stuartbishop.net/
>

It looks like it still doesn't work without setuptools.

>>> import pytz
>>> pytz.__version__
'2009i'
>>> 'US/Eastern' in pytz.common_timezones
False
>>> 'US/Eastern' in pytz.common_timezones_set
False
>>> 'US/Eastern' in pytz.all_timezones
False
>>> pytz.country_timezones('US')
['America/Kentucky/Louisville', 'America/Kentucky/Monticello',
'America/Indiana/Vincennes', 'America/Indiana/Winamac',
'America/Indiana/Marengo', 'America/Indiana/Petersburg',
'America/Indiana/Vevay', 'America/Indiana/Tell_City',
'America/Menominee', 'America/North_Dakota/Center',
'America/North_Dakota/New_Salem', 'America/Boise', 'America/Juneau',
'America/Yakutat', 'America/Nome']
>>> 'America/New_York' in pytz.all_timezones
False
>>>

Brent

[Brent S Elmer (webe3vt)]

On Thu, 2009-06-18 at 06:23 +0000, Stuart Bishop wrote:
> ** Changed in: pytz
> Status: Fix Released => Confirmed
>
It doesn't seem to work yet.

$python
Python 2.6 (r26:66714, Oct 28 2008, 16:33:58) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytz
>>> pytz.__version__
'2009j'
>>> 'US/Eastern' in pytz.common_timezones
False
>>> 'US/Eastern' in pytz.all_timezones
False
>>> pytz.country_timezones('US')
['America/Kentucky/Louisville', 'America/Kentucky/Monticello',
'America/Indiana/Vincennes', 'America/Indiana/Winamac',
'America/Indiana/Marengo', 'America/Indiana/Petersburg',
'America/Indiana/Vevay', 'America/Indiana/Tell_City',
'America/Menominee', 'America/North_Dakota/Center',
'America/North_Dakota/New_Salem', 'America/Boise', 'America/Juneau',
'America/Yakutat', 'America/Nome']

Brent

[Brent S Elmer (webe3vt)]

Is there any information on this yet? Is there going to be a solution or Is the only option to install setuptools?

Brent

[Stuart Bishop (stub)]

I have had no luck reproducing this.

[tpederson (ted-pederson)]

I perhaps have a related problem. The timezones are in the common dictionary, but some actual timezone files on my windows system are blank, so when I actually try to instantiate a timezone I get an exception from unpack.

[Brent S Elmer (webe3vt)]

I had the AIX admin install setuptools. Then he reinstalled pytz using easy_install:

$ easy_install --upgrade pytz
Searching for pytz
Reading http://pypi.python.org/simple/pytz/
Reading http://pytz.sourceforge.net
Reading http://www.stuartbishop.net/Software/pytz
Reading http://sourceforge.net/projects/pytz/
Reading http://sourceforge.net/project/showfiles.php?group_id=79122
Best match: pytz 2009l
Downloading http://pypi.python.org/packages/2.6/p/pytz/pytz-2009l-py2.6.egg#md5=3c46972ba8d0585b3d6e1d2abff20220
Processing pytz-2009l-py2.6.egg
Moving pytz-2009l-py2.6.egg to /afs/profit.pok.ibm.com/profit/utils/lib/python2.6/site-packages
Adding pytz 2009l to easy-install.pth file

Installed /afs/profit.pok.ibm.com/profit/utils/lib/python2.6/site-packages/pytz-2009l-py2.6.egg
Processing dependencies for pytz
Finished processing dependencies for pytz

US/Eastern is now working on AIX as can be seen here:

$ python
Python 2.6 (r26:66714, Oct 28 2008, 16:33:58) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytz
>>> pytz.__version__
'2009l'
>>> 'US/Eastern' in pytz.common_timezones
True
>>> 'US/Eastern' in pytz.all_timezones
True
>>> pytz.country_timezones('US')
['America/New_York', 'America/Detroit', 'America/Kentucky/Louisville', 'America/Kentucky/Monticello', 'America/Indiana/Indianapolis', 'America/Indiana/Vincennes', 'America/Indiana/Winamac', 'America/Indiana/Marengo', 'America/Indiana/Petersburg', 'America/Indiana/Vevay', 'America/Chicago', 'America/Indiana/Tell_City', 'America/Indiana/Knox', 'America/Menominee', 'America/North_Dakota/Center', 'America/North_Dakota/New_Salem', 'America/Denver', 'America/Boise', 'America/Shiprock', 'America/Phoenix', 'America/Los_Angeles', 'America/Anchorage', 'America/Juneau', 'America/Yakutat', 'America/Nome', 'America/Adak', 'Pacific/Honolulu']
>>> 'America/New_York' in pytz.all_timezones
True

I'm not sure if something was fixed in pytz 2009l or it works because setuptools was installed first. Given the thread so far, it looks like it worked because setuptools was used to install since the last message in the thread is that it couldn't be reproduced which means it probably wasn't fixed.

[Stuart Bishop (stub)]

I'm still unable to reproduce this. I have made clean installs of Python 2.4.6 and Python 2.6.2 from source and successfully installed pytz without setuptools available.

Setting this bug to incomplete as I can't see any way of progressing without more information. Perhaps this is a bug in earlier versions of distutils, or perhaps a change in pytz has worked around the issue?

[David M. Cotter (me-davecotter)]

i can repro this and will gladly work with you to figure it out

on Windows, install
ActivePython-2.4.3.12-win32-x86.msi
 (i know it's old, but our team is stuck on it for now)
also installed "setuptoops-0.6c11.win32-py2.4.exe"

these items are installed via downloading the tar.gz and running "> python setup.py install" (no eggs involved)
pytz2010h
python-dateutil-1.5

>>> tzi = pytz.timezone("America/Los_Angeles")
Traceback (most recent call last):
  File "<interactive input>", line 1, in ?
  File "build\bdist.win32\egg\pytz\__init__.py", line 155, in timezone
  File "build\bdist.win32\egg\pytz\tzfile.py", line 17, in build_tzinfo
error: unpack str size does not match format

[David M. Cotter (me-davecotter)]

I can certainly "confirm" this bug, so, i set it to "confirmed", hope that's ok.

[Mike Bayer (zzzeek)]

I can reproduce it, and even get an idea partially for what's going on, by running "easy_install" on the source .tar.gz of pytz. I placed a pdb inside of setup.py when the files are located, and made a new "dist" file. I verified that the "US" files are in fact in the .tar.gz I created. But when running easy_install, I get an output like this:

classics-MacBook-Pro:pytz-2010h classic$ easy_install dist/pytz-2010h.tar.gz
install_dir /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/
Processing pytz-2010h.tar.gz
Running pytz-2010h/setup.py -q bdist_egg --dist-dir /var/folders/bv/bvGj7UN02RWNJ++8ZRcrpU+++TI/-Tmp-/easy_install-hmAoDz/pytz-2010h/egg-dist-tmp-y3lhzl
> /private/var/folders/bv/bvGj7UN02RWNJ++8ZRcrpU+++TI/-Tmp-/easy_install-hmAoDz/pytz-2010h/setup.py(43)<module>()
-> basepath = dirpath.split(os.path.sep, 1)[1]
(Pdb) os.listdir(os.path.join('pytz', 'zoneinfo', 'US'))
[]

If I peek into the tmp directory, I can see that the "US" folder is blank, that's what I have above with pdb and its also what I see when I look in the tmp directory from the outside. Its very strange - as far as I can tell, the issue here is occurring when setuptools does whatever it does to untar pytz-2010h.tar.gz into the tmp directory - files are just lost, seemingly randomly. Seems like a setuptools bug, actually. (directories named "US" ? directories that are all capital letters ? some other weird thing ?)

[David M. Cotter (me-davecotter)]

can we get this bug assigned?

[Stuart Bishop (stub)]

Assigned to whom? It is looking like someone who can reproduce needs to chase this upstream with the setuptools maintainers.

[David M. Cotter (me-davecotter)]

Oh, uh, heh. I guess i thought that this project was managed, i thought Stuart might still be here?

[David M. Cotter (me-davecotter)]

Oh, wait, you are Stuart! Lol. hmm. i guess i don't know how this works. what do i need to do to help solve this?

[Mike Bayer (zzzeek)]

This bug is a lot of work, since a test case needs to be created that reproduces the issue faithfully, then you have to figure out if its a distutils or setuptools bug, or neither, which may require some source diving to figure out where the broken behavior actually occurs. It might be worth posting just to the Distribute bugtracker first since they are in a position of high responsiveness.

[Mike Bayer (zzzeek)]

FWIW the issue seems to disappear when using pip. Perhaps that's what people should do here since pip is supposed to "replace" easy_install.

[Aaron Lav (asl2)]

I can reproduce zzzeek's problem of the US/Eastern timezone being absent if I run easy_install on the tar.gz file. I think the underlying Distribute bug is https://bitbucket.org/tarek/distribute/issue/183/extracting-of-symlinks-looses-original. The original lines from the proposed patch https://bitbucket-assetroot.s3.amazonaws.com/tarek/distribute/20101013/183/tarfile-update.patch show that currently, tarfile members are only extracted if they're files or directories, not symlinks or hardlinks.

I just checked http://pypi.python.org/packages/source/p/pytz/pytz-2011e.tar.gz and US/Eastern, America/New_York, and posixrules are links to the same file.

My workaround is to teach chef to untar with tar first, then do setup.py install. Rebuilding the tarfile with --hard-dereference might also work.

[Simon Hibbs (simon-hibbs)]

I'm having what appears to be same problem with Python 2.5.4 and pytz-2012h on Windows Server 2008.

>>> print pytz.timezone('US/Eastern')
Traceback (most recent call last):
  File "<pyshell#43>", line 1, in <module>
    print pytz.timezone('US/Eastern')
  File "C:\Python25\lib\site-packages\pytz\__init__.py", line 181, in timezone
    _tzinfo_cache[zone] = build_tzinfo(zone, fp)
  File "C:\Python25\lib\site-packages\pytz\tzfile.py", line 30, in build_tzinfo
    typecnt, charcnt) = unpack(head_fmt, fp.read(head_size))
  File "C:\Python25\lib\struct.py", line 87, in unpack
    return o.unpack(s)
error: unpack requires a string argument of length 44

Of course on windows when you untar the package it can't create the symlinks so they end up as empty files.

[Dave Brondsema (dave-brondsema)]

I think Aaron Lav is on the right track. I've been facing this bug and then followed his suggestion of rebuilding the tarfile with dereferences like this: ` tar xvzf pytz-2012d.tar.gz; tar chvzf pytz-2012d.tar.gz pytz-2012d; easy_install pytz-2012d.tar.gz` This resulted in US/Eastern being present, but that doesn't seem to be a perfect solution because I see other oddness: len(pytz.country_timezones['US']) == 17 compared to a `pip` install which has 30.

[Stuart Bishop (stub)]

I'm breaking hard links as a workaround to the upstream bug (http://bugs.python.org/issue8876, and possibly others in the various other packaging tools). Even if it is fixed, it won't get backported to Python releases pytz still supports.