buildout not building pytz egg correctly

[Stuart Bishop]

buildout fails to build the pytz egg correctly if provided with a tarball. It works fine if provided with eggs.

To demonstrate:
 - Update versions.cfg to change the pytz version to 2009n
 - Download pytz-2009n.tar.bz2 from http://pypi.python.org/pypi/pytz, putting the tarball in download-cache/dist
 - run make
 - run find eggs/pytz-2009n-py2.4.egg/pytz/zoneinfo | wc -l. There should 592 files and directories, but a lot will be missing. The missing files seem to be random (although consistent).
 - look at eggs/pytz-2009n-py2.4.egg/EGG-INFO/SOURCES.txt . Note that the missing files are missing here too.

Replacing the tarball with a egg (after trashing the corrupt egg and running 'make clean'), everything works fine.

The tarball builds and installs correctly with a clean source-built python 2.4 and 2.6.

[Gary Poster (gary)]

Hi Stuart. This behavior is usually indicative of the source package being broken. An easy way to verify this is to untar the source package and see if it has the files you expect. If it does not, there's the problem. This is often the case with packages from non-subversion version control systems that have data files. My usual solution in to write a MANIFEST.in, as in the lazr.* packages.

Could you verify that the sdist has the files you expect?

Thank you

Gary

[Stuart Bishop (stub)]

On Tue, Sep 29, 2009 at 7:14 PM, Gary Poster <email address hidden> wrote:
> Hi Stuart. This behavior is usually indicative of the source package
> being broken. An easy way to verify this is to untar the source package
> and see if it has the files you expect. If it does not, there's the
> problem. This is often the case with packages from non-subversion
> version control systems that have data files. My usual solution in to
> write a MANIFEST.in, as in the lazr.* packages.
>
> Could you verify that the sdist has the files you expect?

Yes - the sdist has the correct files. Also, I can use it to install pytz just fine using fresh compiled-from-sources Python 2.4 and 2.6. I have only seen the bad install generated by buildout. Bug #383171 is somewhat similar, but I have not been able to reproduce that.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Stuart Bishop (stub)]

I don't know if this really should be high priority - the work around is to use the .egg rather than the .tar.bzr.

[Gary Poster (gary)]

Good point, thanks.

[Gary Poster (gary)]

I looked into this tonight a bit, first by attempting to verify that pytz itself did not have a problem in the source tarball. Here are the steps to repeat my experiment.

sudo aptitude install virtualenv
virtualenv --no-site-packages pytztest
cd pytztest
wget http://pypi.python.org/packages/source/p/pytz/pytz-2009n.tar.gz
./bin/easy_install --always-unzip pytz-2009n.tar.gz
cd lib/python2.6/site-packages/pytz-2009n-py2.6.egg
find lib/python2.6/site-packages/pytz-2009n-py2.6.egg/pytz/zoneinfo | wc -l

The result of the last line is 450, not 592. I think you will also get 450. This appears to mean that, at least with a clean site-packages, the tarball has a problem. Our buildout builds packages with a Python that has a clean site-packages (essentially with python -S) so this is a parallel.

However, I then retried the experiment with a virtualenv that does have my system's site-packages:

virtualenv pytztest2
cd pytztest2
wget http://pypi.python.org/packages/source/p/pytz/pytz-2009n.tar.gz
./bin/easy_install --always-unzip pytz-2009n.tar.gz
find lib/python2.6/site-packages/pytz-2009n-py2.6.egg/pytz/zoneinfo | wc -l

That result for me is also 450. Do you get 450, or the correct number (592)? If you get the wrong number, what are the steps you took to verify that you got the right number from your sdist before?

I then decided to look at the pytz branch to see if I could see anything helpful. The first thing I tries was to make my own source distribution using your Makefile. I did this with a Python without setuptools, and with a Python with only setuptools, and both times got an sdist with the correct number of files. When I tried to install my own generated sdist, I again got the incorrect number of files (449, oddly enough). Therefore, this seemed to rule out a hypothesis I had that you had setuptoolsbzr installed or something similar.

Therefore, what I'm left with is that there is something in the pytz sdist that breaks for me in all cases, with and without buildout. After some time investigating, I don't see anything wrong with your MANIFEST.in, nor do I see anything wrong with the MANIFEST.in in the sdist. I'll be very curious to hear what your results are when repeating the tests above. I will look into it again when I have time.

I'm marking as incomplete again, to indicate that I'm hoping to get feedback from you. I'm suspicious that I should instead move this bug to pytz, but I'm not confident yet.

Gary

[Gary Poster (gary)]

By the way, I don't think this is related, but when I made a sourcedist, the egg I made from it had some differences from the egg that I made from your sdist. The following is a diff of a sorted list of files from the two eggs, with the egg from your original sdist as the "old" and from mine as the "new". Since some of these are spelling differences an Asmara/Asmera, and Jujuy moves to Argentina, and so on, I wonder if this is because I have a bad or old copy of the Olsen data, or from some other mistake I made? Or is this a clue to the problem? Or is it indicative of some other problem?

7c7
< pytz/zoneinfo/Africa/Asmera
---
> pytz/zoneinfo/Africa/Asmara
58a59,60
> pytz/zoneinfo/America/Argentina/Cordoba
> pytz/zoneinfo/America/Argentina/Jujuy
69d70
< pytz/zoneinfo/America/Atikokan
85a87
> pytz/zoneinfo/America/Coral_Harbour
92a95
> pytz/zoneinfo/America/Edmonton
95d97
< pytz/zoneinfo/America/Ensenada
104a107
> pytz/zoneinfo/America/Halifax
115,116d117
< pytz/zoneinfo/America/Jamaica
< pytz/zoneinfo/America/Jujuy
119d119
< pytz/zoneinfo/America/Kentucky/Louisville
122a123
> pytz/zoneinfo/America/Louisville
127d127
< pytz/zoneinfo/America/Mazatlan
130d129
< pytz/zoneinfo/America/Mexico_City
152a152
> pytz/zoneinfo/America/Regina
154d153
< pytz/zoneinfo/America/Rosario
158a158
> pytz/zoneinfo/America/St_Johns
160a161
> pytz/zoneinfo/America/St_Thomas
165a167
> pytz/zoneinfo/America/Toronto
167c169,171
< pytz/zoneinfo/America/Virgin
---
> pytz/zoneinfo/America/Vancouver
> pytz/zoneinfo/America/Whitehorse
> pytz/zoneinfo/America/Winnipeg
175d178
< pytz/zoneinfo/Antarctica/McMurdo
177a181
> pytz/zoneinfo/Antarctica/South_Pole
180,181d183
< pytz/zoneinfo/Arctic
< pytz/zoneinfo/Arctic/Longyearbyen
197d198
< pytz/zoneinfo/Asia/Calcutta
199c200
< pytz/zoneinfo/Asia/Chungking
---
> pytz/zoneinfo/Asia/Chongqing
208c209
< pytz/zoneinfo/Asia/Hong_Kong
---
> pytz/zoneinfo/Asia/Ho_Chi_Minh
217c218,219
< pytz/zoneinfo/Asia/Katmandu
---
> pytz/zoneinfo/Asia/Kathmandu
> pytz/zoneinfo/Asia/Kolkata
237d238
< pytz/zoneinfo/Asia/Saigon
240d240
< pytz/zoneinfo/Asia/Shanghai
243c243
< pytz/zoneinfo/Asia/Thimphu
---
> pytz/zoneinfo/Asia/Thimbu
257a258
> pytz/zoneinfo/Atlantic/Reykjavik
262c263
< pytz/zoneinfo/Australia/Brisbane
---
> pytz/zoneinfo/Australia/Canberra
263a265
> pytz/zoneinfo/Australia/Darwin
266d267
< pytz/zoneinfo/Australia/LHI
268c269,270
< pytz/zoneinfo/Australia/North
---
> pytz/zoneinfo/Australia/Lord_Howe
> pytz/zoneinfo/Australia/Melbourne
269a272
> pytz/zoneinfo/Australia/Queensland
271,272d273
< pytz/zoneinfo/Australia/Sydney
< pytz/zoneinfo/Australia/Victoria
279,287d279
< pytz/zoneinfo/Canada
< pytz/zoneinfo/Canada/Atlantic
< pytz/zoneinfo/Canada/Central
< pytz/zoneinfo/Canada/East-Saskatchewan
< pytz/zoneinfo/Canada/Eastern
< pytz/zoneinfo/Canada/Mountain
< pytz/zoneinfo/Canada/Newfoundland
< pytz/zoneinfo/Canada/Pacific
< pytz/zoneinfo/Canada/Yukon
322,323d313
< pytz/zoneinfo/Etc/UCT
< pytz/zoneinfo/Etc/Universal
328a319
> pytz/zoneinfo/Europe/Bratislava
332d322
< pytz/zoneinfo/Europe/Chisinau
336d325
< pytz/zoneinfo/Europe/Helsinki
339c328
< pytz/zoneinfo/Europe/Ljubljana
---
> pytz/zoneinfo/Europe/Lisbon
342a332
> pytz/zoneinfo/Euro...

[Stuart Bishop (stub)]

On Sat, Oct 3, 2009 at 5:22 AM, Gary Poster <email address hidden> wrote:
> I looked into this tonight a bit, first by attempting to verify that
> pytz itself did not have a problem in the source tarball.  Here are the
> steps to repeat my experiment.
>
> sudo aptitude install virtualenv
> virtualenv --no-site-packages pytztest
> cd pytztest
> wget http://pypi.python.org/packages/source/p/pytz/pytz-2009n.tar.gz
> ./bin/easy_install --always-unzip pytz-2009n.tar.gz
> cd lib/python2.6/site-packages/pytz-2009n-py2.6.egg
> find lib/python2.6/site-packages/pytz-2009n-py2.6.egg/pytz/zoneinfo | wc -l
>
> The result of the last line is 450, not 592.  I think you will also get
> 450.  This appears to mean that, at least with a clean site-packages,
> the tarball has a problem.  Our buildout builds packages with a Python
> that has a clean site-packages (essentially with python -S) so this is a
> parallel.
>
> However, I then retried the experiment with a virtualenv that does have
> my system's site-packages:
>
> virtualenv pytztest2
> cd pytztest2
> wget http://pypi.python.org/packages/source/p/pytz/pytz-2009n.tar.gz
> ./bin/easy_install --always-unzip pytz-2009n.tar.gz
> find lib/python2.6/site-packages/pytz-2009n-py2.6.egg/pytz/zoneinfo | wc -l
>
> That result for me is also 450.  Do you get 450, or the correct number
> (592)?  If you get the wrong number, what are the steps you took to
> verify that you got the right number from your sdist before?

I get the same result here.

Last time I did not use virtualenv - I built and installed Python from source and used that installation. I also installed with '/path/python setup.py ...' rather than using easy_install.

Using virtualenv and 'python setup.py install', I get the correct number of zoneinfo files in the installed .egg (unzip -v pytz-2009n-py2.6.egg | grep zoneinfo | wc -l == 570 files). Also, hacking setup.py to change zip_safe to False generates an unzipped egg with the correct number of files (592 files and directories in zoneinfo).

> I'm marking as incomplete again, to indicate that I'm hoping to get
> feedback from you.  I'm suspicious that I should instead move this bug
> to pytz, but I'm not confident yet.

Yup. I don't know if this is a pytz, buildout, setuptools or distutils bug.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Stuart Bishop (stub)]

On Sat, Oct 3, 2009 at 5:29 AM, Gary Poster <email address hidden> wrote:
> By the way, I don't think this is related, but when I made a sourcedist,
> the egg I made from it had some differences from the egg that I made
> from your sdist.  The following is a diff of a sorted list of files from
> the two eggs, with the egg from your original sdist as the "old" and
> from mine as the "new".  Since some of these are spelling differences an
> Asmara/Asmera, and Jujuy moves to Argentina, and so on, I wonder if this
> is because I have a bad or old copy of the Olsen data, or from some
> other mistake I made?  Or is this a clue to the problem?  Or is it
> indicative of some other problem?

I can't reproduce this with my system Python 2.6 or a virtualenv Python 2.6:

$ tar xzf pytz-2009n.tar.gz
$ cd pytz-2009n/
$ find . | grep zoneinfo | wc -l
592
$ python setup.py sdist
[...]
$ tar tzf dist/pytz-2009n.tar.gz | grep zoneinfo | wc -l
592

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Max Bowsher (maxb)]

*sigh*.

So the problem is that when setuptools on-the-fly extracts a tarfile, it only extracts files and directories............ not symlinks.

[Max Bowsher (maxb)]

setuptools-0.6c9-py2.6.egg/setuptools/archive_util.py

def unpack_tarfile(filename, extract_dir, progress_filter=default_filter):
.....
            if member.isfile() or member.isdir():
^^^^ WHOOPS!

[Gary Poster (gary)]

Perhaps the right thing to do is to fix it in Distribute (same problem over there as of this writing) and then switch over.

[Stuart Bishop (stub)]

It looks like I can work around this bug in pytz by breaking the hardlinks created by the zoneinfo utilities. It will likely double the tarball size though.

[Gary Poster (gary)]

This bug has been fixed in setuptools.

[Colin Watson (cjwatson)]

We're now using pip rather than buildout, and a modern version of setuptools; so this should be fixed on the Launchpad side.