CherryPy fails to download even if *external* is set to True

[Sridhar Ratnakumar]

Other than CherryPy-2.1.1.tar.gz and CherryPy-2.2.0beta.tar.gz, all other versions (including 3.1.x) are missing. This makes it impossible to install, say, TurboGears from a pypi mirror.

> pypimirror -I -c etc/pypimirror.cfg
/home/sridharr/as/pypm/eggs/z3c.pypimirror-1.0.14-py2.6.egg/z3c/pypimirror/mirror.py:28: DeprecationWarning: the sets module is deprecated
  import sets
/home/sridharr/as/pypm/eggs/z3c.pypimirror-1.0.14-py2.6.egg/z3c/pypimirror/mirror.py:32: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  from md5 import md5
2009-06-09 15:23:46,492 DEBUG Processing package CherryPy
http://www.cherrypy.org
http://download.cherrypy.org/cherrypy/3.1.0rc1/
http://trac.cherrypy.org/cgi-bin/trac.cgi/wiki/CherryPyDownload
2009-06-09 15:23:49,628 WARNING Unload downloading http://trac.cherrypy.org/cgi-bin/trac.cgi/wiki/CherryPyDownload ()
http://download.cherrypy.org/cherrypy/3.1beta/
2009-06-09 15:23:49,825 WARNING Unload downloading http://download.cherrypy.org/cherrypy/3.1beta/ ()
http://download.cherrypy.org/cherrypy/3.0.2/
http://download.cherrypy.org/cherrypy/3.0.3/
http://download.cherrypy.org/cherrypy/2.3.0/
http://download.cherrypy.org/cherrypy/3.0.1/
http://download.cherrypy.org/cherrypy/3.1.0beta3/
http://download.cherrypy.org/cherrypy/3.0.0/
http://download.cherrypy.org/cherrypy/3.1.2/
http://download.cherrypy.org/cherrypy/2.2.1/
http://download.cherrypy.org/cherrypy/3.1.0/
http://download.cherrypy.org/cherrypy/3.0RC1/
2009-06-09 15:23:51,904 WARNING Unload downloading http://download.cherrypy.org/cherrypy/3.0RC1/ ()
http://download.cherrypy.org/cherrypy/3.1.1/
http://sourceforge.net/project/showfiles.php?group_id=56099
2009-06-09 15:23:52,726 DEBUG Package not available: HTML parse error: bad end tag: u"</sc'+'ript>", at line 530, column 38
2009-06-09 15:25:59,661 DEBUG Statistics
2009-06-09 15:25:59,661 DEBUG ----------
2009-06-09 15:25:59,662 DEBUG Found (cached): 0
2009-06-09 15:25:59,662 DEBUG Stored (downloaded): 0
2009-06-09 15:25:59,662 DEBUG Not found (404): 1
2009-06-09 15:25:59,662 DEBUG Invalid packages: 0
2009-06-09 15:25:59,663 DEBUG Invalid URLs: 0
2009-06-09 15:25:59,663 DEBUG Runtime: 2m13s
(pypm:^/trunk:r302*) ~/as/pypm
> ls /net/nas/data/languages/pypm/pypimirror/CherryPy/
CherryPy-2.1.1.tar.gz CherryPy-2.2.0beta.tar.gz index.html
>

[Sridhar Ratnakumar (srid)]

from the config file,

external_links = True
follow_external_index_pages = True

[Sridhar Ratnakumar (srid)]

Further analysis uncovered the root cause of this error:

The HTML of http://sourceforge.net/project/showfiles.php?group_id=56099 contains the following:

   if(typeof(urchinTracker)!='function')document.write('<sc'+'ript src="'+

BeautifulSoup consideres the JavaScript string '<sc'+'ript src="' as a HTML tag and fails. So much for a permissive parser?

[Andreas Jung (ajung)]

On 10.06.09 00:28, Sridhar Ratnakumar wrote:
> Public bug reported:
>
> Other than CherryPy-2.1.1.tar.gz and CherryPy-2.2.0beta.tar.gz, all
> other versions (including 3.1.x) are missing. This makes it impossible
> to install, say, TurboGears from a pypi mirror.
>
>
>> pypimirror -I -c etc/pypimirror.cfg
>>
> /home/sridharr/as/pypm/eggs/z3c.pypimirror-1.0.14-py2.6.egg/z3c/pypimirror/mirror.py:28: DeprecationWarning: the sets module is deprecated
> import sets
> /home/sridharr/as/pypm/eggs/z3c.pypimirror-1.0.14-py2.6.egg/z3c/pypimirror/mirror.py:32: DeprecationWarning: the md5 module is deprecated; use hashlib instead
> from md5 import md5
> 2009-06-09 15:23:46,492 DEBUG Processing package CherryPy
> http://www.cherrypy.org
> http://download.cherrypy.org/cherrypy/3.1.0rc1/
> http://trac.cherrypy.org/cgi-bin/trac.cgi/wiki/CherryPyDownload
> 2009-06-09 15:23:49,628 WARNING Unload downloading http://trac.cherrypy.org/cgi-bin/trac.cgi/wiki/CherryPyDownload ()
> http://download.cherrypy.org/cherrypy/3.1beta/
> 2009-06-09 15:23:49,825 WARNING Unload downloading http://download.cherrypy.org/cherrypy/3.1beta/ ()
> http://download.cherrypy.org/cherrypy/3.0.2/
> http://download.cherrypy.org/cherrypy/3.0.3/
> http://download.cherrypy.org/cherrypy/2.3.0/
> http://download.cherrypy.org/cherrypy/3.0.1/
> http://download.cherrypy.org/cherrypy/3.1.0beta3/
> http://download.cherrypy.org/cherrypy/3.0.0/
> http://download.cherrypy.org/cherrypy/3.1.2/
> http://download.cherrypy.org/cherrypy/2.2.1/
> http://download.cherrypy.org/cherrypy/3.1.0/
> http://download.cherrypy.org/cherrypy/3.0RC1/
> 2009-06-09 15:23:51,904 WARNING Unload downloading http://download.cherrypy.org/cherrypy/3.0RC1/ ()
> http://download.cherrypy.org/cherrypy/3.1.1/
> http://sourceforge.net/project/showfiles.php?group_id=56099
> 2009-06-09 15:23:52,726 DEBUG Package not available: HTML parse error: bad end tag: u"</sc'+'ript>", at line 530, column 38
> 2009-06-09 15:25:59,661 DEBUG Statistics
> 2009-06-09 15:25:59,661 DEBUG ----------
> 2009-06-09 15:25:59,662 DEBUG Found (cached): 0
> 2009-06-09 15:25:59,662 DEBUG Stored (downloaded): 0
> 2009-06-09 15:25:59,662 DEBUG Not found (404): 1
> 2009-06-09 15:25:59,662 DEBUG Invalid packages: 0
> 2009-06-09 15:25:59,663 DEBUG Invalid URLs: 0
> 2009-06-09 15:25:59,663 DEBUG Runtime: 2m13s
> (pypm:^/trunk:r302*) ~/as/pypm
>
>> ls /net/nas/data/languages/pypm/pypimirror/CherryPy/
>>
> CherryPy-2.1.1.tar.gz CherryPy-2.2.0beta.tar.gz index.html
>

Which BeautifoulSoup version are you running? I think this is caused
by some issue within the newest BeautifulSoup version. Try to install
BeautifulSoup 3.0.7 (instance of the newest one).

Andreas

[Sridhar Ratnakumar (srid)]

On 09-06-09 10:26 PM, Andreas Jung wrote:
> Which BeautifoulSoup version are you running?

BeautifulSoup-3.1.0.1-py2.6.egg

> I think this is caused
> by some issue within the newest BeautifulSoup version.

Indeed. http://www.crummy.com/software/BeautifulSoup/3.1-problems.html

 > Try to install
> BeautifulSoup 3.0.7 (instance of the newest one).

I will.

[Sridhar Ratnakumar (srid)]

Ok, that fixed the problem. Should install_requires be locked to <3.1.0 in setup.py, then?

[Andreas Jung (ajung)]

Fixed on the trunk