text_type can give IndexError: string index out of range in zope 2.11 & 2.12

[Simon Michael]

Reporting for Justin Dunsworth:

<sm> The fix for https://bugs.launchpad.net/zope2/+bug/143948 , in zope.contenttype since 3.4.1, causes your bug in zope 2.11 and 2.12
<sm> http://gist.github.com/242321 is the code that's failing, or close enough.. line 39
<sm> to reproduce, have a dtml document return whitespace greater than 14 chars in length

[Simon Michael (simon)]

In case the gist disappears:

def text_type(s):
    """Given an unnamed piece of text, try to guess its content type.

    Detects HTML, XML, and plain text. Returns a MIME type string
    such as 'text/html'.
    """
    # at least the maximum length of any tags we look for
    iMAXLEN=14
    if len(s) < iMAXLEN: return 'text/plain'

    i = 0
    while s[i] in string.whitespace: # <- bug
       i += 1 # <-

[Tres Seaver (tseaver)]

What is the buggy behavior which returning more than 14 characters of whitespace triggers?

[Simon Michael (simon)]

See the last two lines I commented ? Lines 39 and 40 in the paste. i grows too big, causing the IndexError.

[Tres Seaver (tseaver)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Simon Michael wrote:

> See the last two lines I commented ? Lines 39 and 40 in the paste. i
> grows too big, causing the IndexError.

What IndexError? The attached patch adds two calls to the test case for
that function which prepend 15 spaces to text: they pass without
errors. This patch is made against a trunk checkout, after cleaning up
some typical "old Zope" ugly testing cruft.

Tres.
- --
===================================================================
Tres Seaver [phone] <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksNSWwACgkQ+gerLs4ltQ7IzwCgm8xyUcO5cxCm9rRssN4rv+8f
imkAoJ9TdzRDz8SjfTULDSGD8IQ6dnQR
=lacZ
-----END PGP SIGNATURE-----

[Andrew P. Ho (aho)]

Importance of this error should be bumped up to "Critical" because it affects DTML Document and DTML Method. This renders Zope 2.12.3 unusable.

I don't know whether this is exactly the same bug but it is most likely highly related.
For example, I am getting the following traceback:

Traceback (innermost last):
  Module ZPublisher.Publish, line 127, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 47, in call_object
  Module OFS.DTMLDocument, line 150, in __call__
   - <DTMLDocument at /production/test2>
   - URL: http://192.168.2.23:8080/
   - Physical Path: /production/test2
  Module zope.contenttype, line 76, in guess_content_type
  Module zope.contenttype, line 39, in text_type
IndexError: string index out of range
---------
DTML Document content in test2

<dtml-call "REQUEST.set('jump_back', 'no')">
<dtml-call "REQUEST.set('jump_back', 'no')">
<dtml-call "REQUEST.set('jump_back', 'no')">
<dtml-call "REQUEST.set('jump_back', 'no')">
--------
Zope Version (2.12.3, python 2.6.4, linux2)
Python Version 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1]

[Andrew P. Ho (aho)]

Here is a solution applied to
lib/python2.6/site-packages/zope.contenttype-3.4.2-py2.6.egg/zope/contenttype/__init__.py

*** Please incorporate into next Zope 2 release ***
--------
   # at least the maximum length of any tags we look for
    iMAXLEN=14
    if len(s) < iMAXLEN: return 'text/plain'
    i = 0

# here is the fix starting line 39
    try:
        while s[i] in string.whitespace:
         i += 1
    except IndexError:
         i=i-1

[Tres Seaver (tseaver)]

zope.contenttype 3.5.1 released with a test and fix:

  http://pypi.python.org/pypi/zope.contenttype/3.5.1