TALES string expressions cannot co-exist with dynamic path expressions

[Jeff Shell]

In Zope 2.10.6, a string TALES expression's results cannot then be used in a dynamic path expression. This seems to be a result of using the unicode-friendly 'zope.tales' from Zope 3 with the ancient 'unrestrictedTraverse'. I've been using this with expressions like 'galleries/?gallery_id', where I might define 'gallery_id' as 'string:submissions-${gallery_name}'.

Steps to reproduce:

In an empty folder, add two subfolders, 'gallery', and 'submission-gallery' (the names don't matter, but having the 'submission-' or some other prefix leading a common name is). Add a page template with the following contents:

    <ul tal:define="gallery_name python:'gallery'">
      <li tal:define="gallery container/?gallery_name">
        <strong tal:content="gallery/title_or_id">main gallery</strong>
      </li>
      <li tal:define="sub_gal_id string:submission-${gallery_name}; subgal container/?sub_gal_id">
        <strong tal:content="subgal/title_or_id">Submission Gallery</strong>
      </li>
    </ul>

The first define simulates a function call (without the call) in my code which returns a classic 'str' object. The first dynamic path expression, 'container/?gallery_name' succeeds. However, the second one fails with ``KeyError: u's'``, and the Error Log will show something like this::

  Module OFS.Traversable, line 284, in unrestrictedTraverse
   - __traceback_info__: ([u'y', u'r', u'e', u'l', u'l', u'a', u'g', u'-', u'n', u'o', u'i', u's', u's', u'i', u'm', u'b', u'u'], u's')
KeyError: u's'

Which comes from what happens in OFS.Traversable's unrestrictedTraverse lines 143-147 (Zope 2.10.6 release):

        if isinstance(path, str):
            # Unicode paths are not allowed
            path = path.split('/')
        else:
            path = list(path)

It turns out that tal:define="sub_gal_id string:submission-${gallery_name}" returns ``u'submission-gallery'``, and when that is passed into unrestrictedTraverse, that gets turned into a list since "unicode paths are not allowed".

In Products.PageTemplates.Expressions, there are a few comments saying how much unicode is desired and is good, but here in the all-too-important traversal steps, it's verboten; which runs contrary to all of the muscle behavior I've acquired since using Zope 3.

I don't know if unrestrictedTraverse needs to be patched to try to encode a unicode path value to a common, basic encoding such as ASCII (since that's presumably what most current paths are anyways); or if the Traversers being used for PageTemplate expressions should try to handle that encoding since they're dealing with output from the unicode-friendly Zope 3 tales expressions and are having to work with the more unicode-hostile-or-unaware Zope 2.

It appears that the Zope 2.11.0 code has the same issue.

This is not a critical issue for me - I know there are numerous ways around it (ie - changing ``string:submission-${gallery_name}`` to ``python:'submission-%s' % gallery_name`` works just fine with the dynamic path expression); but it's yet another fun and surprising obscure issue found when trying to get our few remaining Zope 2 customers up to a relatively modern platform.

Bug 142117 opened this issue up years ago, and was rejected with 'if a str is expected then thats what it should be'. That may have been OK in 2002-2003, but not in 2008 when Zope 3 and semi-modern Python-isms are being used.

https://bugs.launchpad.net/bugs/142117

[Philipp von Weitershausen (philikon)]

I agree with your conclusion.

The problem stems from Zope 2 using 8bit strings for object ids. So
while it's certainly possible to use non-ASCII characters in path
names, they'll always be stored in some encoding. I imagine the
default encoding would be latin-1 and there would probably be a way to
change it (frankly I don't know how, but I think Plone somehow manages
to use UTF8 I think). Anyway, certainly the publisher must be using
this implicit encoding when mapping unicode URLs to object paths, so
there must be some way of figuring this out. Whatever it is, I think
the TALES traversal machinery in Products.PageTemplates should use the
same logic.

Since I'm the one who ported Zope 2 to use the Zope 3 ZPT
implementation, I feel responsible for making this legacy use case
work again. However, it would help me a lot (and probably speed things
up) if you could perhaps create some tests for this use case. There
should be some tests that you can extend in Products.PageTemplates.
Submitting a patch that just adds failing tests would be very helpful
already.

Thanks!

[Jeff Shell (jshell)]

I haven't used a development (checkout) of either Zope 2 or 3 for
years. I'm trying to wrap up this legacy project so I can get myself
back to the much happier land of Zope 3 :).

But I'd like to help and keep Zope 2 solid, so I'll try to supply some
tests, patches, or whatever is necessary. Are there up-to-date docs
about how to work with a development checkout, run tests, etc?

On Jul 2, 2008, at 2:24 PM, Philipp von Weitershausen wrote:

> I agree with your conclusion.
>
> The problem stems from Zope 2 using 8bit strings for object ids. So
> while it's certainly possible to use non-ASCII characters in path
> names, they'll always be stored in some encoding. I imagine the
> default encoding would be latin-1 and there would probably be a way to
> change it (frankly I don't know how, but I think Plone somehow manages
> to use UTF8 I think). Anyway, certainly the publisher must be using
> this implicit encoding when mapping unicode URLs to object paths, so
> there must be some way of figuring this out. Whatever it is, I think
> the TALES traversal machinery in Products.PageTemplates should use the
> same logic.
>
> Since I'm the one who ported Zope 2 to use the Zope 3 ZPT
> implementation, I feel responsible for making this legacy use case
> work again. However, it would help me a lot (and probably speed things
> up) if you could perhaps create some tests for this use case. There
> should be some tests that you can extend in Products.PageTemplates.
> Submitting a patch that just adds failing tests would be very helpful
> already.
>
> Thanks!
>
> --
> TALES string expressions cannot co-exist with dynamic path expressions
> https://bugs.launchpad.net/bugs/244715
> You received this bug notification because you are a direct subscriber
> of the bug.

Thanks,
Jeff Shell
<email address hidden>

[Philipp von Weitershausen (philikon)]

El 2 Jul 2008, a las 22:49 , Jeff Shell escribió:
> But I'd like to help and keep Zope 2 solid, so I'll try to supply some
> tests, patches, or whatever is necessary. Are there up-to-date docs
> about how to work with a development checkout, run tests, etc?

Sorry for the late reply.

Getting a working checkout of Zope 2 is easy:

   $ export z=svn+ssh://<email address hidden>/repos/main
   $ svn co $z/Zope/trunk Zope

   $ cd Zope
   $ ./configure --with-python=...
   $ make

   $ python test.py

[Hanno Schlichting (hannosch)]

Jeff promised to supply some tests and patches but those never made it.

[Colin Watson (cjwatson)]

The zope2 project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope2.