unicode error when using umlauts in a metadata element which is indexed

[Joachim Scmitz]

for testing of issue 820 I created a new metadata set with a string field which
is indexed. When entering umlauts into this field I get:

Traceback (innermost last):
  Module ZPublisher.Publish, line 98, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 39, in call_object
  Module Products.FileSystemSite.FSPythonScript, line 104, in __call__
  Module Shared.DC.Scripts.Bindings, line 252, in __call__
  Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec
  Module Products.FileSystemSite.FSPythonScript, line 160, in _exec
  Module Script (Python), line 17, in save_metadata
   - <FSPythonScript at /UniHd/service_views/Silva/edit/save_metadata used for
/UniHd/service_views/SilvaDocument/edit/VersionedContent/Document>
   - Line 17
  Module Products.SilvaMetadata.Binding, line 138, in setValues
  Module Products.SilvaMetadata.Binding, line 580, in _setData
  Module Products.ZCatalog.ZCatalog, line 500, in catalog_object
  Module Products.ZCatalog.Catalog, line 377, in catalogObject
  Module Products.ProxyIndex.ProxyIndex, line 147, in index_object
  Module Products.PluginIndexes.TextIndex.TextIndex, line 316, in index_object
  Module Products.PluginIndexes.TextIndex.Lexicon, line 167, in Splitter
UnicodeError: ASCII encoding error: ordinal not in range(128)

since this indicates that the error has something todo with indexing, I created
another string field which is not indexed. Here entering of umlauts is possible.

Simply setting indexed to "no" on the first field had no effect.

[Samuel Schluep (schluep)]

I use the 'ZCTextIndex' instead of the 'TextIndex'. The 'ZCTextIndex' can also
handle Unicode characters. See also Issue828.

[Joachim Scmitz (js-aixtraware)]

apparently some key value pairs have to be set there, like lexicon_id
What is the value for this are there others ?

[Jan-Wijbrand Kolman (jw-infrae)]

assigning to me.

[Jan-Wijbrand Kolman (jw-infrae)]

An example for the key/value pairs can be found in the subject element in the
silva-extra set. The key/values are:

   index_type Cosine Measure
   doc_attr proxy_value
   lexicon_id silva_lexicon

[Jan-Wijbrand Kolman (jw-infrae)]

Once a field is set to be indexed in the Catalog, this index will not be removed
even if the field in the metadata set is told not to get indexed anymore.

I'll create a new issue for this. For now, please remove the index from the
catalog by hand.

See issue852.

[Jan-Wijbrand Kolman (jw-infrae)]

Do these answers (the new issue and samuels remark) resolve this issue?

[Joachim Scmitz (js-aixtraware)]

using a ZCTextIndex resolves this issue. But it should be documented. Is there
documentation for metadata-sets ? I find the ZMI-userinterfaces not very
intuitive to use.

[Jan-Wijbrand Kolman (jw-infrae)]

OK, then I will resolve this issue.