Problem with unicode in oaipmh

[Benno Luthiger]

I stumbled over a unicode problem in the oaipmh.client module.

The OAI PMH repository I'd like to harvest returns setNames containing unicodes when I request their ListSets (requestUrl?verb=ListSets). Unfortunately, the module returns lxml.etree._ElementUnicodeResult object when it processes text with Umlauts. Later then, the program tries to store the listSets by pickling the entries and crashes.

To work around this problem, the _ElementUnicodeResult objects have to be converted to proper unicode objects.
The following code can achieve this:
In oaipmh.client.buildSets() [line 260/261]
==
            setSpec = e('string(oai:setSpec/text())')
            setName = u'%s' %e('string(oai:setName/text())') # e('string(oai:setName/text())') may return _ElementUnicodeResult
==

May be there's a better solution.
Can you fix the problem and prepare a new release?
Benno

[Benno Luthiger (benno-luthiger)]

The clean solution, of course, is the following:
==
            # make sure we get back unicode strings instead
            # of lxml.etree._ElementUnicodeResult objects.
            setSpec = unicode(e('string(oai:setSpec/text())'))
            setName = unicode(e('string(oai:setName/text())'))
==
I have fixed that bug in client.py and made a check in (r44465).

Benno

[Jasper Op de Coul (jasper-infrae)]

Hi Benno,

Thanks for the unicode fixes you committed.
I already added those protections in other parts of the library but apparently not in the client code.

I have send a question to the lxml project about this. Especially in the Zope world there is a big
chance that these values get pickled at some point, and then you end up with these problems.

Maybe they will fix it in the lxml codebase so we can remove these conversions.
I released pyoai 2.3.4 today which contains your patch.

Kind regards,
Jasper Op de Coul