Zope fails badly if it can't encode a page with the chosen encoding

[Björn Tillenius]

If a browser says that it accepts only ISO-8859-1, and tries to get a page containing non ISO-8859-1 characters, Zope will fail badly, resulting in an error page that doesn't tell the user about the real problem.

I can think of two ways of solving this problem:

  * Return some HTTP response code, I can't remember which, but
    I'm quite sure there is a special one for encoding errors.

  * Encode the page in utf-8 anyway, letting the client handle
    the error handling

The latter is probably the most acceptable solution, since most clients do handle utf-8 even though they don't say it, and anyway, it saves them from seeing an error page.

We do have to consider the encoding of the returned data though. If we send the data as ISO-8859-1, the browser will most likely send POST data as ISO-8859-1 as well. What happens if the user tries to POST non-ISO-8859-1 characters in that case? In which encoding will the POST data be sent if the browser says utf-8 isn't acceptable, but we encode the page in utf-8 anyway?

[Björn Tillenius (bjornt)]

Another way is to encode the page in an encoding acceptable by the browser, replacing all unencodable characters with '?' or something like that.

[Jim Fulton (jim-zope)]

If the browser provides accepted encodings, I suggest we should
only use one of those. If we fail, we should return an error
indicating that the response could not be encoded in any of the requested encodings.

[Dmitry Vasiliev (hdima)]

I think the right solution is to return "406 Not Acceptable" HTTP response code, as by RFC 2616:

 """The resource identified by the request is only capable of generating
 response entities which have content characteristics not acceptable
 according to the accept headers sent in the request."""

If you want custom handling for encoding then you always can customize it per project.

[Björn Tillenius (bjornt)]

Status: Pending => Accepted

 Supporters added: BjornT

Ah yes, HTTP response code 406 was that one I was thinking of. I agree this is the right thing to do, since it still leave subprojects with the possibility to override the behaviour.

[Stephan Richter (srichter)]

Changes: submitter email, edited transcript, importance (medium => critical)

[Martijn Faassen (faassen)]

BjornT, any chance you can look into this soon? I'm thinking it might go beyond a simple bugfix, and I personally suggest deferring this issue to Zope 3.4, leaving it unfixed in Zope 3.3.

[Björn Tillenius (bjornt)]

Oh, I had forgotten about this one. I'll try to find time to fix it soon, but I have no problem deferring this to 3.4. This problem has existed since 3.0, so I wouldn't say it's critical for 3.3.

[Albertas Agejevas (alga)]

Fixed in zope.publisher rev 101467.

[Wolfgang Schnerring (wosc)]

has long been released