wanted: sb-ext:source-external-format

[Nikodemus Siivola]

Sticking

 (sb-ext:source-external-format :utf-8)

a the toplevel in a file should change the external format used to compile/load that file from that point on.

[Alastair Bridgewater (alastair-bridgewater)]

A few things:

  First, how do you make this work for utf-16?

  Second, can you make this work for arbitrary input prior to the
first such form in a file (especially arbitrary encoded characters in
comments)?

  Third, how do you handle an external-format change that alters the
encoded character width (switching from utf-16 to utf-8 in the middle
of a file... or vice versa)?

  Fourth, what about when the encoding directive /lies/? This isn't
quite as far-fetched as I'd like, as I've dealt with systems which
take a UTF-8 encoded XML message and transcode it to UTF-16 during
various processing, and then see the encoding specified as UTF-8 in
the header still. Not pretty. Or do the same with, say, latin-1 or
shift-jis and utf-8.

[James Y Knight (foom)]

Wouldn't it be better to do charset sniffing ala python2?
http://www.python.org/dev/peps/pep-0263/

[Douglas Katzman (dougk)]

Now that UTF-8 is the assumed default, this seems less important.