utf16 file detected as binary file

[JoeDevSys]

bzr 1.5 on windows

Create new text file in notepad, save file with encoding="unicode" in SaveAs dialog. (First two bytes of file are 0xff 0xfe)
Commit.
Edit file with new text.
bzr diff treats the file as binary and does not report text differences.

Interestingly "UTF8" files are treated correctly.

[Wesley J. Landaker (wjl)]

Sounds like your file is saved as UTF-16 LE, which is looks just like a binary file to almost any tool, not just Bazaar.

Saving your files as UTF-8 is almost always the right thing to do, as it is the only Unicode encoding fully supported by almost every tool.

[Jelmer Vernooij (jelmer)]

we probably need something more complex than the simple check for null bytes that we use atm afaik

[Jelmer Vernooij (jelmer)]

a more complex check /to determine if a file is a binary file/

[Martin Pool (mbp)]

As a workaround for this, you can use the --using option to diff to run an external program that does understand utf-16 or ucs-2

[Tymek (maju7)]

This is minor.

We had "text as binary" problem and we
(BranchA and BranchB are branches of Trunk)
1. Saved FileA as ANSI 1252 in branch A -> the std diff worked in the text mode-> good
2. Merged BranchA into Trunk
3. Saved FileA ANSI 1252 in branch B -> now the std diff worked in the text mode-> good
4. Merged BranchB in from trunk

Bzr still "complained" it's a binary file. We did it twice to make sure that .BASE file was 1252.

Only when
BranchB used the file in BranchA and added the changes after (with an external diff) and committed,
bzr started to report conflicts as text conflicts.

[John A Meinel (jameinel)]

I don't think this is actively in-progress anymore. So I'm pulling this off of Gary's plate.
I will mark it as "work to do" for ~canonical-bazaar since he did do some work on this, and we would like that work to not be lost. Consider it as something a Patch Pilot can pick up as desired.