[25 mod] Wrong characters appear in some Mac apps ([‘] → [⁸], [’] → [⁹], [–] → [ẃ], [—] → [Ẅ])

[Jay]

Problem occurs with:
    Chrome, Keynote, Pages, Byword, Ulysses; OS X 10.9.3–10.10.2
    Prince; OS X, Windows 8
Does not occur with:
    Illustrator, Photoshop, Textmate, Firefox, Safari

1. Set the font to Ubuntu (in Chrome, do this in Google Docs)
2. Copy this text, including the quotes:
    “About 8–9 o’clock — but no earlier”
3. Paste it into the app.

What you see: ⁸About 8ẃ9 o⁹clock Ẅ but no earlier⁹

From dupe bug 1400534:
The Ubuntu-R.ttf file appears to have a broken Unicode cmap subtable, where character U+2019 (single quote) is mapped to glyph 373, which is the superscript 9 glyph. The Apple Roman cmap subtable is correct, and maps it to glyph 111. This leads to inconsistent behaviour in different applications, depending on whether they access the Unicode or Apple Roman cmap subtable.

From <http://www.princexml.com/forum/topic/2867/ubuntu-font-not-rendered-properly#13544>:
macroman 183 maps to glyph 413, but U+2211 maps to glyph 1168
macroman 189 maps to glyph 414, but U+03a9 maps to glyph 949
macroman 206 maps to glyph 109, but U+0152 maps to glyph 360
macroman 207 maps to glyph 121, but U+0153 maps to glyph 372
macroman 208 maps to glyph 115, but U+2013 maps to glyph 361
macroman 209 maps to glyph 116, but U+2014 maps to glyph 362
macroman 210 maps to glyph 112, but U+201c maps to glyph 372
macroman 211 maps to glyph 113, but U+201d maps to glyph 373
macroman 212 maps to glyph 110, but U+2018 maps to glyph 372
macroman 213 maps to glyph 111, but U+2019 maps to glyph 373
macroman 219 maps to glyph 98, but U+00a4 maps to glyph 127
macroman 220 maps to glyph 108, but U+2039 maps to glyph 360
macroman 221 maps to glyph 120, but U+203a maps to glyph 372
macroman 222 maps to glyph 428, but U+fb01 maps to glyph 1170
macroman 223 maps to glyph 430, but U+fb02 maps to glyph 1172
macroman 226 maps to glyph 99, but U+201a maps to glyph 361
macroman 227 maps to glyph 101, but U+201e maps to glyph 361
macroman 246 maps to glyph 105, but U+005e maps to glyph 65
macroman 247 maps to glyph 117, but U+02dc maps to glyph 936
macroman 249 maps to glyph 353, but U+02d8 maps to glyph 1172
macroman 250 maps to glyph 354, but U+02d9 maps to glyph 1173
macroman 251 maps to glyph 355, but U+02da maps to glyph 1174
macroman 253 maps to glyph 357, but U+02dd maps to glyph 1176
macroman 254 maps to glyph 356, but U+02db maps to glyph 1175
macroman 255 maps to glyph 351, but U+02c7 maps to glyph 360

Workaround involving the TransType utility: https://discussions.apple.com/thread/6510381

[Paul Sladen (sladen)]

That's a pretty special find Jay! Not something I've seen before. To help debug this, please could you supply the exact list of characters that show this phenomenon (ie. the "apostrophes and quotes"). Please could you paste the exact list of characters, or codepoints.

  -Paul

[Paul Sladen (sladen)]

Jay, could you also post '⁹' into the same line; so that we can see which glyph it is more likely to be?

[Jay (jselway)]

Sure, no problem... One second.

[Jay (jselway)]

https://www.dropbox.com/s/uhlg7mr0d3luyzw/Screenshot%202014-06-25%2015.12.50.png

[Paul Sladen (sladen)]

And which "apostrophes and quotes" and quotes is this occuring with? Could you copy and paste them into this bug report?

[Jay (jselway)]

 I'm not sure exactly what you mean, but here's the text copy / pasted...

This shouldn’t be happening ⁹

----

Basically, if I type anything with a quote or apostrophe, it's replacing it with those weird characters.

"If I was typed this, it would appear with goofy apostrophe's"

[Jay (jselway)]

Ok, so I opened illustrator so I could pull out the glyphs.

u+2018
u+2019
u+201c
u+201d

Are the offending characters. Strangely, it displays correctly in Illustrator and Photoshop.

However not in Keynote. (OSX)

[Paul Sladen (sladen)]

This appears to be specific to Keynote 6:

  https://discussions.apple.com/message/26138086

From that discussion it would appear that it is not solely restricted to the Ubuntu Font Family; and occurs with (some?) other fonts all as a result of the Smart Quotes feature in:

  Edit > Substitutions > Smart Quotes.

Is it that you are typing " ' ", which is being "smartly" converted to " ’ ", yet displayed as '⁹'; does this occur with any other fonts on the system which you have installed? (Perhaps we can find some commonality.

[Kevin Godby (godbyk)]

I don't have a Mac, so I can test any of this myself.

However, I don't notice any substitutions that would go from U+2019 (right single quotation mark) to U+2079 (superscript nine) in the font.

I wonder if somewhere in the source code for Keynote there's some code that says, 'When the user types an apostrophe, convert it to U+2079' when they meant '...convert it to U+2019'.

Do any other fonts with a superscript nine (U+2079) exhibit this behavior in Keynote?

[Jay (jselway)]

Hey Paul, you are right. It's a Keynote thing. Thanks so much for finding that link.

There other instances of it happening that I found online, but this fixed my issue.

[Jay (jselway)]

It is kind of odd that smart quotes would trigger a super 9, and not automatically degrade to a normal quote.

[Paul Sladen (sladen)]

Thank you Jay; I think we can be reasonably confident that this was/is a bug in Apple Keynote 6 with the Smart Quotes features. I think Kevin's hypothesis of a typo between U+2019/U+2019 is a likely one, which would also explain why such a bug would have been fixed in a later version of Apple Keynote.

The bug in Keynote can be worked around by disabling Smart Quotes in Keynote using:

  Edit > Substitutions > Smart Quotes

[Paul Sladen (sladen)]

There's a hint that somebody has seen something similar with Apple Pages 5, where they get superscript eight '⁸' (U+2078) aswell. Superscript eight has the Unicode codepoint one lower, and so probably the opening ‘single smart quote’):

  https://discussions.apple.com/message/25685825

Another message that shows this in action (with Ubuntu Regular in Pages 5.2):

  https://discussions.apple.com/thread/6093729

I also found a reference to MS Outlook on Mac OSX replacing single and double quotes with superscript 1,2,3, but I think that's something different purely down to email and eight-bit encodings! As the same can occur with Thunderbird too:

  http://cabbagesofdoom.blogspot.co.uk/2014/01/how-to-stop-outlook-on-mac-osx.html
  http://forums.mozillazine.org/viewtopic.php?f=28&t=333695

Jay: out of interest, what Locale (language) are you using?

[Paul Sladen (sladen)]

This one mentions problems in Apple Fontbook too; and also in (apparently) in converting the Czech characters 'Š'/š' to superscript 8/9 aswell!

  https://discussions.apple.com/message/23597436

[Jay (jselway)]

English.

Jay Selway

Sent from my iPhone.

> On Jun 25, 2014, at 7:10 PM, "Paul Sladen" <email address hidden> wrote:
>
> There's a hint that somebody has seen something similar with Apple Pages
> 5, where they get superscript eight '⁸' (U+2078) aswell. Superscript
> eight has the Unicode codepoint one lower, and so probably the opening
> ‘single smart quote’):
>
> https://discussions.apple.com/message/25685825
>
> Another message that shows this in action (with Ubuntu Regular in Pages
> 5.2):
>
> https://discussions.apple.com/thread/6093729
>
> I also found a reference to MS Outlook on Mac OSX replacing single and
> double quotes with superscript 1,2,3, but I think that's something
> different purely down to email and eight-bit encodings! As the same can
> occur with Thunderbird too:
>
> http://cabbagesofdoom.blogspot.co.uk/2014/01/how-to-stop-outlook-on-mac-osx.html
> http://forums.mozillazine.org/viewtopic.php?f=28&t=333695
>
> Jay: out of interest, what Locale (language) are you using?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1334363
>
> Title:
> Apple Keynote 6 Smart Quotes: Apostrophe and quotes displaying as
> &u2079 (superior 9)
>
> Status in Ubuntu Font Family:
> Invalid
>
> Bug description:
> When using Ubuntu Font (all weights) in OSX, apostrophes and quotes
> are displaying as a superior 9.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-font-family/+bug/1334363/+subscriptions
>

[Cody Boisclair (codeman38)]

I had this same issue in Adium - em-dashes showed up as an umlauted W.

Apparently the issue is that some of the tables in the font are larger than recommended, and this causes problems with Apple's TrueType engine. The easiest fix is just to decompile and then recompile the font with 'ttx' (from the Python 'fonttools' package).

[Cody Boisclair (codeman38)]

To be more specific, here's the output from TTX when the font is recompiled - it seems to indicate that the problem lies in overly large GPOS (glyph positioning) lookup tables:

    Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 7, 'SubTableIndex:', None, 'ItemName:', None, 'ItemIndex:', None)
    Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 8, 'SubTableIndex:', None, 'ItemName:', None, 'ItemIndex:', None)

Recompiling the font reorganizes this table so that it no longer causes an overflow; that was enough to fix the issue for me.

[Paul Sladen (sladen)]

Cody: regarding the "Apparently the issue is that some of the tables in the font are larger than recommended" do you have any links that we can start to work from? I did found:

  "About the Mac OS X v10.6.7 Snow Leopard Font Update"
  http://support.apple.com/kb/HT4605

and would be interested to know if applying/unapplying that update alters the issue.

We can perhaps look into trying to workaround the issue; Cody: could you also paste precisely which commands you are suggesting running (and that have worked for you).

[Cody Boisclair (codeman38)]

This is what I ran to decompile and recompile the fonts:

    ttx Ubuntu-*.ttf
    rm Ubuntu-*.ttf
    ttx Ubuntu-*.ttx

The 10.6.7 update is not applicable for me, as I'm running OS X 10.9.3.

[Cody Boisclair (codeman38)]

To add to that last comment - I don't recall having experienced this issue before 10.9, so I'm guessing the issue was probably introduced with that version of the OS.

[Launchpad Janitor (janitor)]

[Expired for Ubuntu Font Family because there has been no activity for 60 days.]

[mikeday (mikeday)]

We get the same problem affecting Prince and other applications on Linux, as described here:
http://www.princexml.com/forum/topic/2867/ubuntu-font-not-rendered-properly
https://bugs.launchpad.net/ubuntu-font-family/+bug/1400534

The Unicode cmap subtable appears to be wrong, is any action being taken to fix this?

[Paul Sladen (sladen)]

Mike: as I understand it to-date, this substitution only occured when applications running on certain (recent) versions of Mac OSX.

Could you clarify "other applications on Linux", have you been able to reproduce this on Linux now?

[mikeday (mikeday)]

Yes, it's not operating system specific, it's due to a broken Unicode cmap subtable. If you like, I can run a quick comparison of the two subtables and find all the inconsistencies, that will be much more thorough than mucking around with different applications.

[Paul Sladen (sladen)]

I appreciate that it is related to the CMap tables. Do you have a method to reproduce the issue under Linux or Ubuntu. This would allow it to be pinpointed considerably quicker.

[mikeday (mikeday)]

You can reproduce it with Prince on Linux, or apparently with LibreOffice or Pages as well, although I haven't tried those. See the thread on our forum.

Here is the complete list of cmap inconsistencies described in more detail on the thread:

macroman 183 maps to glyph 413, but U+2211 maps to glyph 1168
macroman 189 maps to glyph 414, but U+03a9 maps to glyph 949
macroman 206 maps to glyph 109, but U+0152 maps to glyph 360
macroman 207 maps to glyph 121, but U+0153 maps to glyph 372
macroman 208 maps to glyph 115, but U+2013 maps to glyph 361
macroman 209 maps to glyph 116, but U+2014 maps to glyph 362
macroman 210 maps to glyph 112, but U+201c maps to glyph 372
macroman 211 maps to glyph 113, but U+201d maps to glyph 373
macroman 212 maps to glyph 110, but U+2018 maps to glyph 372
macroman 213 maps to glyph 111, but U+2019 maps to glyph 373
macroman 219 maps to glyph 98, but U+00a4 maps to glyph 127
macroman 220 maps to glyph 108, but U+2039 maps to glyph 360
macroman 221 maps to glyph 120, but U+203a maps to glyph 372
macroman 222 maps to glyph 428, but U+fb01 maps to glyph 1170
macroman 223 maps to glyph 430, but U+fb02 maps to glyph 1172
macroman 226 maps to glyph 99, but U+201a maps to glyph 361
macroman 227 maps to glyph 101, but U+201e maps to glyph 361
macroman 246 maps to glyph 105, but U+005e maps to glyph 65
macroman 247 maps to glyph 117, but U+02dc maps to glyph 936
macroman 249 maps to glyph 353, but U+02d8 maps to glyph 1172
macroman 250 maps to glyph 354, but U+02d9 maps to glyph 1173
macroman 251 maps to glyph 355, but U+02da maps to glyph 1174
macroman 253 maps to glyph 357, but U+02dd maps to glyph 1176
macroman 254 maps to glyph 356, but U+02db maps to glyph 1175
macroman 255 maps to glyph 351, but U+02c7 maps to glyph 360

[Paul Sladen (sladen)]

So what myself and others need is a minimum test method: a way of testing and showing the problem, and then testing again with various modification and seeing whether the problem still exists. This is developing a test case; at present we don't have a *precise* test-case:

For example, a hypothetical test-case might be:

  1. download X.exe from $URL.
  2. download Y.input file from $URL
  3. run X.exe Y.input
  4. choose "Export as PDF"

What happens:

  4. on the fourth line, the nineth character shows as '`' shows as superscript '8':

What should happen:

  4. on the fourth line, the nineth character should show as '`'

This is where your help (collectively) would be really useful.

[mikeday (mikeday)]

Download Prince from princexml.com, any version, run it on a HTML file containing any of the characters listed above, eg. U+2018 and U+2019, and see that they display incorrectly.

[Joel (iosdev)]

This is not limited to Keynote. It also occurs in Pages, TextEdit and Sketch. It occurs in the latest versions of all those apps, and more, on the latest version of Yosemite.

I also see opening quotes rendered as 8 and closing quotes rendered as 9.

Turning off smart quotes in Pages seems to help.

I have never seen this with any other font. It seems to be a serious issue with Ubuntu (but regular, it doesn't seem to happen in the Ubuntu monospace font).

Any ideas?

We need a fix - this is making it very difficult, almost impossible, to use Ubuntu Regular on OS X.

Please let me know how I can help.

[Joel (iosdev)]

This is easy to reproduce.

Here's a test case:
1. Download the Ubuntu Regular font from http://font.ubuntu.com
2. Install it on your Mac running Yosemite
3. Launch the free Pages app.
4. Try typing any text using double quotes like "hello, world!" and you'll see it as 8hello, world!9
5. This is uniformly and consistently reproducible for me - I can make it happen 100% of the time on 3 difference Macs.

Can you try this. It should work for you.

I'm happy to help however I can. Please let me know.

[mikeday (mikeday)]

As posted earlier, the Unicode cmap is broken. So the font needs to be rebuilt with a valid Unicode cmap. But I don't know who is the author of the font, and what tools were used to create it.

[Joel (iosdev)]

As per http://font.ubuntu.com/about/:

- The development is funded by Canonical http://www.canonical.com/
- The technical font design work and implementation is being undertaken by Dalton Maag http://www.daltonmaag.com/

So it seems like Dalton Maag should fix the font ... Shall we contact them?

Thanks

[mikeday (mikeday)]

Yes, definitely. Follow the chain until finding the person that created Ubuntu-R.ttf, and ask them to create it again, but better :)

[Paul Sladen (sladen)]

mikeday: that person would like to "create it better", but needs to know what to change, and how to test that the change has worked.

[Joel (iosdev)]

They can test their changes using my "test in Pages or TextEdit" steps above. Both of those are free software on a Mac.

Here's how somebody solved the issue on the Apple forums (https://discussions.apple.com/thread/6510381?start=0&tstart=0):

--- Begin
I managed to solve this problem!
Anyone who is experiencing the same issue, may follow these steps:

Uninstall the Ubuntu Font Family from Font Book;
get the .ttf files from http://font.ubuntu.com again;
download TransType (7-days trial version);
open it;
drag the folders containing the font (ubuntu and Ubuntu_Mono) into its window;
select all the styles, this way
Screen Shot 2014-08-29 at 17.35.50.png;
re-convert the fonts — File > Convert Selected;
open re-converted .ttfs and install.
--- End

Does that help?

How do we contact the original font author at Dalton Maag?

[Paul Sladen (sladen)]

Joel: I don't have easy access to a Mac, which is while I'm on finding a way to test the fix under Ubuntu in a way that can be fully scripted and automated as a unit-test within the build-chain.

[Joel (iosdev)]

Hi Paul,

But I'm not sure if the bug actually occurs on platforms other than OS X ... So it needs to be tested on a Mac. There "mac in the cloud" type services where you can rent a Mac virt and ssh / VNC into a Mac in the cloud for only a few dollars per month. I could also test any fixes you make if you send me the repaired font files.

I agree about the desire for unit tests, but if we can fix it for now, it may stay fixed for some time and that will help a lot of people. :-)

Thanks again and please let me know how I can help,

Joel

[mikeday (mikeday)]

For unit testing, probably writing a little program to access the cmaps would be the easiest way. Or using some other font debugging tool. Then it could be reproduced on every platform. But I wouldn't expect this problem to reoccur, unless the tools being used are truly broken.

[mach (j-mach-wust)]

I can confirm this bug on the most recent Mac OS X 10.10.2 Yosemite with Google Chrome 41.0.2272.89 and with Keynote, whereas neither Firefox 36.0.1 nor Safari 8.0.3 are affected.

All weights and styles of Ubuntu seem to be equally affected, not only Ubuntu-R.ttf

[Dominik Röttsches (drott)]

Google Chrome instance of this issue is tracked in crbug.com/426056

[Michael Rawling (2-mike-v)]

Guys, any update on this?

Any idea when this might get looked at?

thanks!

M

[Paul Sladen (sladen)]

Hello Michael, the reason this was re-triaged by mpt three months ago was to allow it to be "looked at"! Hopefully Magdalena can answer any questions about timelines if there is more up-to-date information that what I know.

[Matthew Paul Thomas (mpt)]

Success! This bug appears to be fixed in a new version supplied by Dalton Maag.

I had time to test only the regular weight, in TextEdit on OS X 10.10.2. But all the characters in the test case appeared as expected.

[Michael Rawling (2-mike-v)]

Superb! Thanks

Is that v0.82? If so - I can't see a way/place to download the ttf easily - is it released yet?

[Paul Sladen (sladen)]

Michael: it's a test release to check that the fix works. Should have a 0.83 to push out in a couple of weeks for you (and everyone else!). Would you be willing to help beta the 0.83 before we push it out to huge numbers of users?

[Michael Rawling (2-mike-v)]

Brilliant!! Thanks, Paul!

Sure - hit me up for the beta: I/We use Ubuntu as our brand's primary font across blog, website, and many SAAS/browser software application UIs and seem to get this problem a lot, perhaps because of the type of content and industry language

[Michael Rawling (2-mike-v)]

So, I've got a bunch of different use contexts and users (from web apps, sites, blogs through to users of MS and Adobe apps and so on) here, ready to test but I'm not set up to pull the repo and compile it right now..

...for expediency would it be at all possible to send/PM/somehow make available the v0.83 .ttf? Can get testing it right away

Cheers

m

[Michael Rawling (2-mike-v)]

(whenever you able to :)

[Paul Sladen (sladen)]

Yes, Michael, my intention would be to send you a zip file of .ttf (v0.82) when they are received—probably next week.

This would be on the understanding that you won't distribute them outside of your control, so that they don't "spread in the wild" if we find something further is required.

[Michael Rawling (2-mike-v)]

Thanks, Paul.

Great. Understood - I totally expected that.

I'll keep a close eye on the test users and only test the browser based implementations in private staging servers or on in browser based software hosted on local 'dev' machines.

If you need me or any of the test users to sign anything we'd be more than happy to do so.

[Steven (stev7n)]

Hi everyone,

I'm actually having the same issue with sketch.

@Paul : Do you have an idea of when the new font would be released ?
It's very hard to work on this way ! :)

Thanks for your research anyway, i thought i was an isolated case.

Steven

[Paul Sladen (sladen)]

Hello Steven, I'm hopeful of receiving something releasable in the next week or so.

[Paul Sladen (sladen)]

Ubuntu Font Family 0.83 proportional updates received and distributed for private test. The diffs look good, so hopefully it can be pushed to font.ubuntu.com in the next few days.

[Jorn (jornhenkes-deactivatedaccount)]

Hello Paul,
Great that this issue is almost fixed!

Any idea when the new version will be pushed to font.ubuntu.com? Does the new version also include an updated version for the web fonts?
I'm about to launch a website using Ubuntu tomorrow, so it would be awesome to use the updated version on that website.

Thanks for all the work,
Jorn

[Magdalena Mirowicz (magdalena-mirowicz)]

File with bug #1334363 fixed

[thebitguru (farhan)]

Hi Paul,

I noticed that the version of this font on font.ubuntu.com is still at version 0.8. When will that version if updated?

Thanks!

[Paul Sladen (sladen)]

@farhan, @jornhenkes: 0.83 is now available from:

  http://font.ubuntu.com/

Thank you all for the help and assistance in tracking this down, and to Marc Foley at Dalton Maag for helping make the fix a reality!

(For those waiting for the Ubuntu .debs these should follow in due course, if everything works—although except for Sketch(?) this are probably not likely to make any visible difference).

[thebitguru (farhan)]

Excellent. Thank you very much!

[Jorn (jornhenkes-deactivatedaccount)]

Excellent it is! Thanks everyone who helped fixing this!

[Loïc Minier (lool)]

Hi,

I've removed locally installed user fonts on OSX and installed the .ttf files from the 0.83 release, but when I type in the font preview box of the OSX Font Book app, the first chars are displayed correctly but after typing a handful I get the dreaded 8/9s again.

This is with OSX 10.10.5.

Separately, under Chrome 44.0 visiting web sites using Google fonts I still get the incorrect rendering for the chars; https://www.google.com/fonts/specimen/Ubuntu does mention an August 2014 (!) update to 0.83, but the typecast preview still has the issue, despite clearing my browser's cached files.

Thanks,