Monospaced font is not monospaced

[Sam Bull]

The "FreeMono" font in Ubuntu 12.04 seems to not be monospaced.

More information can be read at: http://<email address hidden>/msg16829.html

A little further testing after this discussion, and I've noticed I don't have the problem with the "Ubuntu Mono" font, it just seems to be the "FreeMono" font which is causing the problem.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: apport-gtk 2.0.1-0ubuntu7
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
Uname: Linux 3.2.0-24-generic x86_64
ApportLog:

ApportVersion: 2.0.1-0ubuntu7
Architecture: amd64
CheckboxSubmission: d7186b4470c29fb9bde4930265e9b32c
CheckboxSystem: 7e42599bda39ea7ff8b528272b6ef52b
Date: Thu May 17 23:33:07 2012
ExecutablePath: /usr/share/apport/apport-gtk
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Release amd64 (20120425)
InterpreterPath: /usr/bin/python2.7
PackageArchitecture: all
SourcePackage: apport
UpgradeStatus: No upgrade log present (probably fresh install)

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1001033/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Steve White (stevan-white)]

Hi Sam,

FreeMono is tested on multiple OSs, with several applications which work only with monospace fonts, as well as with scripts that specifically look for any variations in glyph width. I can assure you, FreeMono is monospaced.

The problem you posted on pygame-users (appended here for completness) is coming from intermediate rendering software. You are talking about width of glyphs in terms of rendered pixels: this is a far cry from how wide the glyphs are, as described by the font.

Some of the responses to your post state that pygame.font.SysFont is unreliable for this sort of thing, and suggest using other interfaces. Did you try that?

Cheers!

Original posting describing the problem:
---------------------------------
Before I go and file a bug against Ubuntu, can somebody confirm I'm not
being stupid.

For my input boxes, the cursor position depends on a monospaced font.
This means I check the length of a rendered letter ("e") using the font,
and then set the cursor position as a multiple of this length.

This worked fine before, but in Ubuntu 12.04, it no longer seems to be
monospaced. Getting the length of the character "e" on my system gives
me 9 pixels, but it seems that some characters, such as "h", are 10
pixels, which starts offsetting my cursor position and messing up my
input box.

The line I use to load the font is:
        pygame.font.SysFont("FreeMono, Monospace", 16)

That should definitely return a monospaced font, right?

And, the line used for getting the width is:
        mono_font.render("e", False, (0,0,0)).get_size()[0]

Is this a bug, or am I doing something stupid?
---------------------------------

[Sam Bull (dreamsorcerer)]

Hmm, I got the impression it was only unreliable in returning the correct font. For example, if I use "Monospace" it still returns me the FreeMono font. I tried the alternative of using the ttf font directly, using pygame.font.Font("/usr/share/fonts/truetype/freefont/FreeMono.ttf") and get the same problem.

I'm sure I didn't have this problem on 11.10, and I do not have this problem with the "Ubuntu Mono" or any other mono fonts I have tried.

OK, I just looked up the font separately, and it is only the latest FreeMono font that is having these troubles, found here:
http://ftp.gnu.org/gnu/freefont/freefont-ttf-20100919.tar.gz
If I download any of the older versions, they work perfectly, such as:
http://ftp.gnu.org/gnu/freefont/freefont-ttf-20090104.tar.gz
http://ftp.gnu.org/gnu/freefont/freefont-ttf.tar.gz

If you're sure it's not the font, then I'll go back to the pygame mailing list, and see if we can work out where the bug is. It's just odd that it only seems to occur with one version of one particular font.

[Sam Bull (dreamsorcerer)]

It even works perfectly with the newest .otf, and the .otf released at the same time as the problematic .ttf:
http://ftp.gnu.org/gnu/freefont/freefont-otf-20120503.tar.gz
http://ftp.gnu.org/gnu/freefont/freefont-otf-20100919.zip

[Steve White (stevan-white)]

Hi, Sam,

It isn't even clear to me that the software is behaving incorrectly, depending on just what it is meant to accomplish.
Rendering software is geared towards displaying letters legibly on digital devices, not on making those images all of the same width. It's just saying, when its libraries render a given letter at a certain resolution using a certain algorithm, the image appears to be of a certain width (I guess).

Is the software renderer used by your libraries intended to preserve the width of characters? That is, if two letters have the same width, then the rendered versions of the two letters should also have the same width? Does the documentation say that?

You may be mixing up some concepts here. (It is also possible some software layer is misbehaving, but that is a second guess -- again, I don't know exactly what your libraries are meant to do.) The font is monospaced in that the width of all the nonzero-width glyphs are identical. The measure is not in device pixels, it is in device-independent EM; the fact that you refer to pixels indicates a miscomprehension. A measure in terms of pixels happens only after the glyph is rendered for a certain device.

When you say, "It works perfectly", what sort of tests are you doing? Are you running a test on the whole font, or a large part of it, looking for variations in the rendered width value, or are you just trying a few letters? What exactly is your test?

A full survey of rendered widths reported by your libraries, say of all the Latin characters, between various versions and various fonts, might be interesting.

The TTF and OTF versions of FreeMono are identical with regard to glyph bounds. The outlines of the OTF and TTF version are identical to within about 1 part in 1000. However, rendering software does in fact use rather different algorithms for TTF and OTF -- that accounts for the differences you are seeing.

Let's keep looking at it.

[Sam Bull (dreamsorcerer)]

I am using the fixed-width length of a character in pixels, to decide where to draw the cursor in an input box. This makes it very easy to quickly see if there are any discrepancies, with only 1 pixel difference between characters as shown below, the cursor can visibly move to the wrong place after only a couple of characters entered into the input box. On all other mono-spaced fonts, this effect does not happen, the cursor renders correctly in the gap between two characters.

I have also confirmed this by printing out some of the sizes of characters and of strings of characters. In all other mono-spaced fonts, all characters have the same length, such as 10 pixels when rendered. And, if 3 different characters are rendered, it will always be 30 pixels wide (for a 10 pixel font). This means I can place the cursor at multiples of 10 pixels (or whatever) and have it always render in the correct place.

I've just got round to getting those lengths for you, using the below code.

lengths = {}
for c in string.letters + string.digits:
    lengths[c] = mono_font.render(c, False, (0,0,0)).get_size()[0]
print lengths

Result:
{'1': 10, '0': 10, '3': 10, '2': 10, '5': 10, '4': 10, '7': 10, '6': 10, '9': 10, '8': 10, 'A': 10, 'C': 10, 'B': 10, 'E': 10, 'D': 10, 'G': 10, 'F': 10, 'I': 9, 'H': 10, 'K': 10, 'J': 9, 'M': 10, 'L': 10, 'O': 10, 'N': 10, 'Q': 10, 'P': 9, 'S': 9, 'R': 10, 'U': 9, 'T': 9, 'W': 10, 'V': 10, 'Y': 9, 'X': 10, 'Z': 10, 'a': 10, 'c': 10, 'b': 10, 'e': 9, 'd': 10, 'g': 10, 'f': 10, 'i': 9, 'h': 10, 'k': 10, 'j': 10, 'm': 9, 'l': 9, 'o': 10, 'n': 10, 'q': 10, 'p': 10, 's': 11, 'r': 10, 'u': 10, 't': 10, 'w': 10, 'v': 10, 'y': 10, 'x': 10, 'z': 10}

As you can see, there's seems to be an inconsistent switch between 9 and 10 pixels wide, while 's' is even 11 pixels wide.

With all other mono-spaced fonts, all characters return the same size, for example with Ubuntu Mono all characters are 8 pixels, Droid Sans Mono is all 10 pixels and Liberation Mono is also all 10 pixels.

I'll ask on the mailing list again to see if any of the developers can add any information.

[Steve White (stevan-white)]

Hi Sam,

Is it possible there is some peculiarity of FreeMono that makes this happen on your system?
Well sure. Fonts are terribly complicated really.

But I don't think so.

I can tell you, the spacing glyphs of FreeMono are uniformly 600EM wide (including 'H' and 'I'--I just triple-checked.)
The flag indicating that it is monospaced is set (and another flag indicating it is "fixed pitch" is also set,
at least in the builds on the FreeFont site.)

And again, you are talking about rendered sizes. I have to say, it looks to me as if there is some misunderstanding somewhere, about what is to be done when scaling the integer EM size to a pt size. It can be tricky business. But this is not the business of the font, but rather the rendering software.

Could send me some code that illustrates the problem? Otherwise I'm quite in the dark here.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sam Bull (dreamsorcerer)]

Yes, it's looking more likely to be a problem with the rendering software, but we're still unsure if this is in Pygame, SDL or something else. We also don't know what is different about this font to cause the bug.

You can test some code with the little script below. You need to 'apt-get install python-pygame' first. It's also interesting to look at the metrics stuff, which is even more confusing (http://www.libsdl.org/projects/SDL_ttf/docs/SDL_ttf.html#SEC38). You can also look at the input box in my project at https://launchpad.net/simplegc.

import pygame
import string

pygame.font.init()

m = pygame.font.SysFont("FreeMono", 16)
#m = pygame.font.Font("/path/to/file.ttf", 16)
#print zip(m.metrics(string.letters), string.letters)
for c in string.letters:
    print "%s:%s, " % (c, m.render(c, False, (0,0,0)).get_size()[0]),

[Steve White (stevan-white)]

Hi, Sam

Thanks for the code! I'm having a look at it.

Try your script on "Courier New" (yes the non-free one packaged with Windows).

[Weeble (clockworksaint)]

I ran a few tests to try to narrow this down, but I don't have a Ubuntu 12.04 system yet, so they may be of limited use.

I used this test script:

http://pastebin.com/tgmShFQm

I ran it on Windows with two different versions of libfreetype (while keeping the same version of PyGame and everything else) and saw different results - with one version all glyphs showed the same "advance" metric, with the other they varied by two pixels over a wide range of font sizes. I ran it on two Ubuntu machines and in both cases all glyphs showed constant advance. However, in all cases I saw no difference between the two versions of FreeMono that I tried, 20100919 and 20090104, as linked above by Sam.

It would be interesting to see what the result is on 12.04. Here are the results from the systems I used:

http://pastebin.com/8SK3vRtT

About the script:

The script expects you to have the ttf fonts to test in the working directory. I've focused on the results of the font.metrics method, because as far as I can tell it correlates with the observed width of rendered glyphs, but it eliminates some potential confounding factors during rendering, such as kerning.

[Weeble (clockworksaint)]

Sorry, the results pastebin link didn't work. This one should:

http://pastebin.com/6gvWvMHj

[Weeble (clockworksaint)]

Okay, I built libfreetype from their git repo, installed it to /usr/local and reran my test:

$ LD_LIBRARY_PATH=/usr/local/lib/ python testcase.py
freefont-20090104-FreeMono.ttf at pixel size 16, advances:
   10 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

freefont-20090104-FreeMono.ttf at pixel size 100, advances:
   60 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

freefont-20090104-FreeMono.ttf at pixel size 200, advances:
   120 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

freefont-20090104-FreeMono.ttf at pixel size 300, advances:
   180 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

freefont-20100919-FreeMono.ttf at pixel size 16, advances:
   9 for 23 characters: #*+;IJPSTUY[]deilmq{|}~
   10 for 71 characters: !"$%&'(),-./0123456789:<=>?@ABCDEFGHKLMNOQRVWXZ\^_`abcfghjknoprtuvwxyz
   11 for 1 characters: s

freefont-20100919-FreeMono.ttf at pixel size 100, advances:
   59 for 11 characters: $?IS^fjrsuz
   60 for 80 characters: !"#&'()*+,-/0123456789;<=>@ABCDEFGJKLMNOPQRTUVWXYZ[\]_`abcdeghiklmnopqtvwxy{|}~
   61 for 4 characters: %.:H

freefont-20100919-FreeMono.ttf at pixel size 200, advances:
   120 for 71 characters: !"%&'(),.0123456789:;<=>@ABCDEFHJMNPRTUVWXYZ[]_`abcdefhjkmopqrtuvwyz{|
   121 for 13 characters: $*/?GIKL\^gns
   119 for 11 characters: #+-OQSilx}~

freefont-20100919-FreeMono.ttf at pixel size 300, advances:
   179 for 4 characters: =]xy
   180 for 77 characters: !"$&'()*,./0123456789:;<>?@BCDEFGIJKLMNOPQRUVWXYZ[\^_`abcdefghkmnqrstuvwz{|}
   181 for 14 characters: #%+-AHSTijlop~

This does indeed show what Sam observed: the two versions of FreeMono.ttf show different advances for some characters. So the weird behaviour only materializes with a recent version of FreeType and a specific version of FreeMono.ttf. Does that help us figure out what's happening and whether there's actually a bug somewhere?

[Steve White (stevan-white)]

Hi, Weeble,

Yes I too saw the same results as Sam.

Please try your script on "Courier New".

[Weeble (clockworksaint)]

I've narrowed it down. The change in behaviour was introduced with this commit in freetype: http://git.savannah.gnu.org/cgit/freetype/freetype2.git/commit/?id=8c82ec5b17d0cfc9b0876a2d848acc207a62a25a

The description is:

"Always ignore global advance.
This makes FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTH redundant,
deprecated, and ignored. The new behavior is what every major user
of FreeType has been requesting. Global advance is broken in many
CJK fonts. Just ignoring it by default makes most sense."

[Weeble (clockworksaint)]

I'm not in front of the Windows machine right now, but I can run the script on "/usr/share/fonts/truetype/msttcorefonts/Courier_New.ttf" on Ubuntu here. This is with a new version of libfreetype:

Courier_New.ttf at pixel size 16, advances:
   10 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

Courier_New.ttf at pixel size 100, advances:
   60 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

Courier_New.ttf at pixel size 200, advances:
   120 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

Courier_New.ttf at pixel size 300, advances:
   180 for 95 characters: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

[Steve White (stevan-white)]

Hi Weeble,

Regarding this FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTH,
Do you have more information on this, beyond the freetype patch submission?
What OpenType table field is it referring to?
(I would be interested to know if there is something to be improved in FreeMono!)

It is very odd that some fonts give more consistent results from these library calls, and others don't.
Maybe it has something to do with the very fine stem width of these Courier-like faces.

Again, it's unclear to me how a renderer can maintain a scaling from (say) 1000EM to 9px, and
retain the "monospace" property. A choice of rounding method would change the width of a line by more than 10%.
When a whole line of text is rendered, this can be accounted for by antialiasing, but then you're going to have some
interesting problems deciding just where the boundary lies between one letter an the next.

[Weeble (clockworksaint)]

I'm afraid I'm not sure. This is all new to me, I'm just investigating because I follow the pygame mailing list and found it interesting.

I do think it's interesting that the variation appears always to be +/-1 pixel, regardless of the scaling factor. I'll perhaps try some much larger scales and see if that pattern holds up.

I think at this point it might be worth discussing on the SDL and FreeType mailing lists. I'm not sure what FreeType functions SDL uses to determine an advance in pixels, and I still don't quite understand what information FreeType uses from the font file to determine advance widths.

[Steve White (stevan-white)]

Sam,

The effect not only happens with FreeMono and Courier New; it happens with DejaVu Sans Mono as well.
It just occurs at different point sizes.

This is not a FreeFont bug (if indeed it is a bug at all).

I'm willing to continue looking at the issue with you, but could you please move the report to a different package?

See attached.

[Steve White (stevan-white)]

Another experiment with pygame's font API. Looks very strange.

Printed out the width (in pixels, presumably) of a string of 100 letter 'x' rendered at different point sizes.

Now FreeMono is 1000 units high and 600 wide.
The point size is the height, and a point is 72 pt/in.
My screen's x-resolution is 125 DPI (from /var/log/Xorg.0.log)

s = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
for fontsize in range( 5, 25 ):
        m = pygame.font.SysFont("FreeMono", fontsize)
        print "(", fontsize, "pt) ", m.render(s, False, (0,0,0)).get_size()[0]

I would expect a number more like 1042px at 10 pt:
100 * 10 pt high * (0.6 FreeMono width/height) * (125 px/in ) / (72 pt/in)

But see below.
Totally don't understand the numbers I get.

I also looked at the "height" measure returned by get_size()... it gives at 10 pt
   13
while
   10 pt * (119 px/in Y-DPI) / (72 pt/in) = 16.52 px
These units are not pixels, I guess. I wonder what they are.
This and, maybe my arithmetic isn't right.

Widths of 100 x's at a range of point sizes.
----------------------------------------------------------
FreeMono:
( 5 pt ) 300
( 6 pt ) 400
( 7 pt ) 400
( 8 pt ) 500
( 9 pt ) 500
( 10 pt ) 600
( 11 pt ) 700
( 12 pt ) 700
( 13 pt ) 800
( 14 pt ) 800
( 15 pt ) 900
( 16 pt ) 1000
( 17 pt ) 1000
( 18 pt ) 1100
( 19 pt ) 1100
( 20 pt ) 1200
( 21 pt ) 1300
( 22 pt ) 1300
( 23 pt ) 1400
( 24 pt ) 1400

DejaVu Sans Mono:
( 5 pt) 301
( 6 pt) 400
( 7 pt) 400
( 8 pt) 500
( 9 pt) 500
( 10 pt) 600
( 11 pt) 700
( 12 pt) 700
( 13 pt) 800
( 14 pt) 800
( 15 pt) 900
( 16 pt) 1000
( 17 pt) 1000
( 18 pt) 1100
( 19 pt) 1100
( 20 pt) 1200
( 21 pt) 1300
( 22 pt) 1300
( 23 pt) 1400
( 24 pt) 1400

[Weeble (clockworksaint)]

I think this part of the Pygame API is a relatively thin wrapper around SDL. I am pretty sure that the size parameter to pygame.font.Font/SysFont is a pixel size, not a point size, to remain consistent with the rest of that system.

I'm pretty certain that I can remove Pygame from the equation. I'll do that when I have time. I also expect that SDL can be removed too, although that's more speculative. There's certainly something weird in FreeType that it can discern some difference between these fonts even when all the relevant glyph metrics are identical.

[Weeble (clockworksaint)]

Okay, I think I see what's happening.

Here's a C program that shows the behaviour only using FreeType: http://pastebin.com/Jm4XjtNR

Example:

$ gcc fonttest.c `freetype-config --cflags --libs` -o fonttest
$ ./fonttest freefont-20090104-FreeMono.ttf abcdefg
a:448
b:448
c:448
d:448
e:448
f:448
g:448
$ ./fonttest freefont-20100919-FreeMono.ttf abcdefg
a:448
b:384
c:448
d:384
e:448
f:384
g:384

The output widths are in 64ths of a pixel.

Now, if you change FT_LOAD_DEFAULT to FT_LOAD_NO_AUTOHINT or FT_LOAD_NO_HINTING, everything becomes the same width. Does FreeMono have hinting? It seems the FreeType's auto-hinting algorithm is rounding some glyphs up and some glyphs down. But it only appears to kick in for the 20100919 version, not the earlier ones. If you use FT_LOAD_FORCE_AUTOHINT then even the earlier FreeMono versions get inconsistent sizes.

As I don't know much about font rendering, I can't really answer these questions:

1. Is this expected behaviour from FreeType? Is it reasonable for hinting to cause a monospace font to have different advance widths for different glyphs?
2. Is FreeMono supposed to provide its own hints?

[Steve White (stevan-white)]

Hi Weeble,

First, I want to re-iterate that this is not a FreeMono issue. We saw the same effect using Sam's script on Courier New, and using the script I provided, showed the same effect happening in several monospace fonts, including DejaVu Sans Mono.
It happens with *all* of them -- Sam's first code just happened to pick out FreeMono.

It is good you have clarified for yourself, that the base renderer produces different values. (You should also confirm that it does this with all the other Mono fonts.)

I don't know much beyond this as to what algorithm is applied by each renderer.

The advance width is a specified number, which ought to be involved in the overall placement of letters in a string.
After that, the letters are rendered. Just which pixels are colored in this process is a very complicated question.

Depending on the antialiasing algorithms applied, hinting (and TrueType instructions) can definately have an effect.
Yes, FreeMono (as packaged on the GNU FreeFont site) is hinted and instructed.

I too don't understand the numbers being returned by some of these libraries, but (for example) the remark in your code "inconsistent hinting" leads me to think you haven't sorted out the notion of the glyph description as in the font file, vs the rendered graphic produced by the font renderer. (Maybe the authors of some of these libraries hadn't thought that through either.)

The point is: what makes you think that the renderer should preserve the width of each letter?
In what sense is it to be preserved, in light of rounding, etc?
This is not a rhetorical question: where is the documentation that explains this?

You know, renderers will do many tricks, such as making pixels lighter between letters to make them more distinct. What is the "width" of a rendered letter? The distance between the outermost pixel colored? Some rounded fraction of the advance width? And my question before: should the width of a string then be a multiple of the rounded fraction obtained for a single character, or (the number of characters in the string) * (advance width) / (size factor). The two would give different values, you see.

It's a messy question.

[Steve White (stevan-white)]

Hi Sam,

I have done some more investigating of this issue, based on Weeble's code, and can provide further information.

However, I feel that I am still defending FreeMono, although it has been established that the effect in question happens with many fonts, and that this has more to do with rendering software than the fonts itself.

If you would please move this issue to a more appropriate package (maybe libfreetype), I would be glad to participate!

As it is, I may continue to participate, but my mood is already deteriorating.

[Steve White (stevan-white)]

Weeble,

1) the current version of GNU FreeFont is 2012-05-03
    http://ftp.gnu.org/gnu/freefont/freefont-otf-20120503.tar.gz
    http://ftp.gnu.org/gnu/freefont/freefont-ttf-20120503.zip
    You may want to use these for further experiments.

2) I built and ran your program, and see similar results. But I ran it also
    on the current version of GNU FreeFont, and get constant values, for both
    the TTF and OTF versions (with FT_LOAD_DEFAULT).

3) The difference in your output in the 2010 version of FreeMono may be due
   partly to the internal representation of glyphs in OTF and TTF in the different versions.
   In the current version, and the 2009 version,
       TTF : TrueType quadratic splines.
       OTF : PS Type1 cubic splines.
   This was reversed in the 2010 release. The policy decision was mine, and has to do with which
   versions work best on different platforms, among other things.

4) Current TTF is (TrueType) hinted and instructed; OTF is only PS hinted.

5) I don't understand why the 2010 version of FreeMono gave varying results for TTF and
constant results for OTF. Could be the hinting settings. This may take some work to sort out.

6) The flipping between two values (448-384 = 64, is one pixel) in the results would be consistent with a pixel
being turned on or not. But then, one would expect a range of widths, or else all the same width... unless...
no, I can't see why. You would expect 'm' to come out wider than 'l'¸ but the numbers are reversed.
It does seem wrong.

7) The effects in question are not a problem with FreeMono. I perceive layers of misunderstandings.
    There may be bugs in the renderer calls. But the question is, again, exactly what results are *expected*?

Cheers!

[Weeble (clockworksaint)]

Sorry Steve, I didn't mean to pick on FreeMono. I agree that it's not even clear if there's *any* bug here, but there's certainly something surprising.

I also agree that the rendered width is a poor metric, for the reasons you describe. That's why my investigations have focused on glyph metrics, specifically the advance width in pixels as reported by FreeType after the face has been loaded and a scaling size selected.

I believe that even if we could find a way to get consistent advance widths out of FreeType, Sam would still need to change his program, because measuring the width of the rendered text does not seem like it could possibly work in all circumstances. For example, a glyph may have negative xMin or it may have xMax greater than the advance width, as in italic faces. Either the rendered text must have a total width greater than the sum of the advance widths of the characters involved, or it must have chopped off parts of the characters. Still, I am less concerned with whether Sam's program is correctly written than with whether it is possible to write his program correctly, and it seems to me that FreeType reporting differing advance widths is an issue that prevents this.

I will try my Python script and the C test on a wider range of fonts tonight. I apologise that I had not understood until now that the Courier example showed up the same issue.

Sorry, my comment "inconsistent hinting in monospaced font" was not deeply considered, and is clearly misleading. I do not claim to know anything about the hinting specified in the font file, I can only say that for some reason FreeType appears to decide that it should apply auto-hinting, and that the observed result of this auto-hinting process on this particular monospaced font is that different glyphs end up with different advance widths, and that for other font files, including older versions of the same font, for some reason FreeType appears *not* to apply auto-hinting.

I have asked about this on the FreeType mailing list: http://lists.nongnu.org/archive/html/freetype/2012-05/msg00029.html

Thanks for all your help so far Steve.

[Sam Bull (dreamsorcerer)]

OK, I've reassigned it as requested. This is just becoming mind-boggling to me now though, I don't think there's much more I can contribute to this bug report. I'll continue to watch it, and see if you two manage to come up with any solutions.

Worst case scenario, I can use the width from each individual character to work out where to place the cursor. I already do this for my labels using a non-mono font, I'd just prefer not to do it for something where the text is changing repeatedly (and has to be recalculated every time), as it's a little more CPU usage than being able to use a fixed number.

[Steve White (stevan-white)]

Thanks, Sam!

(Sorry for being testy.)

Your problem of where to put the cursor is kind of messy for general fonts (and I don't know the right answer -- probably one has to ask the font rasterizer). But for monospace fonts, there should be a comprehensible solution at some higher level, such as your pygame thing.

Here's the arithmetic problem, as I see it.

For simplicity, say the advance widths of all characters are equal in the font. Suppose you scale the font, so the scaled width is some non-integer multiple of the device pixel width.

Now you have made an API call to get the character width, and it returns an integer.

What does that integer mean? Maybe it's the non-integer multiple, rounded. Or maybe as I suggested it has something to do with which pixels are turned on in the rendered graphic. (And maybe there really is a bug--I'm still not sure about this.) In any case, the documentation ought to explain it -- but I checked, and it doesn't.

Suggestion:

To place your cursor between letters. Where should it go? If you just get the width of a single character, and multiply by the character string index, the cursor is likely to be badly misplaced, due to rounding.
It might be better to find the pixel width of a longer string, and divide by the number of characters to get a floating-point value, and use that as the single-character width. I the simple scenario, this should get you within a pixel of the expected position every time.

For completeness:

In general, the placement of the cursor in text is substantially more subtle than that. Even in a monospace font, it's permissible for a character to have 0 width. Furthermore, OpenType fonts can do various tricks such as ligature substitution, that change the glyphs that are actually rendered. For this, the only safe way to place the cursor is to ask the font rendering interface. (This is just a deduction--I've never done it myself.)

I don't know if Pygame has a representation of such an interface...

(And... let's please not talk about CPU cycles here. It's just embarrassing.)

Cheers!

[Sam Bull (dreamsorcerer)]

Unfortunately, that suggestion is no better. The rendered width of each character varies, so the there is no single value that can place the cursor in the correct place. If the character 'h' is 10 pixels and 'i' is 9 pixels, then 'hih' will be 29 pixels and the cursor would need to be placed at 0, 10, 19 and 29 pixels to make sense.

My solution for the labels, which works fine, is to find the rendered length of the first character, then the length of the first two characters, then the length of the first 3 characters and so on. Storing all these values allows me to work out the correct position.

Because there's quite a bit of repetition, this can take a little bit of CPU time. My only concern with the input box, is that it needs to recalculate all of this, everytime the user types something, and it is designed for use in games where a fraction of a millisecond can still be important to keep the game running smoothly.

[Weeble (clockworksaint)]

I ran the C test against every font on my Ubuntu 11.10 system with "Mono" or "Courier" in the name like so:

    for name in /usr/share/fonts/truetype/*/*Mono*.ttf; do ./fonttest $name abcdefgh > `basename $name`.testresult; done
    for name in /usr/share/fonts/truetype/*/*Courier*.ttf; do ./fonttest $name abcdefgh > `basename $name`.testresult; done

Here are the results:

    http://pastebin.com/QiwJsy8e

And here are the font versions I have installed:

    http://pastebin.com/4PTaBUpN

I only see variation in advance width for FreeMono. As Steve describes above, perhaps this behaviour is due to how FreeType handles different outline formats? I'm not sure.

I haven't had an answer yet on the FreeType mailing list, but I've added this to their bug tracker[1]. I've also taken discussion of how to position the cursor in Pygame back to the Pygame mailing list[2], as I was worried I'd just clog up this bug report with information irrelevant to any (possible) problem in Ubuntu software.

[1] - https://savannah.nongnu.org/bugs/index.php?36593
[2] - http://archives.seul.org/pygame/users/Jun-2012/msg00015.html

[Weeble (clockworksaint)]

This was posted on the FreeType bug:

"Both versions of FreeMono (20090104 and 20100919) are TTFs, the former has hints, the latter has not. Consequently, FreeType uses the auto-hinter for the latter.
Both fonts don't have the `isFixedPitch' flag set in the `post' table, and this is the very problem. While the TrueType hints in the 20090104 version take care of the advance width (and FreeType simply obeys), the 20100919 version the font doesn't tell the auto-hinter not to change advance widths.
In the original report, the font `Courier New' has been mentioned. This font has the `isFixedPitch' flag set.
My conclusion: It is a bug in the font in both versions, but it only shows up if the auto-hinter is used. So the central question is: Why is the `isFixedPitch' flag set to false in this font? " -- Werner LEMBERG

It looks like that solves the mystery. And since the newer 2012 version has isFixedPitch true, I think this is fixed upstream. How do we get this into Ubuntu?

[Steve Langasek (vorlon)]

based on the last message, reassigning back to the ttf-freefont package.

[Steve White (stevan-white)]

Hi Werner,

Now this has gotten really messy.
I don't know what we can do with it.

1) if the flag isFixedPitch is involved, it's going to get crazy.

     - This flag *is* set in the current version of FreeMono.
        (Why don't you download it and try it? -- Please! It has a lot of new stuff!)

    - I agree the flag *should* be set for a monospace font. It wasn't set in the old version
      of FreeMono, as you point out, because FontForge un-sets it.

    - I submitted a patch to FontForge that would cause it to *set* this flag to conform with the
     most obvious reading of the TrueType standards. But somebody has been objecting to it,
     with very dubious arguments and tactics. (This is a very unfortunate slant to the present issue.)

    - But I thought this flag only had an effect in Windows. Now you're telling me recent releases
      of FreeType *use* the flag??? Oh man!

2) Didn't we established that the original complaint, using PyGame, really arose in
    *any* monospace font (depending on the resolution)?

    I just ran a modified version of Sam's script, which looks for characters of varying width,
    in *his* sense. Again, it shows *all* the fonts have this issue.
    (On my Ubuntu 12.04 with stock FreeType)

    See attached, please!

   Was that incorrect?
   Or does it have to do with the FreeType version?

3) Are we 100% sure Sam's complaint is identical with this issue you're reporting,
     having to do with recent changes to FreeType?

4) None of the questions I asked has been answered:
     What units are these PyGame units anyway?
     In what sense is the monospaced property to be preserved in rendering?
     (Your recent results say to me, the FreeType group recently changed their policy regarding this.)

:P !

[Weeble (clockworksaint)]

I'll pass for 1). I don't think Werner has subscribed here - I posted a link back to here when I posted the Savannah bug and I copied his response here, but I don't think he's following this bug. Did you CC him?

With regard to 2), I think Sam's test based on rendered string dimensions did show variations in other monospaced fonts. I think I agree with you above that Sam could not be confident of consistent results with that method, such as any time xMin < 0 or xMax > advance. If you look at the reported glyph metrics for the characters that mismatch in that test, in all cases either xMin < 0, xMax > advance or both. However, while investigating Sam's problem, I found that in FreeMono the advance width varies too. Would it be better to split this into a separate bug? I definitely think this contributes to Sam's problem, but I think Sam's test is unable to distinguish between these two different causes.

With regard to 3), see 2).

With regard to 4), the Pygame Font & SysFont constructors are documented[1] as taking an integer number of pixels, they pass it through[2] to SDL's TTF_OpenFont[3][4], which then uses it to call FreeType's FT_Set_Char_Size[5] like so:

FT_Set_Char_Size( font->face, 0, ptsize * 64, 0, 0 );

I believe that specifies a default 72dpi, so that the pixel size = the point size.

The methods that get widths and heights back out of the font in Pygame are definitely in pixels. font.render(...) returns a bitmap object and get_size() returns its dimensions in pixels.

Hopefully this information reduces the amount of spelunking through Pygame and SDL needed for people investigating this.

[1] - http://www.pygame.org/docs/ref/font.html#pygame.font.Font
[2] - http://hg.pygame.org/pygame/src/e6b5ebf91ac4/src/font.c
[3] - http://www.libsdl.org/projects/SDL_ttf/docs/SDL_ttf.html#SEC14
[4] - http://hg.libsdl.org/SDL_ttf/file/c37a9410f6a7/SDL_ttf.c
[5] - http://www.freetype.org/freetype2/docs/reference/ft2-base_interface.html#FT_Set_Char_Size

[Werner Lemberg (wl)]

Steve,

disassembling the 20090104 version I see that some glyphs have different widths, for example `.notdef' and `.null'. FontForge is correct to not set the `isFixedPitch' flag. The same is true for the 20100919 version, where the `.null' glyph has a different width.

Regarding FontForge: I've followed the thread, and Khaled is right. I'm sorry that it became personal. Please bear in mind that the `isFixedPitch' flag comes from a very ancient time where non-spacing glyphs (let alone Unicode or OpenType) was something exotic. It's also unfortunate that the OpenType standard doesn't provide a better explanation for the flag which can't be misinterpreted.

Note that I only react to the input. Here on LaunchPad I'm just a guest observer,; I only take care of the FreeType bug report. It's my job to analyze the data and point out the problems. It's *not* my job to search the internet whether a better solution exists. Sometimes I know a better one, but usually I simply don't have enough time for that, sorry. Besides that it's good to know that recent versions of the font have this issue fixed!

[Werner Lemberg (wl)]

Oops, correction: Empty glyphs don't count in the calculation of the `isFixedPitch' flag. In the 20100919 version, glyph `acutecomb' and other, non-empty glyphs have a different advance width.

[Steve White (stevan-white)]

Werner,

Thanks for your input! I appreciate your position regarding the bug reports.

I disagree with most of what you say about the FontForge issue. I think
I argued my point cogently and completely elsewhere. Since you bring it
up, I will write to you privately on the matter (once I re-read that disturbed wad of material),
as it only obliquely touches on the present bug.

Unfortunately, what was one discussion is now three or four.
I'm running around putting out fires here.

Weeble,

First, it was very important for me to be aware of this change to FreeType.
It changes the way FreeFont is packaged.
I will have to take some actions (not sure what yet).
Thank you!

Although the FreeType issue would likely have an effect on the PyGame values,
it really seems to me that it is something separate.
Tell you what, if you like, open up a support issue, something like
    "change to FreeType handling of 'isFixedPitch'"
on the FreeFont site.
    https://savannah.gnu.org/projects/freefont/
That way you can be apprised of what I do with it.

 As to it being a bug per se, I don't see it that way. It isn't a bug that old versions
of FreeFont conflict with development versions of FreeType.

For the time being, I would like to better understand Sam's original problem.
Thanks for locating the germane code.
I hope I have time to settle the matter soon.

[Sam Bull (dreamsorcerer)]

Thanks to a bit of help from Weeble and some tweaking to my label method, I have a fast-enough solution for my toolkit. As it seems this probably isn't a bug, I've closed it. Thanks for all your help.