Non-ASCII characters get corrupted in framebuffer console

Bug #355057 reported by Uwe Geuder on 2009-04-04
24
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Low
Tim Gardner
Hardy
Low
Stefan Bader
Intrepid
Low
Stefan Bader
Jaunty
Low
Stefan Bader

Bug Description

SRU justification:

Impact: The use of 512 character fonts on a vga console is broken since
seemingly ever until now somebody noticed. While not being a panic spreading
issue it can simply be fixed.

Fix: Let the get_font function return the complete character set and not
only the first 256 characters, while pretending to have done all. Fix
sent to upstream (accepted into -mm)

Testcase: Call showconsolefont after loading a 512 character font and do not
panic (seems to be another bug in there), switch console and back and the
second half of the font is corrupted.

---

This issue has been reported several times before in various
forms. Typically about one national character missing. I try to
summarize all information I have collected in this new general report
and mark the earlier reports about special cases as duplicates.

Several non-ASCII characters get corrupted in the framebuffer
console. (see attached photos) This happens when returning from X back
to the console using Ctrl-Alt-F1 ... Ctrl-Alt-F6.

The characters are are displayed correctly when booting into single
user mode and not starting X. (see attached
intrepid-single-user-OK.jpg) So the problem is not incomplete font
implementation or some locale issue. This is why I report the bug to
the kernel instead of to console-setup, if feels that console-setup
should not cause random pixel "characters" (But of course the bug could
really be somewhere else. I'm far from understanding completely how
this all works.) I also attach my /etc/defaults/console-setup

Actually the problem is described in /usr/share/doc/console-setup/FAQ.gz
However, it says that this happens only in text mode, not in framebuffer
mode. So how can I find out in which mode my console is actually running?

I believe that my console should be in framebuffer mode, because:

$ lsmod | grep fb
fbcon 47648 0
tileblit 10880 1 fbcon
font 16512 1 fbcon
bitblit 13824 1 fbcon

However there is a file /etc/modprobe.d/blacklist-framebuffer and it says

# Framebuffer drivers are generally buggy and poorly-supported, and cause
# suspend failures, kernel panics and general mayhem. For this reason we
# never load them automatically.

So do we use fbcon, but not the underlaying framebuffer???
Some documentation is here http://www.mjmwired.net/kernel/Documentation/fb/fbcon.txt, but I'm still confused.

The problem is not specifc to the reported kernel version. I see it
also with the current Hardy kernel and the Jaunty Beta LiveCD (see
jaunty-liveCD.jpg). The problem is not specific to the reported
grahics card. I see it also on Nvidia, on Radeon, and also on the
virtualized card created by VMware. The previously reported bugs (now
linked as duplicates) cover also various hardware and software
configurations.

I have tested the range Unicode U+00A0 ... U+00FF, my console coding
is UTF-8. http://www.utf8-chartable.de/ can be used for
referece. (From the previousily reported national character issues I
conclude that also characters outside this range are affected)

You can easily test yourself using the attached file

$ cat sampletable.txt

The character corruption comes in 2 different forms:

- Random pixel character range: All characters between and including
  U+00E9 and U+00FF are corrupted. Often the pixels are completely
  random (See attached intrepid-random-pixel-range.jpg) and they are
  different on each console tty1, tty2, ... Sometimes they are just
  all white or black.

  The somehow weird thing is that the U+00E8 (e grave) is always the
  last character displayed correctly, whereas U+00E9 (e acute) is
  always the first corrupted character.

- missing accents. There is no random corruption in this case, but
  accents are missing from capital letters. It looks more like an
  intentional low-grade variant of the font. Accents are present on
  lowercase letters. A couple of very rare characters (e.g. thorn)
  are displayed as white blocks. (see attached
  intrepid-no-accents.jpg) I guess this variant of the problem may
  lead to the belief that a certain configuration is not affected, if
  you just test a couple of lowercase characters.

The same kernel on the same machine can sometimes show "random pixel
character range" and sometimes "missing accents". E.g. when boot first
into single user mode and start your X later in contrast of starting
everything the standard way. (intrepid-random-pixel-range.jpg and
intrepid-no-accents.jpg are from the same machine using the same kernel)

P.S. Not sure how important this problem really is. I started to dig
when investigating some serious display corruption (affecting also X
on framebuffer 7) So originally I thought if even the console gets
corrupted things are really bad. But in the meantime I have come to
the conclusion that this character corruption is most likely a
completely different issue from my original problem. For me personally
non-ASCII characters on the console are of minor importance. On the
other side according to Ubunutu philosophy software should be
available in the user's local language. This of course requires that
national characters work, if somebody works on the console.

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 8.10
Package: linux-image-2.6.27-14-generic 2.6.27-14.30
ProcCmdLine: root=UUID=f6bf9b65-0cad-45bd-b3a0-a7353a3900af ro quiet splash
ProcEnviron:
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
 LANG=fi_FI.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.27-14.30-generic
SourcePackage: linux

Uwe Geuder (ubuntulp-ugeuder) wrote :
Uwe Geuder (ubuntulp-ugeuder) wrote :
Uwe Geuder (ubuntulp-ugeuder) wrote :
Uwe Geuder (ubuntulp-ugeuder) wrote :
Uwe Geuder (ubuntulp-ugeuder) wrote :

Oops launchpad does not really display the file name.

Picture 1/4 was intrepid-no-accents.jpg
Picture 2/4 was intrepid-random-pixel-range.jpg

Uwe Geuder (ubuntulp-ugeuder) wrote :
Uwe Geuder (ubuntulp-ugeuder) wrote :

There are 2 more reports, which I originally thought they report the same issue. However, after reading them again, I decided not to mark them as duplicates, because they contain also other aspects, which might be unique to them. For cross reference, the links are https://launchpad.net/bugs/190746 and https://launchpad.net/bugs/273189

Uwe Geuder (ubuntulp-ugeuder) wrote :

Oops, couldn't read my own handwritten notes and the previous comment got wrong...

These 2 reports might contain duplicate aspects (but also some other things)

https://launchpad.net/bugs/273189
https://launchpad.net/bugs/303561

This one sounds somehow loosely related to the topic (but not a duplicate)

https://launchpad.net/bugs/190746

Sorry about the noise.

Stefan Bader (smb) on 2009-04-15
Changed in linux (Ubuntu):
assignee: nobody → stefan-bader-canonical
importance: Undecided → Low
status: New → Triaged
Stefan Bader (smb) wrote :

Gosh, I think I found the little bugger. It seems this has been that way near forever but probably did not always matter (as long as your font only has 256 characters). The problem lies in the function that returns the font data of a vga console. It will only write back the first 256 characters but claims the returned font has 512 (as it had). So after every switch of the console there is the chance of corruption.
Some of my systems seemed to have initially 256 character fonts, so your example text and also those special characters on my keyboard where fine. Just after loading a font with 512 characters in it you will immediately see the corruption.
The attached patch fixed the problem for me. I will also upload some Jaunty based kernels with that fix, so you could test with them. Should this work for you as well, would it be ok to add you with your email as a "Tested-by"?

The kernels are at http://people.ubuntu.com/~smb/bug355057

Uwe Geuder (ubuntulp-ugeuder) wrote :

Hmm, your kernels are for Jaunty. I have only Hardy and Intrepid systems. I have reproduced the problem in Jaunty by using the live CD. But I guess I cannot update the kernel when using the live CD, can I?

I can probably install a Jaunty to some disk corner, but I might take a couple of days or week until I find the time.

Thanks for working on the patch!

Uwe Geuder wrote:
> Hmm, your kernels are for Jaunty. I have only Hardy and Intrepid

Maybe simpler: I added Intrepid kernels (based on proposed) to the same place.

Uwe Geuder (ubuntulp-ugeuder) wrote :

Thanks for the intrepid versions. I've just tested the i386 version in the same machine as this report was created (see attached files above).

No character corruption occurs anymore in my sampletable.txt in tty1 ... tty6. Correct characters seem to be displayed all the time. (Well I have not double checked every single character with a Unicode table. But those characters I happen to know by heart are correct and all other look correct to me)

However, the characters previously corrupted appear in a different color now. While all characters, which worked before are displayed in some kind of grey, the "new" characters are displayed in white. Any explanation for this? The problem is already mentioned in https://bugs.launchpad.net/ubuntu/bugs/303561

The setfont(8) man page mentions that only 8 instead of 16 can be displayed if a font has more than 256 glyphs. But I understand this affects the whole console and not only the last 256 glyphs. If this understanding is correct, the reason should not be that the "default grey" is no longer available, because it is still used for all "have always worked" characters.

Uwe Geuder (ubuntulp-ugeuder) wrote :

Oops, broken link in previous comment. New attempt: https://bugs.launchpad.net/bugs/303561

Stefan Bader (smb) wrote :

> No character corruption occurs anymore in my sampletable.txt in tty1 ...
> tty6. Correct characters seem to be displayed all the time. (Well I have
> not double checked every single character with a Unicode table. But
> those characters I happen to know by heart are correct and all other
> look correct to me)

Great to hear.

> However, the characters previously corrupted appear in a different color
> now. While all characters, which worked before are displayed in some
> kind of grey, the "new" characters are displayed in white. Any
> explanation for this? The problem is already mentioned in
> https://bugs.launchpad.net/ubuntu/bugs/303561

> The setfont(8) man page mentions that only 8 instead of 16 can be
> displayed if a font has more than 256 glyphs. But I understand this
> affects the whole console and not only the last 256 glyphs. If this
> understanding is correct, the reason should not be that the "default
> grey" is no longer available, because it is still used for all "have
> always worked" characters.

The man page is correct and it _will_ only affect the last 256 glyphs. In fact
these characters are always using a different colour. This is due to the "old
days" when memory was expensive and engineers squeezed much into little.
Simply said, as you can only encode 256 values into a byte, you are missing a
bit for 512 character fonts. In order not to waste bytes, this bit is taken
from the colour index nibble of the attribute byte. So the lower 256 characters
use the colours with index 0-7 and the upper 256 characters the colours with
index 8-15.
The trick is to mirror the colours indexed by 0-7 to the ones indexed by 8-15.
I would have to look whether this is done in the console driver. It is a bit
hard to see for me as on my LCD this seems to be ntsc (nearly the same colour),
so I never saw that difference.

--

When all other means of communication fail, try words!

Uwe Geuder (ubuntulp-ugeuder) wrote :

> The man page is correct and it _will_ only affect the last 256 glyphs.

You are right, I didn't read it carefully enough and understood it wrong.

> The trick is to mirror the colours indexed by 0-7 to the ones indexed by 8-15.

If this were possible it would be another bug, because it does clearly not happen at least for my console.

Or is there some limitation that the same color can only be used once and you have to use "similar" colors? Then people with poor monitors would have an advantage :)

As said before I don't think this is high priority for me personally, because I prefer to use
computers in English (as long as I can decide myself) and work in X11 (as long as I don't have to do trouble shooting). But if Ubuntu intends to be a localized system, it might of course be nice to eventually get the console working flawlessly also for languages using non-ASCII letters after so many years.

Uwe Geuder (ubuntulp-ugeuder) wrote :

Forgot to answer...

> would it be ok to add you with your email as a "Tested-by"

I'm not familiar with LKML policies and I couldn't find anything in the FAQ at least quickly. If looking at some characters on the console can be considered testing in that context, it's OK to add my email.

However, if this somehow includes the promise that this does not cause any regression, I would not feel comfortable to make such promise after such little usage and with my non-existing backgound knowledge about the involved code.

Stefan Bader (smb) wrote :

IMO it can only cover that it fixed the issue for you. But I ask someone else
to be safer.

Stefan Bader (smb) wrote :

> However, if this somehow includes the promise that this does not cause
> any regression, I would not feel comfortable to make such promise after
> such little usage and with my non-existing backgound knowledge about the
> involved code.

I double checked, this only means you looked at the problem and it has been
fixed for you. There is no promise of checking for regressions of any sort.

Uwe Geuder (ubuntulp-ugeuder) wrote :

OK, so please feel free to add my email as tested-by.

Steve Beattie (sbeattie) on 2009-04-27
Changed in linux (Ubuntu Jaunty):
importance: Undecided → Low
status: New → Triaged
Changed in linux (Ubuntu Intrepid):
importance: Undecided → Low
status: New → Triaged
Changed in linux (Ubuntu Hardy):
importance: Undecided → Low
status: New → Triaged
Stefan Bader (smb) wrote :

Committed to Jaunty (target Ubuntu-2.6.28-12.43)

description: updated
Stefan Bader (smb) wrote :

Committed to Jaunty (target 2.6.28-12.43)

Changed in linux (Ubuntu Jaunty):
status: Triaged → Fix Committed
Stefan Bader (smb) wrote :

Fix committed to Intrepid (target 2.6.27-14.34)

Changed in linux (Ubuntu Intrepid):
assignee: nobody → stefan-bader-canonical
status: Triaged → Fix Committed
Changed in linux (Ubuntu Jaunty):
assignee: nobody → stefan-bader-canonical
Stefan Bader (smb) wrote :

Committed to Hardy (target 2.6.24-24.54)

Changed in linux (Ubuntu Hardy):
assignee: nobody → stefan-bader-canonical
status: Triaged → Fix Committed
Stefan Bader (smb) wrote :

Has been committed to Karmic

Changed in linux (Ubuntu):
assignee: stefan-bader-canonical → timg-tpi
status: Triaged → Fix Committed
Tim Gardner (timg-tpi) wrote :

This patch was included in the first Karmic upload.

linux (2.6.30-1.1) karmic; urgency=low

  * Initial release after rebasing against v2.6.30-rc3

 -- Tim Gardner <email address hidden> Thu, 12 Mar 2009 19:16:07 -0600

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Steve Langasek (vorlon) wrote :

Accepted linux into jaunty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed
aleksey (shaferov) wrote :

This is very important problem!
I cant work in console because Russian characters are broken.
It is impossible to work with translated programs (eg midnight commander) and to work with Russian filenames.
Ls command returns list of garbage.
Its looks like characters on intrepid-random-pixel-range.jpg.

P.S.
Sorry for bad English.

Martin Pitt (pitti) wrote :

Accepted linux into intrepid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 2.6.28-13.44

---------------
linux (2.6.28-13.44) jaunty-proposed; urgency=low

  [ Stefan Bader ]

  * Revert "SAUCE: [jaunty] ALSA: Add retry for Intel8x0 clock measurement"
  * Revert "SAUCE: [jaunty] ALSA: Fix clock and buffer calculations for
    Intel8x0"
  * Revert "SAUCE: [jaunty] ALSA: Fix buffer positions and checks"

linux (2.6.28-12.43) jaunty-proposed; urgency=low

  [ Amit Kucheria ]

  * Enable SYN_COOKIES for iop32x and versatile flavours
    - LP: #361687
  * SAUCE: Quirk for BT USB device on MacbookPro to be reset before use
    - LP: #332443

  [ Brad Figg ]

  * [jaunty] Add missing mvsas (Marvel SAS 6440) module configuration.
    - LP: #352336

  [ Chuck Short ]

  * SAUCE: [USB] Unusual Device support for Gold MP3 Player Energy
    - LP: #125250

  [ Daniel T Chen ]

  * SAUCE: [jaunty] ALSA: Fix buffer positions and checks
    - LP: #345627
  * SAUCE: [jaunty] ALSA: Fix clock and buffer calculations for Intel8x0
    - LP: #345627
  * SAUCE: [jaunty] ALSA: Add retry for Intel8x0 clock measurement
    - LP: #345627

  [ Luke Yelavich ]

  * disable CONFIG_SND_HDA_INPUT_BEEP on amd64 and i386
    - LP: #331589

  [ Makito SHIOKAWA ]

  * [ARM] 5404/1: Fix condition in arm_elf_read_implies_exec() to set
    READ_IMPLIES_EXEC
    - LP: #364358

  [ Manoj Iyer ]

  * SAUCE: Added quirk to fix key release for Samsung NC20
    - LP: #360247

  [ Oleg Nesterov ]

  * posix timers: fix RLIMIT_CPU && fork()
    - LP: #361508

  [ Scott James Remnant ]

  * [Config] Disable libusual and the ub driver
    - LP: #364538

  [ Stefan Bader ]

  * Disable unwanted staging builds
    - LP: #366144
  * Remove prism2_usb driver from ubuntu and use the one from staging
    - LP: #325366
  * SAUCE: Make rtl8187se depend on WIRELESS_EXT
    - LP: #366144
  * Disable CONFIG_RTL8187SE for armel.versatile
    - LP: #366144

  [ Tejun Heo ]

  * libata: handle SEMB signature better
    - LP: #257790

  [ Tim Gardner ]

  * Set USB_SERIAL=m for i386/amd64
    - LP: #345002
  * SAUCE: Jaunty - aic79xx - set reset delay to 5 seconds, down from 15.
    - LP: #79542
  * SAUCE: (drop after 2.6.28) Wifi suspend/resume scan timeout fixes
    - LP: #336055
  * Sony laptop: Sony Vaio laptops do not enable wwan power by default.
    - LP: #364678

  [ Tyler Hicks ]

  * SAUCE: (drop after 2.6.28) eCryptfs: Larger buffer for encrypted
    symlink targets
    - LP: #357345

  [ Upstream Kernel Changes ]

  * V4L/DVB (9999): gspca - zc3xx: Webcam 046d:089d added.
    - LP: #326674
  * V4L/DVB (10044): gspca - pac7311: Webcam 093a:2620 added.
    - LP: #363195
  * hwmon: (it87) Add support for the ITE IT8720F
    - LP: #357766
  * vgacon: Return the upper half of 512 character fonts
    - LP: #355057
  * drm/i915: add support for G41 chipset
    - LP: #365958

 -- Stefan Bader <email address hidden> Mon, 25 May 2009 17:30:40 +0200

Changed in linux (Ubuntu Jaunty):
status: Fix Committed → Fix Released
Norm Pierce (npierce-at2a) wrote :

Uwe Geuder wrote on 2009-04-20:

". . . the characters previously corrupted appear in a different color now. While all characters, which worked before are displayed in some kind of grey, the "new" characters are displayed in white. Any explanation for this? . . ."

Yes, that is LP:392795.

Steve Beattie (sbeattie) wrote :

I was able to reproduce this issue with a previous hardy kernel, 2.6.24-23.52, and can confirm that the kernel in hardy-proposed, 2.6.24-24.56, improves the handling of these characters. One issue I did notice is that the screen blanking due to inactivity doesn't end up applying to the characters with the brightness bit set; however, I've verified this also happens with the unpatched kernel, so it's not a regression introduced here (ah, that's bug 395374). I consider this verification-done for hardy.

Steve Beattie (sbeattie) wrote :

I have also reproduced this issue with the kernel in intrepid, 2.6.27-14.35, and can confirm that the kernel in intrepid-proposed, 2.6.27-14.36, fixes the issue (though showconsolefont has some interesting artifacts in both kernels). Marking verification-done.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 2.6.24-24.56

---------------
linux (2.6.24-24.56) hardy-proposed; urgency=low

  [Stefan Bader]

  * Rebuild of 2.6.24-24.54 with 2.6.24-24.55 security release applied

linux (2.6.24-24.54) hardy-proposed; urgency=low

  [Andy Whitcroft]

  * SAUCE: do not make sysdev links for processors which are not booted
    - LP: #295091

  [Brad Figg]

  * SAUCE: Add information to recognize Toshiba Satellite Pro M10 Alps Touchpad
    - LP: #330885
  * SAUCE: Add signatures to airprime driver to support newer Novatel devices
    - LP: #365291

  [Stefan Bader]

  * SAUCE: vgacon: Return the upper half of 512 character fonts
    - LP: #355057

  [Upstream Kernel Changes]

  * SUNRPC: Fix autobind on cloned rpc clients
    - LP: #341783, #212485
  * Input: atkbd - mark keyboard as disabled when suspending/unloading
    - LP: #213988
  * x86: mtrr: don't modify RdDram/WrDram bits of fixed MTRRs
    - LP: #292619
  * sis190: add identifier for Atheros AR8021 PHY
    - LP: #247889
  * bluetooth hid: enable quirk handling for Apple Wireless Keyboards in
    2.6.24
    - LP: #227501
  * nfsd: move callback rpc_client creation into separate thread
    - LP: #253004
  * nfsd4: probe callback channel only once
    - LP: #253004

 -- Stefan Bader <email address hidden> Sat, 20 Jun 2009 00:14:36 +0200

Changed in linux (Ubuntu Hardy):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 2.6.27-14.41

---------------
linux (2.6.27-14.41) intrepid-proposed; urgency=low

  [ Stefan Bader ]

  * Revert "SAUCE: input: Blacklist digitizers from joydev.c"
    - LP: #300143

linux (2.6.27-14.40) intrepid-proposed; urgency=low

  [ Amit Kucheria ]

  * Disable DEVKMEM for all archs on Intrepid
    - LP: #354221
  * SAUCE: Quirk for BT USB device on MacbookPro to be reset before use
    - LP: #332443

  [ Andy Isaacson ]

  * LIRC_PVR150: depends on VIDEO_IVTV
    - LP: #341477
  * SAUCE: FSAM7400: select CHECK_SIGNATURE
    - LP: #341712

  [ Andy Whitcroft ]

  * SAUCE: hotkey quirks for various Zepto Znote and Fujitsu Amilo laptops
    - LP: #330259
  * SAUCE: unusual devs: add an entry for the ScanLogic SL11R-IDE 0.78
    - LP: #336189

  [ Anton Veretenenko ]

  * SAUCE: sony-laptop: add support for Sony Vaio FW series function/media
    keys
    - LP: #307592

  [ Ayaz Abdulla ]

  * SAUCE: forcedeth: msi interrupt fix
    - LP: #288281

  [ Chuck Short ]

  * SAUCE: [USB] Unusual Device support for Gold MP3 Player Energy
    - LP: #125250

  [ Ike Panhc ]

  * squashfs: correct misspelling
    - LP: #322306
  * SAUCE: Fixing symbol name in HECI module
    - LP: #336549
  * Copy header files for various kernel media driver
    - LP: #322732

  [ Stefan Bader ]

  * SAUCE: vgacon: Return the upper half of 512 character fonts
    - LP: #355057
  * SAUCE: input: Blacklist digitizers from joydev.c
    - LP: #300143

  [ Upstream Kernel Changes ]

  * libata: make sure port is thawed when skipping resets
    - LP: #269652
  * x86-64: fix int $0x80 -ENOSYS return
    - LP: #339743
  * rt2x00: Fix race conditions in flag handling
    - LP: #258985
  * USB: cdc-acm: Add another conexant modem to the quirks
    - LP: #323829
  * Bluetooth: Add fine grained mem_flags usage to btusb driver
    - LP: #268502
  * Bluetooth: Handle bulk URBs in btusb driver from notify callback
    - LP: #268502
  * Bluetooth: Submit bulk URBs along with interrupt URBs
    - LP: #268502
  * hwmon: (abituguru3) Match partial DMI board name strings
    - LP: #298798
  * x86: mtrr: don't modify RdDram/WrDram bits of fixed MTRRs
    - LP: #292619
  * sis190: add identifier for Atheros AR8021 PHY
    - LP: #247889
  * ath9k: implement IO serialization
    - LP: #373034
  * ath9k: AR9280 PCI devices must serialize IO as well
    - LP: #373034
  * acer-wmi: fix regression in backlight detection
    - LP: #333386

 -- Stefan Bader <email address hidden> Wed, 26 Aug 2009 11:48:11 +0200

Changed in linux (Ubuntu Intrepid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers