cached compose data is not available

Bug #221065 reported by Michael Hofmann on 2008-04-23
4
Affects Status Importance Assigned to Milestone
X.Org X server
Fix Released
Wishlist
libx11 (Ubuntu)
Medium
Bryce Harrington

Bug Description

Binary package hint: libx11-data

http://bugs.freedesktop.org/show_bug.cgi?id=3104 describes the compose-table-cache thats available in X11. As far as I can see this is not used in Ubuntu (hardy). Because it can improve startup time (40-200ms) and memory use (240k) for every X application, maybe the postinst script of libx11-data (which contains the compose tables) could generate these caches during installation. A blog post describing the compose table cache in less technical terms is at http://kdemonkey.blogspot.com/2008/04/magic-trick.html.

Created an attachment (id=2509)
Pointerless compose data structure patch

This patch redefines the compose data structures to use indices instead of
pointers. This is necessary to be able to mmap() the internal data
structures instead of parsing the compose table. Memory requirements are a
bit higher than before (I cannot realloc() for every single binding, so I
multiply by 1.5 each time space runs out), but performance is equivalent.

Tested this patch with all locales and input methods known to me and finally
works with SCIM again as well. Cache files are endian-safe and
typesize-safe (both encoded into filenames) and versioned.

There are a couple of issues to be thought about, though:

- I'm not exactly sure about the differences between INT32 and BITS32 types.
  I chose these types for creating fixed sized structures.

- I defined the type of modifiers in the structure to be BITS32. It was just
  'unsigned' before. Yes, no int or anything else.

- How much memory should I allocate in the very beginning? I chose numbers
  so that the 'C' locale bindings just fit nicely into memory.

- I have *not* tested the new XOpenFileMode function that is needed for
  Windows only (_XOpenFileMode) due to lack of an according development
  environment.

- Memory that has been allocated during the creation of the structure is not
  freed again. This could happen, when the Compose files contain errors or
  overwrites already defined keysyms.

- The Thai imput method is inherrently broken, and we should think about
  dropping it altogether.

Created an attachment (id=2510)
Compose cache patch

This is the compose cache itself.

The parsed compose tables are stored in /var/X11R6/compose-cache/ (global
cache) or ~/.compose-cache/ (local cache), respectively. Files in the local
cache expire once per day. File dates are compared with the compose table
source files as well. suid and sgid programs are not allowed to use the
local cache.

Caches currently depend on the used locale (more exactly: on the encoding),
but this isn't respected in the cache file format. This has to be changed.
So at least for global cache files, only UTF8 compose files should be
considered.

For user caches this is less problematic as clashes might only occure if the
user
a) has created a .compose-cache
b) uses different locales at the same time
c) these locales use the same compose table
d) parsing of the compose table differs for the used
   locales

Issues to be discussed:

- The per-user cache directory ~/.compose-cache/ is not created if it does
  not exist. In this case non-system compose tables are not cached at all.
  Should it be created automatically if it does not exist?
  I personally doubt this is really usefull.

Created an attachment (id=2511)
Helper program for creating global cache files

This tool creates a cache file for a given locale and compose file.
Unfortunately, due to the way XIM is integrated into libX11, a Xserver
has to be running in order to create a cache. If somebody sees a clearer
way to deal with this you're very welcome to change it.

For distributors I also add an example script below that shows how global cache

files for the three UTF8 compose tables can created using Xvfb. This is only a
skeleton, though.

Currently paths are hardcoded. Also note that compose tables for non-UTF8
locales currently shouldn't be cached globaly, as the encoding may differ
with different locales. This is something to be addressed.

------> CUT <-------
#!/bin/sh

if [ $# -gt 1 ]; then
  echo "$0 [x11-root]"
  exit 0
fi

if [ $# -eq 1 ]; then
  ROOT=$1
fi

mkdir -p $ROOT/var/X11R6/compose-cache

tmpfile=$(mktemp /tmp/Xvfb.log.XXXXXXXXXX)
$ROOT/usr/X11R6/bin/Xvfb \
  -fp $ROOT/usr/X11R6/lib/X11/fonts/misc/ \
  -sp $ROOT/etc/X11/xserver/SecurityPolicy \
  -co $ROOT/usr/X11R6/lib/X11/rgb \
  :99 &> $tmpfile &
trap "kill $!; rm $tmpfile || true" EXIT

export DISPLAY=:99
export LD_LIBRARY_PATH=$ROOT/usr/X11R6/lib64:$ROOT/usr/X11R6/lib
export XLOCALEDIR=$ROOT/usr/X11R6/lib/X11/locale/
sleep 5
$ROOT/usr/X11R6/bin/xbiff &> /dev/null &

pushd $ROOT/

usr/X11R6/bin/mkcomposecache "en_US.UTF-8"
"usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose" "var/X11R6/compose-cache"
"/usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose"
usr/X11R6/bin/mkcomposecache "pt_BR.UTF-8"
"usr/X11R6/lib/X11/locale/pt_BR.UTF-8/Compose" "var/X11R6/compose-cache"
"/usr/X11R6/lib/X11/locale/pt_BR.UTF-8/Compose"
usr/X11R6/bin/mkcomposecache "el_GR.UTF-8"
"usr/X11R6/lib/X11/locale/el_GR.UTF-8/Compose" "var/X11R6/compose-cache"
"/usr/X11R6/lib/X11/locale/el_GR.UTF-8/Compose"

popd

chmod 444 $ROOT/var/X11R6/compose-cache/*

Egbert asked for it :)

Adding patch keyword. Any plan for merging?

We're in the middle of a release phase. After that I plan to address a couple of
minor issues I'm seeing with my patch, and merge it.

7.1 is out now, so, ping?

Next thing to work on for me. Might be distracted a bit right now due to SLES 10.
The code still needs some cleanup, especially cache file names should include
encoding.

Commited & released.

Michael Hofmann (mh21) wrote :

Binary package hint: libx11-data

http://bugs.freedesktop.org/show_bug.cgi?id=3104 describes the compose-table-cache thats available in X11. As far as I can see this is not used in Ubuntu (hardy). Because it can improve startup time (40-200ms) and memory use (240k) for every X application, maybe the postinst script of libx11-data (which contains the compose tables) could generate these caches during installation. A blog post describing the compose table cache in less technical terms is at http://kdemonkey.blogspot.com/2008/04/magic-trick.html.

Timo Aaltonen (tjaalton) wrote :

This was discussed briefly with the Debian X team, and they agree that it would be good to have. We'll probably have this for 9.04.

Changed in libx11:
importance: Undecided → Medium
status: New → Confirmed

Out of interest, why was the actual compose cache patch applied, but not the patch which adds the helper program (mkcomposecache)?

Changed in xorg-server:
status: Unknown → Fix Released
Martin Pitt (pitti) wrote :

The fix was committed upstream in 2006, it's just weird why they didn't apply the mkcomposecache helper program. I asked upstream about it.

Other than that, it would fit well into the Jaunty meme of faster boot speed.

Bryce Harrington (bryce) on 2009-03-25
Changed in libx11 (Ubuntu):
assignee: nobody → bryceharrington
Bryce Harrington (bryce) wrote :

mkcomposecache is a binary utility, maybe it just seemed inappropriate to put it into a library?

The code looks pretty antique; it still uses Imake...! The code will need to be updated to current, given a new makefile, integrated somewhere (maybe it'd fit better into x11-xserver-utils?) and tested out thoroughly.

I'm also wondering whether mkcomposecache is actually necessary. From the blog post above it sort of sounds like simply creating the directory is sufficient, maybe that's something we all could test out... Looks like it expects this to be located at /var/cache/libx11/compose/

Bryce Harrington (bryce) wrote :

$ gcc mkcomposecache.c -lX11 -o mkcomposecache

$ ./mkcomposecache
Usage: ./mkcomposecache <Locale> <ComposeFile> <CacheDir> [<InternalName>]

Bryce Harrington (bryce) wrote :

MKCOMPOSECACHE(1) MKCOMPOSECACHE(1)

NAME
       mkcomposecache - create a cache file for a compose file explicitely

SYNOPSIS
       mkcomposecache locale compose-file cache-dir [ internal-name ]

DESCRIPTION
       mkcomposecache creates a cache file for the specified compose table and
       saves it in cache-dir. If internal-name has been spezified, it is used
       as the name of the compose table for computing the hash values and
       internal validation. This is necessary if global compose cache files
       shall be created while using DESTDIR during X11 installation. Caches may
       depend on the used locale, UTF-8 based locales are safe, though.

EXAMPLES
       mkcomposecache en_US.UTF-8 /var/tmp/build-
       root/usr/lib/X11/locale/en_US.UTF-8/Compose /var/tmp/build-
       root/var/X11R6/compose_cache /usr/lib/X11/locale/en_US.UTF-8/Compose

FILES
       /var/X11R6/compose_cache Location of global compose cache.

                                __xorgversion__ MKCOMPOSECACHE(1)

Bryce Harrington (bryce) on 2009-09-02
tags: added: hardy
Bryce Harrington (bryce) wrote :

xorg-server patched in lucid to cache xkbcomp data.

Changed in libx11 (Ubuntu):
status: Confirmed → Fix Released
Changed in xorg-server:
importance: Unknown → Wishlist
Changed in xorg-server:
importance: Wishlist → Unknown
Changed in xorg-server:
importance: Unknown → Wishlist
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.