Comment 2 for bug 331375

As I also am being affected by this, maybe I can give some extra info.

The file /etc/X11/rgb.txt used to be installed as part of x11-common (and still is for Debian unstable).
Recent versions no longer include the file, but still install a (broken) symlink to it
 See /usr/share/doc/x11-common/changelog.gz:
    xorg (1:7.4~1) experimental; urgency=low
      [ Julien Cristau ]
   * X.Org 7.4 release candidate
   ...
   * xserver 1.5 doesn't use an external rgb database, don't bother installing

Unfortunately, it's not only xserver that is affected by this file. Its lack breaks netpbm, xcolors, vncserver, and so on.
See other related bugs:
      #304567 #323160 Bug #300935