tl;dr - There is a bug in xfsettingsd that not only affects external monitors, but primary monitors as well.

I believe that I am encountering the same bug, but by a different scenario. This may also be the same bug as  #1313539.

I am running Xubuntu 14.04 on two of my four computers. I have all four computers connected to a single keyboard, monitor and mouse via a 4port KVM switch. Prior 14.04 upgrade, this setup worked like a dream. Post 14.04 upgrade, I have found that switching to either of the Xubuntu machines results in a blank monitor. On one machine, I am usually able to get to a virtual console and reset of the video by restarting lightdm. On the other machine, I am not able to even get a virtual console, and I have to ssh into the machine in order to restart lightdm.

The dying video happens so frequently that I starting the process of migrating away to XFCE. I installed Gnome Shell on one machine and Fluxbox on the other. Neither of these WM's exhibited the dying video phenomena.

Based upon this bug report, I decided to investigate the possibility that xfsettingsd might be the problem. Here's what I did.

1) Log in to a Xubuntu session.

2) Open a terminal window. Because I needed to do part of this from the Xubuntu session and part of this via ssh, I opened a new Byobu session (screen or tmux would work too). Then,

    $ killall xfsettingsd
    $ XFSETTINGSD_DEBUG=1 xfsettingsd --replace --no-daemon 2>&1 | tee output.txt

3) After the initial output from xfsettingsd had finished,

    $ cp output.txt before.txt

4) Here's where the fun begins: I unplugged the monitor from the KVM switch. I waited a few seconds, and then I plugged the monitor back into the switch. No video. I switched the KVM to one of the other computers and verified that the monitor was corrected plugged in and working.

5) On one of the other computers with working video, I ssh'ed into the machine running the Xubuntu session and reconnected to the Byobu session. I found that xfsettingsd had crashed.

6) Executing "sleep 5; xfsettingsd" on the ssh session and switching to the Xubuntu machine restored the video.

7) I diff'ed the "before.txt" with the "output.txt". Here is the output that was generated when the monitor was unplugged and replugged.

xfce4-settings(displays): RRScreenChangeNotify event received.
xfce4-settings(displays): Refreshing RandR cache.
xfce4-settings(displays): Detected CRTC 63.
xfce4-settings(displays): Detected CRTC 64.
xfce4-settings(displays): Detected CRTC 65.
xfce4-settings(displays): Noutput: before = 1, after = 0.
xfce4-settings(displays): Output disconnected: VGA1
xfce4-settings(displays): Disabling CRTC 63.
xfce4-settings(displays): No active output anymore! Attempting to re-enable the internal output.
xfce4-settings(displays): RRScreenChangeNotify event received.
xfce4-settings(displays): Refreshing RandR cache.
xfce4-settings(displays): Detected CRTC 63.
xfce4-settings(displays): Detected CRTC 64.
xfce4-settings(displays): Detected CRTC 65.
xfce4-settings(displays): Noutput: before = 0, after = 0.
xfce4-settings(displays): RRScreenChangeNotify event received.
xfce4-settings(displays): Refreshing RandR cache.
xfce4-settings(displays): Detected CRTC 63.
xfce4-settings(displays): Detected CRTC 64.
xfce4-settings(displays): Detected CRTC 65.
xfce4-settings(displays): Detected output 66 VGA1.
xfce4-settings(displays): Noutput: before = 0, after = 1.
xfce4-settings(displays): New output connected: VGA1
xfce4-settings(displays): RRScreenChangeNotify event received.
xfce4-settings(displays): Refreshing RandR cache.
xfce4-settings(displays): Detected CRTC 63.
xfce4-settings(displays): Detected CRTC 64.
xfce4-settings(displays): Detected CRTC 65.
xfce4-settings(displays): Noutput: before = 1, after = 0.
xfce4-settings(displays): Output disconnected: VGA1
xfce4-settings(displays): Disabling CRTC 89.
The program 'xfsettingsd' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadRRCrtc (invalid Crtc parameter)'.
  (Details: serial 505 error_code 148 request_code 140 minor_code 21)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the --sync command line
   option to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)