I've done some more debugging now. The case where metacity runs but immediately exits without setting SmRestartIfRunning (which also existed in Jaunty) is not an issue, because metacity doesn't set a restart command - so gnome-session won't restart it unless it was autostarted when the session began.
So the only case which triggers this then is when Metacity is autostarted when the session begins, then exits when you run Compiz. In Jaunty, metacity correctly sets SmRestartIfRunning before exiting (I verified this by looking at some debug output from gnome-session), but in Karmic this doesn't happen. In this scenario, gnome-session will respawn it continuously, which is what is happening.
I can see the change which breaks it, but I'm a bit uncomfortable with reverting it without understanding why it is there. Basically, it seems that meta_session_shutdown is called after meta_display_close was already called.
--- metacity-2.25.144/src/core/session.c 2009-02-01 20:33:15.000000000 +0000
+++ metacity-2.27.0/src/core/session.c 2009-02-20 15:26:32.000000000 +0000
@@ -376,6 +376,14 @@ meta_session_shutdown (void)
char hint = SmRestartIfRunning;
+ if (!meta_get_display ())
+ meta_verbose ("Cannot close session because there is no display");
+ warn_about_lame_clients_and_finish_interact (FALSE);
if (session_connection == NULL)