Some thoughts...

A plugin system for handling display type (e.g. console, XDMCP, Xnest, etc.) would be very useful because it would make it easy to enhance the display manager to support new types and would encourage more experimentation.  For example, it would be interesting to use Xvfb to map displays in local memory and then attach the Xvfb to actual monitors to support VT-like switching on non-console displays.  Such techniques are rather daunting to try to integrate into a display manager, but could become trivial to add if the display manager supported a flexible plugin system.

For many display types, the difference in how displays are managed are fairly trivial - how the display is started, how the display is torn down, and how switching is handled if switching is supported.  By moving this code into plugins, the display manager engine should become more clean and abstract and easier to maintain.

One problem with many other display managers out there is that the code turns into spaghetti after it supports a few display types with a lot of odd code to handle state and different cases.  Separating this logic into plugins and keeping the engine simple & consistent for all display types should make code management a lot more simple.

I am not sure that a plugin system for "display servers" is really needed.  You could treat a different kind of server simply as a different display type.  The engine should only be concerned about how to start, how to tear down, and how to handle switching if supported.  In other words, the command to run (Xserver or Wayland) could just be handled by different plugins.

An authentication/audit system plugin would be useful since PAM could be moved into a plugin.  Users might want to use different PAM modules on different displays (e.g. fingerprint reader on one and normal username/password entry on the other).  So it is useful to be able to specify different PAM service names (the 1st argument passed to pam_start) in a per-display fashion.  Also extracting authentication into plugins would make it easy to use the LightDM engine to handle different kinds of authentication - not just PAM.  For example, it would be neat if LightDM also supported screen locking.  To do this properly you really need a plugin system for authentication since the PAM and audit calls for login are different than for screen lock.  If you wanted to use LightDM as an engine for handling password entry for other programs (such as gnome-keyring or PolicyKit), then you might want other non-PAM plugins to be used for these situations that pass the password input along to where it is needed in a secure fashion.

The advantage of the display manager providing a service to programs to handle such authentication is that it would make password entry more secure.  One issue with password entry in the user's session is that anybody who is able to access your xauth key can snoop keypresses, for example.  So password entry is only as secure as Xserver xauth security.  Not that secure, really.  However, if LightDM provided a service that a program could ask LightDM to VT switch to a new display, ask for the password and return back to the user session when done, this would use a separate Xauth connection and would run the GUI as a separate user (like the "gdm" user) preventing such exploits.  The authenciation plugin could handle the glue to pass needed data to the program waiting in the user session, perhaps via D-Bus.  gnome-keyring could then provide its own LightDM authentication plugin and make D-Bus calls to make LightDM provide this service.

For LightDM to provide such a generic password entry service, it would be necessary for LightDM to provide external hooks (perhaps D-Bus calls) so that the switching events for a given display type plugin and authentication plugin could be triggered externally.

One issue about figuring out a good plugin system is how to handle configuration.  Each plugin may have its own configuration needs.  May display managers get bogged down trying to store configuration for different display types in a single configuration file, so storing plugin configuration in separate files would be a good idea, I think.  For example, users may want to modify the Xserver command to add arguments to turn on extensions they need or turn off extensions they don't want, or for other purposes.   It probably makes sense to put configuration files for each plugin in a common directory (perhaps the name of the file could match the plugin name), and to use a common format.  Having a common engine for plugins to use to make loading/saving any configuration that is needed could make this easy to manage.  Plugins could probably use the same code that the LightDM engine uses for configuration, just with the ability for the plugin to specify a different filename to use.