Under your proposal, how would Xfce know which monitor to put which panel on, if no primary monitor was selected?
Under your proposal, how would Xfce know which monitor to put which panel on, if no primary monitor was selected?