Comment 32 for bug 1871644

(In reply to Emilio Cobos Álvarez (:emilio) from comment #20)
Non-composited X11 (the legacy variant of X11: i3, KDE with manually disabled compositor, etc.) does not support transparency. Everything that would usually be transparent/alpha is just black/opaque. Menus and window corners had [shadows on black background](https://bug1479135.bmoattachments.org/attachment.cgi?id=8995686) (=fat black borders).
The autoscroll icon was a [black square with a white circle](https://bug1649246.bmoattachments.org/attachment.cgi?id=9160184) in it.

[XShapeCombineMask](https://searchfox.org/mozilla-central/search?q=XShapeCombineMask&path=) is implemented for software rendering only. Firefox creates a stencil (like a cookie cutter) of where transparency needs to get cut off. X11 then applies this stencil on every frame. The outer shadow of a rounded main menu, addon widget or [window titlebar](https://searchfox.org/mozilla-central/rev/37373cdeed20695af1ff1fd090f0736d9bf15e65/widget/gtk/nsWindow.cpp#6729,6793) has transparency, so it gets cut off by X11, leaving a rounded widget without shadow.

XShapeCombineMask is not implemented for hardware rendering.
That's why SW WR is enforced for menus, addons and autoscroll icon on non-composited X11 to make use of XShapeCombineMask while the main window is still using OpenGL.

XShapeCombineMask
* caused problems on [Mutter](https://searchfox.org/mozilla-central/rev/37373cdeed20695af1ff1fd090f0736d9bf15e65/widget/gtk/nsWindow.cpp#9084) back then
* crashes the GPU process on Nvidia: bug 1730991
* causes performance problems according to the Chromium bugtracker: bug 1733094 comment 18,https://bugs.chromium.org/p/chromium/issues/detail?id=1198080

Could XShapeCombineMask be removed and instead menus and the autoscroll icon loose their border-radius (become rectangular) and loose their shadow (to avoid black background) if gdk_screen_is_composited() is false?
IIUC, that would reduce code complexity, improve stability and performance.

A rectangular autoscroll icon on non-composited X11 would be noticeable. But users choose non-composited X11 to get more performance than with composited X11. Considering this as well, it doesn't make sense to use XShapeCombineMask which contradicts their intent and causes above problems.