RandR fractional scaling patch interferes with manual RandR output scaling

[Alex Goins]

$ lsb_release -rd
Description: Ubuntu 20.04.3 LTS
Release: 20.04

$ apt-cache policy mutter
mutter:
  Installed: 3.36.9-0ubuntu0.20.04.1
  Candidate: 3.36.9-0ubuntu0.20.04.1
  Version table:
 *** 3.36.9-0ubuntu0.20.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     3.36.1-3ubuntu3 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

---

As an example with one output, scaling it with e.g. 'xrandr --output <output> --mode 3840x2160 --scale-from 1920x1080', would usually result in a screen size of 1920x1080, as xrandr sets the screen size to the bounding box of the configured crtcs.

Ubuntu carries an out-of-tree patch for Mutter:
x11-Add-support-for-fractional-scaling-using-Randr.patch, attached. It adds a new function, meta_monitor_manager_xrandr_update_screen_size_derived(), that in response to RandR configuration changes will attempt to set its own screen size. When calculating the new screen size, it applies a scaling factor to each crtc that effectively reverses the requested one. As a result, in the above example the resulting screen size is 3840x2160, despite the fact that there is only a 1920x1080 viewport into it.

With the modesetting driver, this isn't a huge issue. The resulting screen size is larger than you can actually interact with, wasting memory, but it doesn't cause any visible functional issues.

The NVIDIA driver automatically configures panning when the requested outputs' viewports are smaller than the screen size. This increases crtc size, which impacts Mutter's screen size calculation. When Mutter sets the larger-than-outputs screen size, it triggers the NVIDIA driver to automatically configure panning. Mutter reacts to the RandR configuration change by again recalculating the screen size, this time growing it even larger due to the larger crtcs. In response, the NVIDIA driver sets up the new, larger panning configuration, causing the crtcs to be even larger, and causing Mutter to once again recalculate the screen size. This goes on until all memory is consumed and gnome-shell is killed by the OOM killer, or the screen reaches maximum size. In the meantime, the screen is black with only a cursor while gnome-shell spins.

It appears that meta_monitor_manager_xrandr_update_screen_size_derived() is intended for GNOME fractional scaling, adjusting the screen size to accommodate. In this case we are manually configuring output scaling with RandR, but this code still runs, causing unintended consequences.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "x11-Add-support-for-fractional-scaling-using-Randr.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.