Comment 1 for bug 1898601

gdm3 is being pulled in via a "Recommends" chain. Disabling recommends will avoid it[*].
On my groovy system it reduced new packages installed from 452/291MB of archives to 56/165MB of archives. Seems like the right thing to do for at least --gpgpu installs - but maybe for all?

[*]
--- /usr/bin/ubuntu-drivers.orig 2020-07-29 15:24:08.000000000 +0000
+++ /usr/bin/ubuntu-drivers 2020-10-06 19:18:53.208559657 +0000
@@ -155,7 +155,8 @@
         return

     ret = subprocess.call(['apt-get', 'install', '-o',
- 'DPkg::options::=--force-confnew', '-y'] + to_install)
+ 'DPkg::options::=--force-confnew', '-y',
+ '--no-install-recommends'] + to_install)

     oem_meta_to_install = fnmatch.filter(to_install, 'oem-*-meta')