Comment 3 for bug 1992228

Thanks to Sebastien and Julian for the hints. There seem to have been some performance variation in upstream around 6.9.10...6.9.11, but never quite as bad as the Ubuntu package.

Steps taken:

$ sudo apt-get build-dep imagemagick-6.q16
$ git clone https://github.com/ImageMagick/ImageMagick6.git
$ cd ImageMagick6
$ git checkout 6.9.11-60
$ ./configure --prefix=/usr/local/stow/ImageMagick6 --with-modules --with-gs-font-dir=/usr/share/fonts/type1/gsfonts -with-magick-plus-plus --with-djvu --with-heic --with-openjp2 --with-webp --with-wmf --without-gvc --without-dps --without-fpx --with-perl --without-rsvg --with-fftw
$ make
$ sudo make install
$ cd /usr/local/stow
$ sudo stow ImageMagick6
$ sudo ldconfig
$ cd ~
$ time /usr/local/bin/convert -version scan.pnm -compress Fax scan-upstream.pdf
Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-24 https://imagemagick.org
Copyright: (C) 1999-2021 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP(4.5)
Delegates (built-in): bzlib djvu fftw fontconfig freetype heic jbig jng jp2 jpeg lcms lqr ltdl lzma openexr pangocairo png tiff webp wmf x xml zlib

real 0m10.906s
user 0m11.559s
sys 0m0.829s

This is slower than I had on Focal, I believe it was around 5 seconds but I don't have measurements. Unstow it and I'm back at 4m44s:

$ cd /usr/local/stow
$ sudo stow -D ImageMagick6
$ sudo ldconfig
$ cd ~
$ time /usr/bin/convert -version scan.pnm -compress Fax scan-ubuntu.pdf
Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org
Copyright: (C) 1999-2021 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP(4.5)
Delegates (built-in): bzlib djvu fftw fontconfig freetype heic jbig jng jp2 jpeg lcms lqr ltdl lzma openexr pangocairo png tiff webp wmf x xml zlib

real 4m43.623s
user 2m6.014s
sys 2m38.471s
$ ls -la scan-*.pdf
-rw-rw-r-- 1 i i 127070 Okt 15 16:35 scan-ubuntu.pdf
-rw-rw-r-- 1 i i 127046 Okt 15 16:29 scan-upstream.pdf

Configuring the package resulted in the following:

==============================================================================
ImageMagick 6.9.11-60 is configured as follows. Please verify that this
configuration matches your expectations.

Host system type: x86_64-pc-linux-gnu
Build system type: x86_64-pc-linux-gnu

                  Option Value
------------------------------------------------------------------------------
Shared libraries --enable-shared=yes yes
Static libraries --enable-static=yes yes
Module support --with-modules=yes yes
GNU ld --with-gnu-ld=yes yes
Quantum depth --with-quantum-depth=16 16
High Dynamic Range Imagery
                  --enable-hdri=no no

Install documentation: yes

Memory allocation library:
  JEMalloc --with-jemalloc=no no
  TCMalloc --with-tcmalloc=no no
  UMem --with-umem=no no

Delegate library configuration:
  BZLIB --with-bzlib=yes yes
  Autotrace --with-autotrace=no no
  DJVU --with-djvu=yes yes
  DPS --with-dps=no no
  FFTW --with-fftw=yes yes
  FLIF --with-flif=yes no
  FlashPIX --with-fpx=no no
  FontConfig --with-fontconfig=yes yes
  FreeType --with-freetype=yes yes
  Ghostscript lib --with-gslib=no no
  Graphviz --with-gvc=no
  HEIC --with-heic=yes yes
  JBIG --with-jbig=yes yes
  JPEG v1 --with-jpeg=yes yes
  JPEG XL --with-jxl=yes no
  LCMS --with-lcms=yes yes
  LQR --with-lqr=yes yes
  LTDL --with-ltdl=yes yes
  LZMA --with-lzma=yes yes
  Magick++ --with-magick-plus-plus=yes yes
  OpenEXR --with-openexr=yes yes
  OpenJP2 --with-openjp2=yes yes
  PANGO --with-pango=yes yes
  PERL --with-perl=yes /usr/bin/perl
  PNG --with-png=yes yes
  RAQM --with-raqm=yes no
  RAW --with-raw=yes no
  RSVG --with-rsvg=no no
  TIFF --with-tiff=yes yes
  WEBP --with-webp=yes yes
  WMF --with-wmf=yes yes
  X11 --with-x= yes
  XML --with-xml=yes yes
  ZLIB --with-zlib=yes yes
  ZSTD --with-zstd=yes no

Delegate program configuration:
  GhostPCL None pcl6 (unknown)
  GhostXPS None gxps (unknown)
  Ghostscript None gs (9.55.0)

Font configuration:
  Apple fonts --with-apple-font-dir=default
  Dejavu fonts --with-dejavu-font-dir=default none
  Ghostscript fonts --with-gs-font-dir=/usr/share/fonts/type1/gsfonts /usr/share/fonts/type1/gsfonts/
  URW-base35 fonts --with-urw-base35-font-dir=default /usr/share/fonts/type1/urw-base35/
  Windows fonts --with-windows-font-dir=default /usr/share/fonts/truetype/msttcorefonts/

X11 configuration:
  X_CFLAGS =
  X_PRE_LIBS = -lSM -lICE
  X_LIBS =
  X_EXTRA_LIBS =

Options used to compile and link:
  PREFIX = /usr/local/stow/ImageMagick6
  EXEC-PREFIX = /usr/local/stow/ImageMagick6
  VERSION = 6.9.11-60
  CC = gcc
  CFLAGS = -I/usr/include/libxml2 -I/usr/include/libpng16 -pthread -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/fribidi -I/usr/include/harfbuzz -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/uuid -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/OpenEXR -I/usr/include/lqr-1 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/openjpeg-2.4 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/uuid -I/usr/include/freetype2 -I/usr/include/libpng16 -pthread -fopenmp -Wall -g -O2 -mtune=core2 -fexceptions -pthread -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16
  CPPFLAGS = -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16
  PCFLAGS =
  DEFS = -DHAVE_CONFIG_H
  LDFLAGS =
  LIBS = -llcms2 -lfreetype -llqr-1 -lglib-2.0 -lfftw3 -lxml2 -lfontconfig -lfreetype -lXext -lSM -lICE -lX11 -lXt -llzma -lbz2 -lz -lltdl -lm -lpthread
  CXX = g++
  CXXFLAGS = -pthread
  FEATURES = DPC Cipher OpenMP Modules
  DELEGATES = bzlib djvu fftw fontconfig freetype heic jbig jng jpeg lcms lqr lzma openexr openjp2 pango png ps tiff video webp wmf x xml zlib
==============================================================================