QPDF tries to use MD5 in FIPS mode

[Dariusz Gadomski]

[ Impact ]

 * After enabling fips (or fips-updates) libqpdf will fail to calculate MD5 for decoding the PDFs (which are used by cups-filters during the printing process).

* Printing may become impossible after enabling FIPS.

* For such situations gnutls provides LAX mode [1]:

GNUTLS_FIPS140_LAX
The library still uses the FIPS140-2 relevant algorithms but all forbidden by FIPS140-2 operations are allowed; this is useful when the application is aware of the followed security policy, and needs to utilize disallowed operations for other reasons (e.g., compatibility).

[1] https://www.gnutls.org/manual/html_node/FIPS140_002d2-mode.html

[ Test Plan ]

 * Enable FIPS:
pro attach # to be able to enable FIPS mode
pro enable fips-updates
reboot # to boot the fips kernel

* Try to run the filter directly:
/usr/lib/cups/filter/pdftopdf 555 $USER title 1 "" /usr/share/cups/data/confidential.pdf

* With affected qpdf version it aborts with the following messages:
ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.
ERROR: pdftopdf filter function failed.

[ Where problems could occur ]

 * The patch relaxes gnutls restrictions on algorithms in non-security related,
limited contexts (GNUTLS_FIPS140_LAX mode). If there is an automated tooling or auditting software they may detect that not only FIPS-approved algorithms are
being used.

However, in context of decoding PDFs it is not possible to get rid of MD5 entirely. It's usage however is limited to only non-security related taks.

[ Other Info ]

Original bug description:

After enabling fips/fips-updates it is impossible to print anymore.

Requesting a print results in the following message in the cups logs:
`ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.`

I have came up with a small reproducer:
```
pro attach # to be able to enable FIPS mode
pro enable fips-updates
reboot # to boot the fips kernel
# with FIPS mode enabled
/usr/lib/cups/filter/pdftopdf 555 $USER title 1 "" /usr/share/cups/data/confidential.pdf
```

The output ends with:
ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.
ERROR: pdftopdf filter function failed.

I have tracked the problematic code to:
QPDF::compute_data_key in libqpdf/QPDF_encryption.cc

It unconditionally uses MD5 (that in turn asks gnutls for MD5) and in FIPS mode it fails as MD5 is not fips-approved.

The bottomline is: it is not possible to print with fips-mode enabled.

[Dariusz Gadomski (dgadomski)]

gdb backtrace of where the issue occurs:
```
#0 gnutls_hash_init (dig=0x5555555ad818, algorithm=GNUTLS_DIG_MD5) at ../../lib/crypto-api.c:839
#1 0x00007ffff75b0e83 in QPDFCrypto_gnutls::MD5_init (this=0x5555555ad810) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFCrypto_gnutls.cc:46
#2 0x00007ffff756f309 in MD5::MD5 (this=0x7fffffffb8e0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/MD5.cc:11
#3 QPDF::compute_data_key (encryption_key=..., objid=<optimized out>, generation=0, use_aes=<optimized out>, encryption_V=<optimized out>,
    encryption_R=0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDF_encryption.cc:352
#4 0x00007ffff7553509 in QPDFWriter::setDataKey (this=0x7fffffffbdb0, objid=<optimized out>)
    at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:845
#5 0x00007ffff755e683 in QPDFWriter::writeObject (this=this@entry=0x7fffffffbdb0, object=..., object_stream_index=object_stream_index@entry=-1)
    at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:1790
#6 0x00007ffff75603f2 in QPDFWriter::writeStandard (this=0x7fffffffbdb0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:3013
#7 QPDFWriter::write (this=0x7fffffffbdb0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:2200
#8 0x00007ffff7e8e12d in _cfPDFToPDFQPDFProcessor::emit_file (this=0x55555559b320, f=<optimized out>, doc=0x7fffffffbea0, take=<optimized out>)
    at cupsfilters/pdftopdf/qpdf-pdftopdf-processor.cxx:876
#9 0x00007ffff7e822a8 in cfFilterPDFToPDF (inputfd=<optimized out>, outputfd=<optimized out>, inputseekable=<optimized out>, data=<optimized out>,
    parameters=<optimized out>) at cupsfilters/pdftopdf/pdftopdf.cxx:985
#10 0x00007ffff7f519bf in ppdFilterCUPSWrapper (argc=argc@entry=7, argv=argv@entry=0x7fffffffe398,
    filter=0x7ffff7f4fce0 <ppdFilterPDFToPDF(int, int, int, cf_filter_data_t*, void*)>, parameters=parameters@entry=0x0,
    JobCanceled=JobCanceled@entry=0x555555558014 <JobCanceled>) at ppd/ppd-filter.c:178
#11 0x00005555555550db in main (argc=7, argv=0x7fffffffe398) at filter/pdftopdf.c:66
```

[Dariusz Gadomski (dgadomski)]

Added debdiff for Resolute.

[Jay Berkenbilt (ejb)]

FYI this has already been addressed upstream. We are planning to release qpdf 12.3 with this and other fixes in the next two weeks.

[Jay Berkenbilt (ejb)]

Oh, I see the debdiff indicates the upstream repo as the origin. Appreciate the backport.

[Jay Berkenbilt (ejb)]

I have released qpdf 12.3.0, which includes the backported fix.

[Skia (skia)]

Resolute seems to have been fixed through Debian, and there doesn't seem to be any patch ready for sponsoring for stable Ubuntu series, hence I'm unsubscribing ~ubuntu-sponsors. Please re-subscribe when something changes and there is a need for sponsoring.

[Dariusz Gadomski (dgadomski)]

Thanks @skia. I uploaded the patch before it was synced with Debian. I'll sponsor the patches for the stable releases.

[Dariusz Gadomski (dgadomski)]

Marking plucky as "Won't fix" as we won't make it before its EOL.

[Dariusz Gadomski (dgadomski)]

I have sponsored the patches for j, n and q.

[Nick Rosbrook (enr0n)]

Is your current test plan actually able to be exercised on noble or questing right now? Won't the `pro enable fips-updates` step not work, at least on questing?

I realize looking at the patch that the changes are guarded by `fips_mode`, so I guess it should be a no-op on those releases. But if that is the case, I think it would be good to explicitly acknowledge that in the test plan, and perform a basic regression test, e.g. things don't start failing on non-FIPS systems due to some unexpected behavior from `gnutls_fips140_mode_enabled()`.

Also, my understanding is that resolute is fixed, so updating the status for clarity in future SRU reviews.