QPDF tries to use MD5 in FIPS mode

[Dariusz Gadomski]

[ Impact ]

 * After enabling fips (or fips-updates) libqpdf will fail to calculate MD5 for decoding the PDFs (which are used by cups-filters during the printing process).

* Printing may become impossible after enabling FIPS.

* For such situations gnutls provides LAX mode [1]:

GNUTLS_FIPS140_LAX
The library still uses the FIPS140-2 relevant algorithms but all forbidden by FIPS140-2 operations are allowed; this is useful when the application is aware of the followed security policy, and needs to utilize disallowed operations for other reasons (e.g., compatibility).

[1] https://www.gnutls.org/manual/html_node/FIPS140_002d2-mode.html

[ Test Plan ]

1. Try running the cups filter directly on a "clean" system (no fips-updates enabled):
/usr/lib/cups/filter/pdftopdf 555 $USER title 1 "" /usr/share/cups/data/confidential.pdf

2. If applicable for a given Ubuntu release - enable fips-updates
pro attach # to be able to enable FIPS mode
pro enable fips-updates
reboot # to boot the fips kernel

3. Reboot to make sure the FIPS kernel is used.

4. Run the cups filter again:
/usr/lib/cups/filter/pdftopdf 555 $USER title 1 "" /usr/share/cups/data/confidential.pdf

* With affected qpdf version running in FIPS mode it aborts with the following messages:
ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.
ERROR: pdftopdf filter function failed.

Releases without fips-updates support should not see any difference in behavior.

[ Where problems could occur ]

 * The patch relaxes gnutls restrictions on algorithms in non-security related,
limited contexts (GNUTLS_FIPS140_LAX mode). If there is an automated tooling or auditting software they may detect that not only FIPS-approved algorithms are
being used.

However, in context of decoding PDFs it is not possible to get rid of MD5 entirely. It's usage however is limited to only non-security related tasks.

With FIPS mode disabled libqpdf client programs should not see any changes.

[ Other Info ]

Original bug description:

After enabling fips/fips-updates it is impossible to print anymore.

Requesting a print results in the following message in the cups logs:
`ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.`

I have came up with a small reproducer:
```
pro attach # to be able to enable FIPS mode
pro enable fips-updates
reboot # to boot the fips kernel
# with FIPS mode enabled
/usr/lib/cups/filter/pdftopdf 555 $USER title 1 "" /usr/share/cups/data/confidential.pdf
```

The output ends with:
ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.
ERROR: pdftopdf filter function failed.

I have tracked the problematic code to:
QPDF::compute_data_key in libqpdf/QPDF_encryption.cc

It unconditionally uses MD5 (that in turn asks gnutls for MD5) and in FIPS mode it fails as MD5 is not fips-approved.

The bottomline is: it is not possible to print with fips-mode enabled.

[Dariusz Gadomski (dgadomski)]

gdb backtrace of where the issue occurs:
```
#0 gnutls_hash_init (dig=0x5555555ad818, algorithm=GNUTLS_DIG_MD5) at ../../lib/crypto-api.c:839
#1 0x00007ffff75b0e83 in QPDFCrypto_gnutls::MD5_init (this=0x5555555ad810) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFCrypto_gnutls.cc:46
#2 0x00007ffff756f309 in MD5::MD5 (this=0x7fffffffb8e0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/MD5.cc:11
#3 QPDF::compute_data_key (encryption_key=..., objid=<optimized out>, generation=0, use_aes=<optimized out>, encryption_V=<optimized out>,
    encryption_R=0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDF_encryption.cc:352
#4 0x00007ffff7553509 in QPDFWriter::setDataKey (this=0x7fffffffbdb0, objid=<optimized out>)
    at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:845
#5 0x00007ffff755e683 in QPDFWriter::writeObject (this=this@entry=0x7fffffffbdb0, object=..., object_stream_index=object_stream_index@entry=-1)
    at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:1790
#6 0x00007ffff75603f2 in QPDFWriter::writeStandard (this=0x7fffffffbdb0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:3013
#7 QPDFWriter::write (this=0x7fffffffbdb0) at /usr/src/qpdf-11.9.0-1.1build1/libqpdf/QPDFWriter.cc:2200
#8 0x00007ffff7e8e12d in _cfPDFToPDFQPDFProcessor::emit_file (this=0x55555559b320, f=<optimized out>, doc=0x7fffffffbea0, take=<optimized out>)
    at cupsfilters/pdftopdf/qpdf-pdftopdf-processor.cxx:876
#9 0x00007ffff7e822a8 in cfFilterPDFToPDF (inputfd=<optimized out>, outputfd=<optimized out>, inputseekable=<optimized out>, data=<optimized out>,
    parameters=<optimized out>) at cupsfilters/pdftopdf/pdftopdf.cxx:985
#10 0x00007ffff7f519bf in ppdFilterCUPSWrapper (argc=argc@entry=7, argv=argv@entry=0x7fffffffe398,
    filter=0x7ffff7f4fce0 <ppdFilterPDFToPDF(int, int, int, cf_filter_data_t*, void*)>, parameters=parameters@entry=0x0,
    JobCanceled=JobCanceled@entry=0x555555558014 <JobCanceled>) at ppd/ppd-filter.c:178
#11 0x00005555555550db in main (argc=7, argv=0x7fffffffe398) at filter/pdftopdf.c:66
```

[Dariusz Gadomski (dgadomski)]

Added debdiff for Resolute.

[Jay Berkenbilt (ejb)]

FYI this has already been addressed upstream. We are planning to release qpdf 12.3 with this and other fixes in the next two weeks.

[Jay Berkenbilt (ejb)]

Oh, I see the debdiff indicates the upstream repo as the origin. Appreciate the backport.

[Jay Berkenbilt (ejb)]

I have released qpdf 12.3.0, which includes the backported fix.

[Skia (skia)]

Resolute seems to have been fixed through Debian, and there doesn't seem to be any patch ready for sponsoring for stable Ubuntu series, hence I'm unsubscribing ~ubuntu-sponsors. Please re-subscribe when something changes and there is a need for sponsoring.

[Dariusz Gadomski (dgadomski)]

Thanks @skia. I uploaded the patch before it was synced with Debian. I'll sponsor the patches for the stable releases.

[Dariusz Gadomski (dgadomski)]

Marking plucky as "Won't fix" as we won't make it before its EOL.

[Dariusz Gadomski (dgadomski)]

I have sponsored the patches for j, n and q.

[Nick Rosbrook (enr0n)]

Is your current test plan actually able to be exercised on noble or questing right now? Won't the `pro enable fips-updates` step not work, at least on questing?

I realize looking at the patch that the changes are guarded by `fips_mode`, so I guess it should be a no-op on those releases. But if that is the case, I think it would be good to explicitly acknowledge that in the test plan, and perform a basic regression test, e.g. things don't start failing on non-FIPS systems due to some unexpected behavior from `gnutls_fips140_mode_enabled()`.

Also, my understanding is that resolute is fixed, so updating the status for clarity in future SRU reviews.

[Dariusz Gadomski (dgadomski)]

Thank you Nick. I have updated the test plan accordingly (covering also releases without fips-updates available via pro).

Resolute is indeed fixed.

[Nick Rosbrook (enr0n)]

Hello Dariusz, or anyone else affected,

Accepted qpdf into questing-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qpdf/12.2.0-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-questing to verification-done-questing. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-questing. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

Hello Dariusz, or anyone else affected,

Accepted qpdf into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qpdf/11.9.0-1.1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

Hello Dariusz, or anyone else affected,

Accepted qpdf into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qpdf/10.6.3-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dariusz Gadomski (dgadomski)]

Verified for noble:
```
# apt-cache policy libqpdf29t64 | grep Installed
  Installed: 11.9.0-1.1ubuntu0.1
```

No error observed with fips-updates enabled.

Outside of FIPS mode there behavior is as before the update.

[Dariusz Gadomski (dgadomski)]

Repeated same verification procedure for jammy:

apt-cache policy libqpdf28 | grep Installed
  Installed: 10.6.3-1ubuntu0.1

1. Installed libqpdf28 from -proposed.
2. Run the cups-filter/pdf2pdf test.
3. Attached pro and enabled fips-updates.
4. Reboot.
5. Repeated the cups-filter test.

In both cases (non-FIPS and FIPS) there is no sign an any undesired behavior nor the original error.

[Dariusz Gadomski (dgadomski)]

Due to fips-updates and pro not being available on Questing the test procedure is different:

1. Installed libqpdf30 (12.2.0-1ubuntu0.1) from -proposed.
2. Run the cups-filter/pdf2pdf test.

Behavior remained unchanged compared to unpatched version.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qpdf - 12.2.0-1ubuntu0.1

---------------
qpdf (12.2.0-1ubuntu0.1) questing; urgency=medium

  * debian/patches/lp2129676-relax-fips-for-pdfs.patch: Utilize the
    GNUTLS_FIPS140_LAX around MD5 initialization. (LP: #2129676)

 -- Dariusz Gadomski <email address hidden> Tue, 18 Nov 2025 14:43:56 +0100

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for qpdf has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qpdf - 11.9.0-1.1ubuntu0.1

---------------
qpdf (11.9.0-1.1ubuntu0.1) noble; urgency=medium

  * d/p/lp2129676-relax-fips-for-pdfs.patch: Utilize the
    GNUTLS_FIPS140_LAX around MD5 initialization. (LP: #2129676)
  * d/p/lp2131944-fix-failing-tests-on-s390x.patch: Fix failing tests on
    s390x due to different error output. (LP: #2131944)

 -- Dariusz Gadomski <email address hidden> Mon, 24 Nov 2025 13:52:26 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package qpdf - 10.6.3-1ubuntu0.1

---------------
qpdf (10.6.3-1ubuntu0.1) jammy; urgency=medium

  * d/p/lp2129676-relax-fips-for-pdfs.patch: Utilize the
    GNUTLS_FIPS140_LAX around MD5 initialization. (LP: #2129676)
  * d/p/lp2131944-fix-failing-tests-on-s390x.patch: Fix failing tests on
    s390x due to different error output. (LP: #2131944)

 -- Dariusz Gadomski <email address hidden> Mon, 24 Nov 2025 14:11:43 +0100