Wrong SHA256-value computed on kinetic

[Christian Froemmel]

[ Impact ]

OpenLDAP deployments using the contrib pw-sha2 module are not able to authenticate their users because the SHA2 calculation is done incorrectly.

Even though this is a contrib module, from an upstream PoV, it is shipped in the Ubuntu (and Debian) OpenLDAP packages, and available for use.

Some fix possibilities were discussed in comment #5, and we selected the one with the least impact to OpenLDAP users at large, which is to recompile that module only without the strict-aliasing optimization.

This update makes that change, and also includes a DEP8 change to verify it.

We didn't use a patch for the pw-sha2 Makefile because d/rules overrides the OPT variable in the make command line (https://git.launchpad.net/~ahasenack/ubuntu/+source/openldap/tree/debian/rules?h=lunar-slapd-sha2-2000817#n44)

[ Test Plan ]

# Install slapd and openssl
$ sudo apt install slapd openssl

# Run the following script

#!/bin/bash

reference_hash="{SHA256}$(echo -n secret | openssl dgst -sha256 -binary | openssl enc -base64)"
test_hash=$(slappasswd -s secret -h '{SHA256}' -o module-load=pw-sha2)

echo "Reference hash of \"secret\" (openssl): ${reference_hash}"
echo "slapd's pw-sha2 hash: ${test_hash}"

if [ "${reference_hash}" != "${test_hash}" ]; then
    echo "ERROR: hashes differ"
    exit 1
else
    echo "PASS: hashes are identical"
fi

With the affected openldap package installed, the script should print an error. With the packages from proposed, the hashes should be identical.

[ Where problems could occur ]

The fix is a change of a compiler option strictly when building only the pw-sha2 module, so it's very localized. It could affect the performance of this module (for the worse), but it's already not working correctly.

[ Other Info ]
Not at this time.

[Original Description]

The OpenLDAP-contrib module sha2 (located in contrib/slapd-modules/passwd/sha2/) computes a wrong SHA256/SSHA256-hash on Ubuntu kinetic. This breaks our current password-authentication in ldap.

The problematic computation:

    $ slappasswd -s secret -h '{SHA256}' -o module-load=pw-sha2
    {SHA256}WIrrpN3OjEVOUf6yrH1j+o+ODuUuNBo979Od4UXnu54=

The (correct) reference-value on the same system (or older ubuntu Versions):

    $ echo -n "secret" | openssl dgst -sha256 -binary | openssl enc -base64
    K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=

We nailed the problem down to a bug in the gcc-optimizer for strict-aliasing. so most probably the gcc-version on kinetic (v12.2.0) is the reason. The workaround is to compile the sha2-Module with the flag "-fno-strict-aliasing". Then the correct value is computed. An example taken from a git-compiled version of OpenLDAP 2.5.13:

    $ ./servers/slapd/slappasswd -T passwd -s secret -h '{SHA256}' -o module-load=pw-sha2 -o module-path=contrib/slapd-modules/passwd/sha2/.libs
    {SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=

Ubuntu:

    Description: Ubuntu 22.10
    Release: 22.10

    OpenLDAP-Package: 2.5.13+dfsg-1ubuntu1

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "openldap-contrib-sha2.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Bryce Harrington (bryce)]

lunar, kinetic, and jammy all return the first result, while focal provides the second:

triage-lunar+23.04: ~$ slappasswd -s secret -h '{SHA256}' -o module-load=pw-sha2
{SHA256}WIrrpN3OjEVOUf6yrH1j+o+ODuUuNBo979Od4UXnu54=
triage-lunar+23.04: ~$ slapd -VV
@(#) $OpenLDAP: slapd 2.6.3+dfsg-1~exp1ubuntu1 (Nov 18 2022 21:07:45) $

triage-kinetic+22.10: ~$ slappasswd -s secret -h '{SHA256}' -o module-load=pw-sha2
{SHA256}WIrrpN3OjEVOUf6yrH1j+o+ODuUuNBo979Od4UXnu54=
triage-kinetic+22.10: ~$ slapd -VV
@(#) $OpenLDAP: slapd 2.5.13+dfsg-1ubuntu1 (Sep 20 2022 19:30:47) $

triage-jammy+22.04: ~$ slappasswd -s secret -h '{SHA256}' -o module-load=pw-sha2
{SHA256}WIrrpN3OjEVOUf6yrH1j+o+ODuUuNBo979Od4UXnu54=
triage-jammy+22.04: ~$ slapd -VV
@(#) $OpenLDAP: slapd 2.5.13+dfsg-0ubuntu0.22.04.1 (Aug 5 2022 14:51:52) $

triage-focal+20.04: ~$ slappasswd -s secret -h '{SHA256}' -o module-load=pw-sha2
{SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
triage-focal+20.04: ~$ slapd -VV
@(#) $OpenLDAP: slapd (Ubuntu) (May 12 2022 13:11:05) $
triage-focal+20.04: ~$ apt-cache policy slapd
slapd:
  Installed: 2.4.49+dfsg-2ubuntu1.9

On all releases, the openssl dgst call produces the same result, K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=

Here's two other references mentioning the same problem, and same suggested workaround:

* https://www.mail-archive.com/search?<email address hidden>&q=subject:%22%22&o=newest&f=1
* https://stackoverflow.com/questions/74928752/slappasswd-generating-a-strange-password-hash-sha256-only

I don't know whether there might be side effects from adding "-fno-strict-aliasing". However, the patch's compilation modifications looks like it'll affect the performance of only just the sha2 module, so for SRU policy this seems a narrow enough fix. Since this is described in the first link as a contrib module, that may explain why this issue hasn't come to light earlier.

[Andreas Hasenack (ahasenack)]

Writing down some options from a quick handover discussion I had with Sergio, in no particular order:
- rebuild openldap with the attached patch, disabling strict aliasing just for that module
- check if LTO is having an effect on this: maybe disabling LTO also fixes it, and there is precedence for such a change in an SRU
- maybe the module code can be changed to not trigger this bug
- future-land: make the module use gnutls for such functions

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.6.3+dfsg-1~exp1ubuntu2

---------------
openldap (2.6.3+dfsg-1~exp1ubuntu2) lunar; urgency=medium

  * Build the passwd/sha2 contrib module with -fno-strict-aliasing to
    avoid computing an incorrect SHA256 hash with some versions of the
    compiler (LP: #2000817):
    - d/t/{control,sha2-contrib}: test to verify the SHA256 hash
      produced by passwd/sha2
    - d/rules: set -fno-strict-aliasing only when building the
      passwd/sha2 contrib module
  * d/t/smbk5pwd: Allow the openldap user to read the Heimdal master key in the
    smbk5pwd DEP8 test (LP: #2004560)

 -- Andreas Hasenack <email address hidden> Fri, 03 Feb 2023 09:33:14 -0300

[Steve Langasek (vorlon)]

Hello Christian, or anyone else affected,

Accepted openldap into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.5.14+dfsg-0ubuntu0.22.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openldap (2.5.14+dfsg-0ubuntu0.22.10.2) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

cyrus-sasl2/2.1.28+dfsg-6ubuntu2 (armhf)
exim4/4.96-3ubuntu1.1 (ppc64el)
kopanocore/8.7.0-7.1ubuntu11 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#openldap

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Andreas Hasenack (ahasenack)]

Kinetic verification

Confirming the problem with the unfixed slapd package:

$ apt-cache policy slapd
slapd:
  Installed: 2.5.14+dfsg-0ubuntu0.22.10.1
  Candidate: 2.5.14+dfsg-0ubuntu0.22.10.1
  Version table:
 *** 2.5.14+dfsg-0ubuntu0.22.10.1 500
        500 http://br.archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.5.13+dfsg-1ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu kinetic/main amd64 Packages

$ ./test.sh
Reference hash of "secret" (openssl): {SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
slapd's pw-sha2 hash: {SHA256}WIrrpN3OjEVOUf6yrH1j+o+ODuUuNBo979Od4UXnu54=
ERROR: hashes differ

Updating to proposed:
$ apt-cache policy slapd
slapd:
  Installed: 2.5.14+dfsg-0ubuntu0.22.10.2
  Candidate: 2.5.14+dfsg-0ubuntu0.22.10.2
  Version table:
 *** 2.5.14+dfsg-0ubuntu0.22.10.2 500
        500 http://br.archive.ubuntu.com/ubuntu kinetic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.5.14+dfsg-0ubuntu0.22.10.1 500
        500 http://br.archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
     2.5.13+dfsg-1ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu kinetic/main amd64 Packages

Problem fixed:
$ ./test.sh
Reference hash of "secret" (openssl): {SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
slapd's pw-sha2 hash: {SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
PASS: hashes are identical

Kinetic verification succeeded.

[Andreas Hasenack (ahasenack)]

I'm checking the exim4 dep8 failures in kinetic.

[Andreas Hasenack (ahasenack)]

I'm able to reproduce the exim4 dep8 error on kinetic ppc64el with the updated libldap packages, investigating...

[Andreas Hasenack (ahasenack)]

And also without the updated packages. It's a bit random. When exim4 is failing, it stays failing. Restart it, and it might or might not be in failing mode. Sometimes it works, and keeps working. It's not related to the openldap update. I think the migration-reference/0 run that succeeded was a fluke.

[Andreas Hasenack (ahasenack)]

https://bugs.launchpad.net/ubuntu/+source/exim4/+bug/1983605 is the exim4 bug. I would ask the SRU team to ignore the exim4 DEP8 errors on basis of that.

[Timo Aaltonen (tjaalton)]

Hello Christian, or anyone else affected,

Accepted openldap into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.5.14+dfsg-0ubuntu0.22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Jammy verification

Reproducing the problem with the release packages:

$ cat > test.sh
#!/bin/bash

reference_hash="{SHA256}$(echo -n secret | openssl dgst -sha256 -binary | openssl enc -base64)"
test_hash=$(slappasswd -s secret -h '{SHA256}' -o module-load=pw-sha2)

echo "Reference hash of \"secret\" (openssl): ${reference_hash}"
echo "slapd's pw-sha2 hash: ${test_hash}"

if [ "${reference_hash}" != "${test_hash}" ]; then
    echo "ERROR: hashes differ"
    exit 1
else
    echo "PASS: hashes are identical"
fi
^D

# apt-cache policy slapd
slapd:
  Installed: 2.5.14+dfsg-0ubuntu0.22.04.1
  Candidate: 2.5.14+dfsg-0ubuntu0.22.04.1
  Version table:
 *** 2.5.14+dfsg-0ubuntu0.22.04.1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
(...)

root@j-slapd-sha2:~# ./test.sh
Reference hash of "secret" (openssl): {SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
slapd's pw-sha2 hash: {SHA256}WIrrpN3OjEVOUf6yrH1j+o+ODuUuNBo979Od4UXnu54=
ERROR: hashes differ

Installing slapd from proposed:
root@j-slapd-sha2:~# apt-cache policy slapd
slapd:
  Installed: 2.5.14+dfsg-0ubuntu0.22.04.2
  Candidate: 2.5.14+dfsg-0ubuntu0.22.04.2
  Version table:
 *** 2.5.14+dfsg-0ubuntu0.22.04.2 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Test now passes:
root@j-slapd-sha2:~# ./test.sh
Reference hash of "secret" (openssl): {SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
slapd's pw-sha2 hash: {SHA256}K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
PASS: hashes are identical

Jammy verification succeeded.

[Andreas Hasenack (ahasenack)]

Another migration-reference/0 attempt for kinetic[1] confirmed that the previous "success" was a fluke, and exim4 on plain kinetic is already failing[2]. That's bug #1983605

1. https://autopkgtest.ubuntu.com/packages/e/exim4/kinetic/ppc64el
2. https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/ppc64el/e/exim4/20230403_151428_4e639@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.5.14+dfsg-0ubuntu0.22.10.2

---------------
openldap (2.5.14+dfsg-0ubuntu0.22.10.2) kinetic; urgency=medium

  * Build the passwd/sha2 contrib module with -fno-strict-aliasing to
    avoid computing an incorrect SHA256 hash with some versions of the
    compiler (LP: #2000817):
    - d/t/{control,sha2-contrib}: test to verify the SHA256 hash
      produced by passwd/sha2
    - d/rules: set -fno-strict-aliasing only when building the
      passwd/sha2 contrib module

 -- Andreas Hasenack <email address hidden> Sun, 12 Mar 2023 14:09:05 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for openldap has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openldap - 2.5.14+dfsg-0ubuntu0.22.04.2

---------------
openldap (2.5.14+dfsg-0ubuntu0.22.04.2) jammy; urgency=medium

  * Build the passwd/sha2 contrib module with -fno-strict-aliasing to
    avoid computing an incorrect SHA256 hash with some versions of the
    compiler (LP: #2000817):
    - d/t/{control,sha2-contrib}: test to verify the SHA256 hash
      produced by passwd/sha2
    - d/rules: set -fno-strict-aliasing only when building the
      passwd/sha2 contrib module

 -- Andreas Hasenack <email address hidden> Sun, 12 Mar 2023 14:11:53 -0300