[Ubuntu 20.04] OpenSSL bugs in the s390x AES code

[bugproxy]

Problem description:

When passing a NULL key to reset AES EVC state, the state wouldn't be completely reset on s390x.
https://github.com/openssl/openssl/pull/14900

Solution available here:
https://github.com/openssl/openssl/commit/dc67210d909b5dd7a50f60a96f36f3f5a891b1c8

Should be applied to all distros where openssl 1.1.1 is included for consistency reason.
-> 21.10, 20.04, 18.04.
I think not needed for 16.04 anymore....

[Test plan]

$ sudo apt install libssl-dev
$ gcc test.c -o evc-test -lcrypto -lssl # See https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1931994/comments/2 for the test.c program
$ ./evc-test && echo OK

[Where problems could occur]

This patch only touches s390x code paths, so there shouldn't be any regression on other architectures. However, on s390x this could reveal
latent bugs by spreading a NULL key to new code paths.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Simon Chopin (schopin)]

Updated description in preparation for SRU requests

[bugproxy (bugproxy)]

Default Comment by Bridge

[Gunnar Hjalmarsson (gunnarhj)]

Sponsored impish for now.

As regards the SRUs, I suppose that the block-proposed-hirsute tag set at bug #1927161 needs to be removed. Also, should the test plan be expanded to include the test script which bugproxy added?

[Brian Murray (brian-murray)]

On Mon, Jul 26, 2021 at 09:31:51PM -0000, Gunnar Hjalmarsson wrote:
> Sponsored impish for now.
>
> As regards the SRUs, I suppose that the block-proposed-hirsute tag set
> at bug #1927161 needs to be removed. Also, should the test plan be
> expanded to include the test script which bugproxy added?

The tag does need to be removed but an SRU team member could do that
when reviewing the next upload. When somebody uploads the new version
of openssl, it should be built on top of the current version in
-proposed and -v should be used when you run debuild so that the
previous bug is incorporated in the source.changes file.

--
Brian Murray

[Simon Chopin (schopin)]

Regarding the bugproxy test case, it should be disregarded: I was the one who originally added it, but then found a much smaller and self-contained test case, and removed the attachment. For some reason, bugproxy didn't like that.

[Simon Chopin (schopin)]

Oh, and thank you very much for the upload, much appreciated :-)

[Frank Heimes (fheimes)]

@schopin that is a (known) issue with the bugzilla-launchpad bridge that is in place here (bugproxy is the functional user id for this).
I'll ask to get the "Standalone C program from the upstream test case" attachment removed on the bugzilla side, that should help ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-07-27 04:23 EDT-------
Attachment "Standalone C program from the upstream test case" removed on the bugzilla side as requested by Canonical

[Gunnar Hjalmarsson (gunnarhj)]

@Simon: autopkgtest for mosquitto and puma fails on s390x.

https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#openssl

Please investigate.

[Simon Chopin (schopin)]

The mosquitto test seems to be failing regardless of the version of
openssl, I'll take a look at puma.

On Tue, Jul 27, 2021 at 1:35 PM Gunnar Hjalmarsson <
<email address hidden>> wrote:

> @Simon: autopkgtest for mosquitto and puma fails on s390x.
>
> https://people.canonical.com/~ubuntu-archive/proposed-
> migration/update_excuses.html#openssl
>
> Please investigate.
>
> --
> You received this bug notification because you are a member of Canonical
> Foundations Team, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1931994
>
> Title:
> [Ubuntu 20.04] OpenSSL bugs im s390x AES code
>
> Status in Ubuntu on IBM z Systems:
> In Progress
> Status in openssl package in Ubuntu:
> Fix Committed
> Status in openssl source package in Bionic:
> New
> Status in openssl source package in Focal:
> New
> Status in openssl source package in Hirsute:
> New
> Status in openssl source package in Impish:
> Fix Committed
>
> Bug description:
> Problem description:
>
> When passing a NULL key to reset AES EVC state, the state wouldn't be
> completely reset on s390x.
> https://github.com/openssl/openssl/pull/14900
>
> Solution available here:
>
> https://github.com/openssl/openssl/commit/dc67210d909b5dd7a50f60a96f36f3f5a891b1c8
>
> Should be applied to all distros where openssl 1.1.1 is included for
> consistency reason.
> -> 21.10, 20.04, 18.04.
> I think not needed for 16.04 anymore....
>
> [Test plan]
>
> $ sudo apt install libssl-dev
> $ gcc test.c -o evc-test -lcrypto -lssl # See
> https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1931994/comments/2
> for the test.c program
> $ ./evc-test && echo OK
>
> [Where problems could occur]
>
> This patch only touches s390x code paths, so there shouldn't be any
> regression on other architectures. However, on s390x this could reveal
> latent bugs by spreading a NULL key to new code paths.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-z-systems/+bug/1931994/+subscriptions
>
>

[Gunnar Hjalmarsson (gunnarhj)]

I re-run the failed tests, and both of them passed on the second attempt, so it should migrate on impish soon.

The SRUs are now uploaded, following Brian's advice with respect to hirsute.

@Simon: Good if you can unsubscribe ubuntu-sponsors.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.1.1j-1ubuntu5

---------------
openssl (1.1.1j-1ubuntu5) impish; urgency=medium

  * Cherry-pick an upstream patch to fix s390x AES code (LP: #1931994)

 -- Simon Chopin <email address hidden> Fri, 23 Jul 2021 14:32:42 +0200

[Simon Chopin (schopin)]

Thank you, Gunnar!

[Steve Langasek (vorlon)]

Hello bugproxy, or anyone else affected,

Accepted openssl into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.1.1j-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello bugproxy, or anyone else affected,

Accepted openssl into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.1.1f-1ubuntu2.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello bugproxy, or anyone else affected,

Accepted openssl into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.1.1-1ubuntu2.1~18.04.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssl (1.1.1-1ubuntu2.1~18.04.10) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

r-bioc-rtracklayer/1.38.0-1build1 (amd64, arm64, s390x, i386, armhf, ppc64el)
pgbouncer/1.8.1-1build1 (i386)
ruby2.5/2.5.1-1ubuntu1.10 (amd64, arm64, s390x, i386, armhf, ppc64el)
casync/2+61.20180112-1 (s390x)
mysql-5.7/5.7.35-0ubuntu0.18.04.1 (armhf)
ganeti/2.16.0~rc2-1build1 (arm64)
linux-hwe-5.0/5.0.0-65.71 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#openssl

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssl (1.1.1j-1ubuntu3.2) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

ruby-eventmachine/1.3~pre20201020-b50c135-2 (amd64)
scapy/2.4.4-4ubuntu1 (armhf, amd64, arm64, ppc64el)
python-a38/0.1.3-1 (armhf, amd64, s390x, arm64, ppc64el)
xmltooling/3.2.0-2 (amd64, arm64, s390x, ppc64el, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#openssl

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssl (1.1.1f-1ubuntu2.5) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

xmltooling/3.0.5-1build1 (armhf, ppc64el, amd64, s390x, arm64)
python-a38/0.1.2-2 (s390x, armhf, arm64, amd64, ppc64el)
uftp/4.10.1-1 (arm64)
python3.9/3.9.5-3~20.04.1 (armhf, amd64)
casync/2+20190213-1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#openssl

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Simon Chopin (schopin)]

For the focal migration, I have asked for the following britney hints:

https://code.launchpad.net/~schopin/britney/+git/hints-ubuntu/+merge/406594
https://code.launchpad.net/~schopin/britney/+git/hints-ubuntu/+merge/406593

The other tests seem to be flaky.

[Simon Chopin (schopin)]

For hirsute, all but two were flaky that turned green on rerun. The other ones will necessitate hints :

https://code.launchpad.net/~schopin/britney/+git/hints-ubuntu/+merge/406595
https://code.launchpad.net/~schopin/britney/+git/hints-ubuntu/+merge/406596

[Gunnar Hjalmarsson (gunnarhj)]

@Simon: You know that the packages in respective -proposed pocket need to be verified too, right?

[Simon Chopin (schopin)]

what do you mean by "verified"?

[Gunnar Hjalmarsson (gunnarhj)]

I mean (for respective series):

* Accomplish the steps in the Test Plan
* Disclose the result in a comment here
* Change the verification-needed-<series> tag to verification-done-<series>

See comment #22 - #24.

[Simon Chopin (schopin)]

Verified on focal:

ubuntu@ubuntu:~$ dpkg-architecture --is s390x && [ "$(dpkg-query -W -f='${Version}' libssl1.1)" = "1.1.1f-1ubuntu2.5" ] && gcc -o aes-test test.c -lcrypto -lssl && ./aes-test && echo OK
OK

[Simon Chopin (schopin)]

OK, scratch that last comment. I had forgotten to check that the test program would actually trigger on the old case, which it did NOT in my VM, which makes sense as the patched code-path do some capability checking on the CPU. Silly me.

As the only s390x machine I have access to in a short time-frame is running Bionic, would it possible for people over at IBM to do a verification for the focal and hirsute SRUs ? Ping @fheimes

[Simon Chopin (schopin)]

This time properly tested and verified on bionic via juju/canonistack, where I sadly cannot boot any other version. The test fails properly there in the -updates version, and succeeds (without having to recompile) with the version in -proposed.

In addition, I round-tripped a file through AES-256-CBC and AES-128-ECB for sanity-checking.

[Frank Heimes (fheimes)]

Sorry for the late response, but was quite busy.
Yes I always have some systems in place (@schopin you can always ping me per MM is s/t is needed).

I just did the verifications for hirsute and focal and both are fine:

ubuntu@h7:~$ arch && lsb_release -c
s390x
Codename: hirsute
ubuntu@h7:~$ apt-cache policy openssl
openssl:
  Installed: 1.1.1j-1ubuntu3.2
  Candidate: 1.1.1j-1ubuntu3.2
  Version table:
 *** 1.1.1j-1ubuntu3.2 500
        500 http://us.ports.ubuntu.com/ubuntu-ports hirsute-proposed/main s390x Packages
        100 /var/lib/dpkg/status
     1.1.1j-1ubuntu3 500
        500 http://ports.ubuntu.com/ubuntu-ports hirsute/main s390x Packages
ubuntu@h7:~$ gcc test.c -o evc-test -lcrypto -lssl
ubuntu@h7:~$ ./evc-test && echo OK
OK
ubuntu@h7:~$

ubuntu@s15:~$ arch && lsb_release -c
s390x
Codename: focal
ubuntu@s15:~$ apt-cache policy openssl
openssl:
  Installed: 1.1.1f-1ubuntu2.5
  Candidate: 1.1.1f-1ubuntu2.5
  Version table:
 *** 1.1.1f-1ubuntu2.5 500
        500 http://us.ports.ubuntu.com/ubuntu-ports focal-proposed/main s390x Packages
        500 http://ports.ubuntu.com/ubuntu-ports focal-proposed/main s390x Packages
        100 /var/lib/dpkg/status
     1.1.1f-1ubuntu2.4 500
        500 http://ports.ubuntu.com/ubuntu-ports focal-updates/main s390x Packages
     1.1.1f-1ubuntu2.3 500
        500 http://ports.ubuntu.com/ubuntu-ports focal-security/main s390x Packages
     1.1.1f-1ubuntu2 500
        500 http://ports.ubuntu.com/ubuntu-ports focal/main s390x Packages
ubuntu@s15:~$ gcc test.c -o evc-test -lcrypto -lssl
ubuntu@s15:~$ ./evc-test && echo OK
OK
ubuntu@s15:~$

I'm updating the tags accordingly ...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for openssl has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.1.1j-1ubuntu3.2

---------------
openssl (1.1.1j-1ubuntu3.2) hirsute; urgency=medium

  * Cherry-pick an upstream patch to fix s390x AES code (LP: #1931994)

openssl (1.1.1j-1ubuntu3.1) hirsute; urgency=medium

  * Split d/p/pr12272.patch into multiple patchfiles to fix dpkg-source
    error when attempting to build a source package, due to pr12272.patch
    patching files multiple times within the same patch. (LP: #1927161)
    - d/p/lp-1927161-1-x86-Add-endbranch-to-indirect-branch-targets-fo.patch
    - d/p/lp-1927161-2-Use-swapcontext-for-Intel-CET.patch
    - d/p/lp-1927161-3-x86-Always-generate-note-gnu-property-section-f.patch
    - d/p/lp-1927161-4-x86_64-Always-generate-note-gnu-property-sectio.patch
    - d/p/lp-1927161-5-x86_64-Add-endbranch-at-function-entries-for-In.patch

 -- Simon Chopin <email address hidden> Fri, 23 Jul 2021 14:32:42 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.1.1f-1ubuntu2.5

---------------
openssl (1.1.1f-1ubuntu2.5) focal; urgency=medium

  * Cherry-pick an upstream patch to fix s390x AES code (LP: #1931994)

 -- Simon Chopin <email address hidden> Fri, 23 Jul 2021 14:32:42 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.1.1-1ubuntu2.1~18.04.10

---------------
openssl (1.1.1-1ubuntu2.1~18.04.10) bionic; urgency=medium

  * Cherry-pick an upstream patch to fix s390x AES code (LP: #1931994)

 -- Simon Chopin <email address hidden> Fri, 23 Jul 2021 14:32:42 +0200