openvpn crashes when run with fips openssl

[Joy Latten]

[IMPACT]
openvpn segfaults when using fips-mode openssl because of MD5.

xenial has version 2.3.x and subsequent releases have 2.4.x.
MD5 is used in 2 places in 2.3.x and one place in 2.4.x.

First place:
openvpn when estabishing a tls connection will segfault when used with Ubuntu's FIPS 140-2 libcrypto.so (openssl).

openvpn tls connection does TLS PRF(pseudorandom function) to produce securely generated pseudo random output that is used to generate keys.
MD5 is used as the hash in this computation.

FIPS 140-2 does not permit MD5 use except when used for pseudorandom function (PRF). When openvpn requests MD5 operation to FIPS-mode libcrypto.so, since it is not allowed in general, FIPS-mode libcrypto.so goes into an error state.

The context flag value, EVP_MD_CTX_FLAG_NON_FIPS_ALLOW, is defined in both FIPS and non-FIPS libcrypto.so. However, the MD5 check for it is only in FIPS-mode libcrypto.so to permit MD5. In non-FIPS libcrypto.so this check does not exist since it always permits MD5. openvpn should use this flag when it makes its MD5 request.

Second place (only in 2.3.x):
**NOTE: The openvpn 2.3 version in xenial has the above issue and an additional one. It also use MD5 internally for configuration status verification. It is not communicated externally. However, this particular use of MD5 is not allowed by FIPS and thus when openvpn tries to use FIPS-mode libcrypto.so to compute MD5, it results in openvpn segfaulting. This 2nd issue was fixed by upstream openvpn community in subsequent versions(2.4) to not use MD5 and use SHA(256) instead and thus why bionic, cosmic, and disco do not require any change for this 2nd issue.

[TEST]
Test data including commands and parameters are included below.

Testing comprised establishing a tls connection between an openvpn client and server. Once the connection was successfully established, a ping thru the established vpn tunnel was done from the client for assurance.

Interoperability testing was done to ensure no regression. Test data reflects testing was done between openvpn server and client with and without the patch and between various releases (xenial, bionic, and disco).

Test was also done with FIPS-enabled libcrypto.so to ensure everything worked in FIPS mode.

[REGRESSION]
The context flag value, EVP_MD_CTX_FLAG_NON_FIPS_ALLOW, is defined in both FIPS-mode openssl and non-FIPS openssl. However, the MD5-permit check against this flag-value does not occur in non-FIPS libcrypto.so, so there should be no change in behaviour. non-FIPS libcrypto.so should continue to service all MD5 requests.

xenial with version 2.3.x, has additional change of using SHA256 instead of MD5 for configuration status verification. This is an internal hash that is not communicated externally. Thus it should not regress interoperability or ability to establish connections.

[Christian Ehrhardt  (paelzer)]

Hi,
I'm really unsure what the expectation here is now.
This does not contain steps to reproduce the issue nor any suggested changes to make it work better.
Both would be needed.

Also is this actually the code in the main archive or any FIPS special PPA?

[Christian Ehrhardt  (paelzer)]

Checked on IRC, there are debdiff, testing data, etc...
Please set back to new once that was made available.

[Christian Ehrhardt  (paelzer)]

Thanks for the first update, when you attach the rest of the test data please make sure to not only add words like "comprised establishing a tls connection between an openvpn client and server" but more like:
#1 fresh container
$ command 1
$ command 2
<result>

[Joy Latten (j-latten)]

build log for disco: https://launchpad.net/~j-latten/+archive/ubuntu/joyppa/+build/15743680

[Joy Latten (j-latten)]

debdiff.disco

[Joy Latten (j-latten)]

testcase-data contains some of the data produces as a result of interoperability testing. It is applicable to xenial, bionic and disco.

[Joy Latten (j-latten)]

Hi Christian,

Hopefully the testcase-data file follows what you described. If not, let me know and I can reorganize it for improved readability.

[Joy Latten (j-latten)]

build log for bionic: https://launchpad.net/~j-latten/+archive/ubuntu/joyppa/+build/15743676

[Joy Latten (j-latten)]

build log for xenial: https://launchpad.net/~j-latten/+archive/ubuntu/joyppa/+build/15743720

[Joy Latten (j-latten)]

The xenial patch has additional code. In version 2.3.10, openvpn uses MD5 for PRF and internally for configuration status verification. FIPS 140-2 permits MD5 for PRF, but not as a hash for internal verification. Subsequent versions of openvpn (2.4) was changed upstream to not use MD5, instead uses SHA256. The attached patch provided by atsec uses SHA1 instead of MD5.

[Joy Latten (j-latten)]

2 testcases using same parameters for prior testcases, except that installed FIPS-mode libcrypto.so to test and ensure FIPS-mode libcrypto.so honors the flag to allow MD5 in PRF and does not cause openvpn to segfault because MD5 is missing.

[Andreas Hasenack (ahasenack)]

Taking a look

[Andreas Hasenack (ahasenack)]

Thanks for all this testing!

Could you please convert the debdiffs into actual merge proposals against openvpn? It's easier to review.

For example, the dep3 header in the xenial patch:
+Description: Use FIPS algos in openvpn
+Bug-Ubuntu:
+Forwarded: not-needed
+Author: Stephan Mueller <email address hidden>
+---
+OpenVPN uses MD5 for (1) internal configuration status verification
+and (2) TLS PRF. MD5 is not allowed in FIPS 140-2. Sending MD5 request
+to FIPS mode openssl causes it to abort or enter error mode.
+OpenVPN needs to use SHA instead of MD5 for internal verification and
+send EVP_MD_CTX_FLAG_NON_FIPS_ALLOW flag to openssl when using MD5 for
+PRF to indicate the exception.

a) Bug-Ubuntu should point at this bug url
b) Is there an upstream bug report? If yes, use "Bug: <url>" for it
c) The long description should be under the "Description:" header, indented by one space for each line
d) is there a link to the origin of the patch, like a commit?
e) in xenial the patch switched the internal usage of md5 to sha1, but later versions seem to be using sha256, any idea why not use sha256 in xenial as well, to follow upstream?

Feel free to ping me on irc for assistance with creating the MP. Basically you either:
- install the git-ubuntu snap, and run "git ubuntu clone openvpn". You will get some default branches you can branch off: ubuntu/devel is disco, ubuntu/xenial-devel is xenial, and so on.
- or just go to https://code.launchpad.net/ubuntu/+source/openvpn and clone/branch manually what you need

[Joy Latten (j-latten)]

Applied fixes for above comments. After some team discussion, decided to use sha256 for internal hash rather than sha1 in xenial as well. Internal hash is never communicated externally. Performed additional interoperability testing successfully using same test parameters as previously.

cosmic(with patch) <--> xenial (with patch)
cosmic(with patch) <--> xenial (with patch and in fips mode)
xenial(without patch) <--> xenial(with patch)
xenial(without patch) <--> xenial (with patch and fips mode)
xenial(with patch) <--> xenial (with patch)
xenial (with patch) <--> xenial (with patch and fips mode)
xenial (with patch and fips mode) <--> xenial(with patch and fips mode)

[Joy Latten (j-latten)]

This bug has been reported:
1.Upstream Bug: https://community.openvpn.net/openvpn/ticket/725
2.Suse Bug report: https://build.opensuse.org/package/view_file/network:vpn/openvpn/openvpn-fips140-2.3.2.patch

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvpn - 2.4.6-1ubuntu3

---------------
openvpn (2.4.6-1ubuntu3) disco; urgency=medium

  * d/p/openvpn-fips-2.4.patch: Allow MD5 in FIPS mode (openssl) for PRF.
    (LP: #1807439)

 -- Joy Latten <email address hidden> Wed, 09 Jan 2019 12:25:59 -0600

[Robie Basak (racb)]

Hello Joy, or anyone else affected,

Accepted openvpn into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.6-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Joy, or anyone else affected,

Accepted openvpn into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.4-2ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Joy, or anyone else affected,

Accepted openvpn into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.3.10-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Accepted, thanks.

For the record, I think it's a little blurry as to whether this is a bugfix or a new (FIPS-related) feature, but regardless I think it clearly qualifies under the "For Long Term Support releases we sometimes want to introduce new features" from a FIPS perspective. I think the record here shows that "these changes are unintrusive, have a minimal regression potential, and have been tested properly", subject to the usual SRU verification.

On the SRU verification point, please make sure to test again that everything works correctly with both FIPS and non-FIPS openssl, and again report the testing matrix here, including the package versions used.

Thanks!

[Robie Basak (racb)]

Oh, and a special thanks for explaining clearly all the details in the bug description. That got me up to speed quickly and allowed me to review without having to ask any questions!

[Chris Halse Rogers (raof)]

Hello there!

This bug looks like a really well-done SRU bug - a nice clear test case and justification.

The only thing it's missing now is actually testing the upload! :)

There's another openvpn SRU waiting in the queue now; could someone please do the testing for this so that we can release it into -updates and get the next openvpn into the testing queue.

Thanks!

[Joy Latten (j-latten)]

Testing in progress...

[Joy Latten (j-latten)]

Successfully verified xenial, bionic, and cosmic.

[Joy Latten (j-latten)]

Verified using same test data allowing for interoperability testing between the various releases and with fips for xenial and bionic.

[Joy Latten (j-latten)]

verification done on following:
xenial: openvpn-2.3.10-1ubuntu2.2
bionic: openvpn-2.4.4-2ubuntu1.2
cosmic: openvpn-2.4.6-1ubuntu2.1

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for openvpn has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvpn - 2.4.6-1ubuntu2.1

---------------
openvpn (2.4.6-1ubuntu2.1) cosmic; urgency=medium

  * d/p/openvpn-fips-2.4.patch: Allow MD5 in FIPS mode (openssl) for PRF.
    (LP: #1807439)

 -- Joy Latten <email address hidden> Thu, 10 Jan 2019 13:48:21 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvpn - 2.4.4-2ubuntu1.2

---------------
openvpn (2.4.4-2ubuntu1.2) bionic; urgency=medium

  * d/p/openvpn-fips-2.4.patch: Allow MD5 in FIPS mode (openssl) for PRF.
    (LP: #1807439)

 -- Joy Latten <email address hidden> Wed, 09 Jan 2019 15:50:03 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvpn - 2.3.10-1ubuntu2.2

---------------
openvpn (2.3.10-1ubuntu2.2) xenial; urgency=medium

  * d/p/openvpn-fips140-2.3.2.patch: Replace MD5 internal hash
    with SHA256 and allow MD5 for PRF. (LP: #1807439)

 -- Joy Latten <email address hidden> Wed, 09 Jan 2019 16:31:45 -0600