The Diffie-Hellman algorithm "modp6144" is not permitted in FIPS mode

[Pei Pei Xiao]

I have a Ubuntu server, and install the strongswan package with FIPS mode enabled.

1. check the ubuntu version
# cat /proc/version
Linux version 5.4.0-1072-fips (buildd@lcy02-amd64-020) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #81-Ubuntu SMP Wed Jan 25 11:07:24 UTC 2023

2. check the strongswan ipsec version
# ipsec version
Linux strongSwan U5.8.2/K5.4.0-1072-fips
University of Applied Sciences Rapperswil, Switzerland
See 'ipsec --copyright' for copyright information.

3. enable fips in ubuntu server
# sudo ua status
SERVICE ENTITLED STATUS DESCRIPTION
esm-apps yes enabled Expanded Security Maintenance for Applications
esm-infra yes enabled Expanded Security Maintenance for Infrastructure
fips-updates yes enabled NIST-certified core packages with priority security updates
livepatch yes enabled Canonical Livepatch service
usg yes disabled Security compliance and audit tools

NOTICES
A change has been detected in your contract.
Please run `sudo pro refresh`.

Enable services with: pro enable <service>

     Account: sky9090******
Subscription: Ubuntu Pro - free personal subscription

4. when I set the IKE Diffie-Hellman group to `modp6144`, the connection failed to be established.
conn all
       type=tunnel
       auto=start
       esp=aes256-sha256!
       ike=aes256-sha256-modp6144!
       left=%any
       ......
       ......
       leftauth=psk
       rightauth=psk
       keyexchange=ikev2
       lifetime=10800s
       ikelifetime=36000s
       dpddelay=30s
       dpdaction=restart
       dpdtimeout=120s

5. check the ipsec status, and see the connection is not up
# ipsec statusall
Status of IKE charon daemon (strongSwan 5.8.2, Linux 5.4.0-1072-fips, x86_64):
  uptime: 9 minutes, since Feb 13 06:06:00 2023
  malloc: sbrk 2969600, mmap 0, used 835264, free 2134336
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 0
  loaded plugins: charon test-vectors ldap pkcs11 tpm mgf1 rdrand nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf curve25519 agent chapoly ntru curl attr kernel-netlink resolve socket-default connmark farp stroke updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire led addrblock unity counters
Listening IP addresses:
  10.240.4.5
Connections:
Security Associations (0 up, 0 connecting):
  none

6. the following message was found in the log, and it says the algorithm `modp6144` is not permitted in FIPS mode.
Feb 13 06:06:00 ppei-vsi-001 charon: 05[CFG] algorithm 'modp6144' not permitted in fips mode
Feb 13 06:06:00 ppei-vsi-001 charon: 05[CFG] skipped invalid proposal string: aes256-sha512-modp6144

7. but I checked some NIST documentation and the DH algorithm `modp6144` is listed in the Approved Algorithms provided by the bound OpenSSL module. Therefore, based on my test results, I am not sure of the reason why the DH algorithm `modp6144` is not permitted in FIPS mode of Ubuntu strongswan.
https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3648.pdf (Table 9 – Approved Algorithms provided by the bound OpenSSL module)
https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4046.pdf (Table 8 – Approved and Allowed Algorithms provided by the bound OpenSSL module)

I also reported the issue on strongswan github: https://github.com/strongswan/strongswan/discussions/1537

[Lena Voytek (lvoytek)]

Thanks for the bug report.

Based on https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4292.pdf modp6144 should be available in 20.04. I'll mark this bug as todo

[Andreas Hasenack (ahasenack)]

The table titled "Diffie-Hellman groups" at the very end of this document[1], page 31, explicitly does NOT list modp6144:

modp2048 Regular group 14 (2048-bit modulus)
modp3072 Regular group 15 (3072-bit modulus)
modp4096 Regular group 16 (4096-bit modulus)
modp8192 Regular group 18 (8192-bit modulus)
modp2048s224 Modulo Prime Group 23 (2048-bit modulus, 224-bit subgroup)
modp2048s256 Modulo Prime Group 24 (2048-bit modulus, 256-bit subgroup)
ecp224 NIST Elliptic Curve Group 26
ecp256 NIST Elliptic Curve Group 19
ecp384 NIST Elliptic Curve Group 20
ecp521 NIST Elliptic Curve Group 21

But "Table 8 – Approved and Allowed Algorithms provided by the bound OpenSSL module" lists modp_6144, when provided by the openssl module.

I also checked the strongswan source package, and to no surprise, the patch that adds fips support also does not list MODP_6144.

At this point I'm unsure if the pdf is correct and the key thing is "when provided by openssl", or if it's a bug. I'll have to check with the "fips people". As far as I can tell, the openssl plugin is enabled by default. This algorithm can be provided by botan, openssl, and gcrypt. I'll check briefly if I'm using openssl in my test setup, and not one of the others.

1. https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4046.pdf

[Andreas Hasenack (ahasenack)]

swanctl --list-algs does imply that modp_6144 is provided by openssl, though:

# swanctl --list-algs 2>/dev/null| grep 6144
  MODP_6144[openssl]

[Andreas Hasenack (ahasenack)]

There is no FIPS for Jammy or Kinetic, marking those as invalid.

[Andreas Hasenack (ahasenack)]

This is pending a meeting with the certification lab to confirm that modp6144 can be added. If that's the outcome, then we can expect a strongswan update in the fips-update channel, and future (re-)certifications should include it as well.

[Tobias Heider (tobhe)]

After consulting with our FIPS lab, it looks like this is tricky to fix retroactively for 20.04 even with a recertification since it is not a simple security fix and would require an update to the security policy. Since the impact seems pretty low, we're moving this to "Won't Fix" for 20.04. We will however make sure that our future 22.04 module does support modp6144.