BN_cmp regards negative and positive zero as different numbers

[Guido]

"BN_cmp() returns -1 if a < b, 0 if a == b and 1 if a > b." ( https://wiki.openssl.org/index.php/Manual:BN_cmp(3) )

The libcrypto.a shipped with Ubuntu makes a distinction between a negative 0 ("-0") and a positive zero ("0"). This means that BN_cmp(-0, 0) returns -1 and BN_cmp(0, -0) returns 1. The latest versions of OpenSSL (1.0.2m and 1.1.0g) both return 0 for both comparisons, which is what you would expect.

lsb_release -rd:

Description: Ubuntu 16.04.3 LTS
Release: 16.04

Tested on both the 32 bit and 64 bit versions of this Ubuntu release.

Attached is a proof of concept.

The anomaly may also be caused by BN_dec2bn rather than BN_cmp.

The bug is unlikely to have direct security consequences, but may cause unexpected behaviour in other applications that depend on this widespread library.

Found with https://github.com/guidovranken/bignum-fuzzer

[Guido (guidovranken)]

This also affects BN_mod_add(-0, -0, 10) (result should be 0, but is 10)

[Adrien Nader (adrien)]

Thanks for the report and for the reproducer. I've tested it on 23.04 and got the following output:

    BN_cmp(A, B): 0
    BN_cmp(A, B): 0

IIUC the issue is therefore fixed.