hashlib.algorithms_available lists algorithms that cannot be used

[Andreas Hasenack]

Ubuntu 22.10
python-ecdsa 0.18.0~b1-1

==================================== ERRORS ====================================
_ ERROR collecting .pybuild/cpython3_3.10_ecdsa/build/ecdsa/test_malformed_sigs.py _
/usr/lib/python3.10/hashlib.py:160: in __hash_new
    return _hashlib.new(name, data, **kwargs)
E ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:
ecdsa/test_malformed_sigs.py:42: in <module>
    hash_and_size = [
ecdsa/test_malformed_sigs.py:43: in <listcomp>
    (name, hashlib.new(name).digest_size) for name in algorithms_available
/usr/lib/python3.10/hashlib.py:166: in __hash_new
    return __get_builtin_constructor(name)(data)
/usr/lib/python3.10/hashlib.py:123: in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
E ValueError: unsupported hash type whirlpool
=========================== short test summary info ============================
ERROR ecdsa/test_malformed_sigs.py - ValueError: unsupported hash type whirlpool
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.25s ===============================
E: pybuild pybuild:369: test: plugin distutils failed with: exit code=2: cd '/home/ubuntu/x/python-ecdsa-0.18.0~b1/.pybuild/cpython3_3.10_ecdsa/build'; python3.10 -m pytest
dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p 3.10 returned exit code 13
make: *** [debian/rules:6: binary] Error 25
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

[Andreas Hasenack (ahasenack)]

Upstream: https://github.com/tlsfuzzer/python-ecdsa/issues/285

[Andreas Hasenack (ahasenack)]

This seems to be an issue with how python (3.10) is interacting with openssl.

The list of supported hash algorithms that we get is not actually supported:

$ python3 -c "import hashlib; a = {(name, hashlib.new(name).digest_size) for name in hashlib.algorithms_available}"
Traceback (most recent call last):
  File "/usr/lib/python3.10/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <setcomp>
  File "/usr/lib/python3.10/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/usr/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160

ripemd160 is in the list:
>>> hashlib.algorithms_available
{'shake_256', 'sha224', 'shake_128', 'sha512_224', 'blake2b', 'ripemd160', 'sha384', 'sha1', 'sha256', 'sha3_384', 'sha3_512', 'md5-sha1', 'sha512', 'whirlpool', 'sm3', 'md4', 'blake2s', 'sha512_256', 'sha3_224', 'sha3_256', 'md5'}

But like many others, unusable:
>>> hashlib.new("md4")
Traceback (most recent call last):
  File "/usr/lib/python3.10/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/usr/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md4

The list of hash names begins with this, in hashlib.py:
__always_supported = ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512',
                      'blake2b', 'blake2s',
                      'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512',
                      'shake_128', 'shake_256')

It then gets potentially augmented with openssl's list:
    import _hashlib
    new = __hash_new
    __get_hash = __get_openssl_constructor
    algorithms_available = algorithms_available.union(
            _hashlib.openssl_md_meth_names)

And indeed, md4 and ripemd160 (and others) come from openssl's list:
>>> _hashlib.openssl_md_meth_names
frozenset({'shake_256', 'sha224', 'shake_128', 'sha512_224', 'blake2b', 'ripemd160', 'sha384', 'sha1', 'sha256', 'sha3_384', 'sha3_512', 'md5-sha1', 'sha512', 'whirlpool', 'sm3', 'md4', 'blake2s', 'sha512_256', 'sha3_256', 'sha3_224', 'md5'})

and it's unusable:
>>> _hashlib.new("md4")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines] unsupported

This was nicely summarized in https://github.com/tlsfuzzer/python-ecdsa/issues/285#issuecomment-1040319586

Looks like the openssl legacy provider is not fully loaded: it's there "enough" to spit out the legacy hash na...

[Andreas Hasenack (ahasenack)]

And this is the fix for 3.10: https://github.com/python/cpython/pull/32085

[Andreas Hasenack (ahasenack)]

Marking the python-ecdsa task as invalid, as there is nothing to fix there.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simon Chopin (schopin)]

I just tested, and the issue is indeed fixed in Kinetic but is still present in Jammy. We'd need to SRU a cherry-pick of the hashlib patch mentioned in #3