Hi,

Sorry for interrupting your thread, this bug has been prioritized on our end (I work at NVIDIA) so I joined the triaging effort and I believe I found the root cause for the crash. For the record, I used wget, but it behaves the same to curl.

First of all, it seems that it isn't that far off from an old case - https://openssl-dev.openssl.narkive.com/97zf5qT2/what-is-the-really-proper-way-to-use-an-engine. The reason being that the configuration file is loaded twice:

#0  0x0000fffff7c80304 in CONF_modules_load_file () from /lib/aarch64-linux-gnu/libcrypto.so.1.1
#1  0x0000fffff7c805e8 in ?? () from /lib/aarch64-linux-gnu/libcrypto.so.1.1
#2  0x0000fffff7cfa03c in ?? () from /lib/aarch64-linux-gnu/libcrypto.so.1.1
#3  0x0000fffff79dadd4 in __pthread_once_slow (once_control=0xfffff7e44550, init_routine=0xfffff7cfa020) at pthread_once.c:116
#4  0x0000fffff7d434c4 in CRYPTO_THREAD_run_once () from /lib/aarch64-linux-gnu/libcrypto.so.1.1
#5  0x0000fffff7cfa608 in OPENSSL_init_crypto () from /lib/aarch64-linux-gnu/libcrypto.so.1.1
#6  0x0000fffff7c80578 in OPENSSL_config () from /lib/aarch64-linux-gnu/libcrypto.so.1.1
#7  0x0000aaaaaaae9e24 in ?? ()
#8  0x0000aaaaaaacfbe0 in ?? ()
#9  0x0000aaaaaaad29cc in ?? ()
#10 0x0000aaaaaaadde5c in ?? ()
#11 0x0000aaaaaaab8874 in ?? ()

And later again (practically in the same function):

#0  0x0000fffff7c80304 in CONF_modules_load_file () from /lib/aarch64-linux-gnu/libcrypto.so.1.1
#1  0x0000aaaaaaae9e68 in ?? ()
#2  0x0000aaaaaaacfbe0 in ?? ()
#3  0x0000aaaaaaad29cc in ?? ()
#4  0x0000aaaaaaadde5c in ?? ()
#5  0x0000aaaaaaab8874 in ?? ()

This means that the engine parsing and creation flow is done twice, which breaks several assumptions of both OpenSSL and the loaded engines:

1. Parsing the name "pka"
2. Creating a dynamic engine
3. Cloning the dynamic engine inside ENGINE_by_id()
3. Using the dynamic path to call the "bind" method of the loaded .so file (pka.so)
4. Populating the fields of the cloned dynamic engine on top of what the user still holds as ("e"):
        else if (strcmp(ctrlname, "dynamic_path") == 0) {
            e = ENGINE_by_id("dynamic");
5. dynamic_load() will try to once again ENGINE_add() the engine, but it is already in the engine list
6. The Add will fail, and the reference count won't be incremented, hence will stay at 1
7. The error will propagate and ENGINE_free() will get called because the load failed:
            if (!ENGINE_ctrl_cmd_string(e, "LOAD", NULL, 0))
                goto err;
8. The free()ed engine is actually the PKA engine which got created twice, and the destruction of the engine (which assumes it only get created/destroyed once) will free memory pointed by static variables inside of the PKA module.
9. The first PKA engine is still "alive" and when used it will access free()ed memory pointed by the static variables of PKA
10. We get a BUS error when invoking a function using a garbled fptr which got overridden because it is stored in a free()ed memory buffer. On ARM, BUS error is very common because the CPU can't execute commands from misaligned addresses, and with high probability the garbled value isn't divisible by 4.

It seems that several fixes could resolve this issue:
1. The configuration file shouldn't be loaded twice, this breaks several assumptions of the engines (as shown in the bug I linked above, which is from 11 years ago)
2. The fact that the dynamic engine starts as "dynamic" and the load replaces it with the loaded engine is extremely counter intuitive. The caller holds a single pointer, and it gets transformed into the loaded engine. Seeing there is already a following call of "e = ENGINE_by_id(name)", this should be the code line that transforms the user's engine to the one that was loaded, the dynamic engine shouldn't change it in-memory when the user thinks their pointer points at a dynamic engine.
3. If engines are to be expected to get added/freed multiple times, it should be announced clearly, so that engine developers would know they are expected to maintain their own reference count if using static variables. In general, it will probably be best if engines would only be loaded once (and point #1 will be fixed).

I hope my analysis will help resolve this issue. Feel free to reach out if more information is needed.