OpenSMTPd not working with OpenSSL 3

[Alex Conrad]

So, I upgraded to Ubuntu 22.04 earlier today, and just noticed all mail is dismissed with this cryptic error:

smtp disconnected reason="io-error: error:0A080006:SSL routines::EVP lib"

Running smtpd -dv -Tall gives these errors:

debug: SSL library error: io_dispatch_accept_tls:SSL_accept: error:020000B3:rsa routines::missing private key
debug: SSL library error: io_dispatch_accept_tls:SSL_accept: error:1C880004:Provider routines::RSA libdebug: SSL library error: io_dispatch_accept_tls:SSL_accept: error:0A080006:SSL routines::EVP lib
smtp: 0x623af4b1f100: IO_ERROR <io:0x623af4be2400 fd=21 to=300000 fl=R tls=TLSv1.3:TLS_AES_256_GCM_SHA384:256 ib=0 ob=0>
6dd245e51e9972cc smtp disconnected reason="io-error: error:0A080006:SSL routines::EVP lib"

Some googling led me to this:

https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/25393

I'd say this is pretty severe, as opensmtpd doesn't seem to work at all on the latest stable Ubuntu. My mailserver is down. I had to create a temporary mail just to create this account to report this ticket.

So, this is on Ubuntu 22.04, with opensmtpd 6.8.0p2-4build1. I expect mail to be delivered, instead it is dismissed.
This only affects mail delivered through smtp. Local mail works fine. But just local mail is kinda pointless (:

[Alex Conrad (alexco22)]

Here is the upstream bug: https://github.com/OpenSMTPD/OpenSMTPD/issues/1171

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[StephanBeal (sgbeal)]

Like the OP, this is affecting my server as well since the update to 22.04, but not _all_ inbound traffic is affected. Most is. There is one family of servers from whom i can receive mail just fine and spammers somehow manage to get through, but mail sent via gmail or my own local Thunderbird client both fail with the same errors as reported by the OP.

[Thomas Schiex (thomas-schiex)]

Same here. Is there a more recent package somewhere?

[StephanBeal (sgbeal)]

The short answer is no. The latest info, including build instructions which supposedly work, are in the upstream bug in comment #1.

[Nick (kousu)]

+1

comms3# swaks --to <email address hidden> --server forum.spinalcordmri.org -tls -p 587
=== Trying forum.spinalcordmri.org:587...
=== Connected to forum.spinalcordmri.org.
<- 220 forum.spinalcordmri.org ESMTP OpenSMTPD
 -> EHLO localhost
<- 250-forum.spinalcordmri.org Hello localhost [46.23.90.174], pleased to meet you
<- 250-8BITMIME
<- 250-ENHANCEDSTATUSCODES
<- 250-SIZE 36700160
<- 250-DSN
<- 250-STARTTLS
<- 250 HELP
 -> STARTTLS
<- 220 2.0.0 Ready to start TLS
*** TLS startup failed (connect(): error:1400A438:SSL routines:CONNECT_CR_CERT_REQ:tlsv1 alert internal error)
*** STARTTLS attempted but failed

Causes this server-side:

root@forum:~# smtpd -d -v -Tall
[...]
31986e3ca84a940e smtp connected address=X.X.X.X host=example.org
smtp: 0x55d8d89f5240: >>> 220 forum.spinalcordmri.org ESMTP OpenSMTPD
smtp: 0x55d8d89f5240: IO_LOWAT <io:0x55d8d89f6530 fd=16 to=300000 fl=W ib=0 ob=0>
smtp: 0x55d8d89f5240: IO_DATAIN <io:0x55d8d89f6530 fd=16 to=300000 fl=R ib=16 ob=0>
smtp: 0x55d8d89f5240: <<< EHLO localhost
smtp: 0x55d8d89f5240: STATE_CONNECTED -> STATE_HELO
smtp: 0x55d8d89f5240: >>> 250-forum.spinalcordmri.org Hello localhost [X.X.X.X], pleased to meet you
smtp: 0x55d8d89f5240: >>> 250-8BITMIME
smtp: 0x55d8d89f5240: >>> 250-ENHANCEDSTATUSCODES
smtp: 0x55d8d89f5240: >>> 250-SIZE 36700160
smtp: 0x55d8d89f5240: >>> 250-DSN
smtp: 0x55d8d89f5240: >>> 250-STARTTLS
smtp: 0x55d8d89f5240: >>> 250 HELP
smtp: 0x55d8d89f5240: IO_LOWAT <io:0x55d8d89f6530 fd=16 to=300000 fl=W ib=0 ob=0>
smtp: 0x55d8d89f5240: IO_DATAIN <io:0x55d8d89f6530 fd=16 to=300000 fl=R ib=10 ob=0>
smtp: 0x55d8d89f5240: <<< STARTTLS
smtp: 0x55d8d89f5240: >>> 220 2.0.0 Ready to start TLS
smtp: 0x55d8d89f5240: STATE_HELO -> STATE_TLS
smtp: 0x55d8d89f5240: IO_LOWAT <io:0x55d8d89f6530 fd=16 to=300000 fl=W ib=0 ob=0>
mproc: pony -> lka : 11 IMSG_CERT_INIT
imsg: lka <- pony: IMSG_CERT_INIT (len=11)
debug: looking up pki "kousu"
mproc: lka -> pony: realloc 128 -> 8192
mproc: lka -> pony : 5608 IMSG_CERT_INIT
imsg: pony <- lka: IMSG_CERT_INIT (len=5608)
debug: session_start_ssl: switching to SSL
debug: pony: rsae_priv_enc
debug: pony: rsae_bn_mod_exp
debug: SSL library error: io_dispatch_accept_tls:SSL_accept: error:020000B3:rsa routines::missing private key
debug: SSL library error: io_dispatch_accept_tls:SSL_accept: error:1C880004:Provider routines::RSA lib
debug: SSL library error: io_dispatch_accept_tls:SSL_accept: error:0A080006:SSL routines::EVP lib
smtp: 0x55d8d89f5240: IO_ERROR <io:0x55d8d89f6530 fd=16 to=300000 fl=R tls=TLSv1.3:TLS_AES_256_GCM_SHA384:256 ib=0 ob=0>
31986e3ca84a940e smtp disconnected reason="io-error: error:0A080006:SSL routines::EVP lib"

With

root@forum:/var/discourse# apt policy opensmtpd
opensmtpd:
  Installed: 6.8.0p2-4build1
  Candidate: 6.8.0p2-4build1
  Version table:
 *** 6.8.0p2-4build1 500
        500 http://mirrors.digitalocean.com/ubuntu jammy/universe amd64 Packages
        100 /var/lib/dpkg/status
root@forum:/var/discourse# apt policy openssl
openssl:
  Installed: 3.0.2-0ubuntu1.6
  Candidate: 3.0.2-0ubuntu1.6
  Version table:
 *** 3.0.2-0ubuntu1.6 500
        500 http://mirrors.digitalocean.com/ubuntu jammy-updates/main amd64 Packages
        500 ...

[Nick (kousu)]

related: https://salsa.debian.org/debian/opensmtpd/-/merge_requests/3

[Terra Nova (z-ububtu-t)]

I just smashed head first into the problem tonight trying to get OpenSMTPd working and ran into the same problems as described above.

https://github.com/OpenSMTPD/OpenSMTPD/issues/1171

I am happy to report that the recently released 7.3.0p1 has fixed the OpenSSL-v3 problems.
https://github.com/OpenSMTPD/OpenSMTPD/releases/tag/7.3.0p1

I basically took the source package for 6.8.0p2-4build1, made a new entry in the changelog, built, and installed.

Hopefully this will updated soon in the official Ubuntu repository, with a package for 'jammy' as well.

[Rowan Wookey (rwky)]

Debian has it in backports, if we could get the same here that would do https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055758

In the meantime I resorted to compiling+installing into /usr/local and adding a systemd override.