Do not support OpenSSL 3

[Lucas Kanashiro]

[Impact]

The ruby-net-ssh package in jammy fails most operations due to its incompatibility with OpenSSL 3.0, thus breaking most of its reverse-dependencies.

[Test case]

Amend the following one-liner for a host to which you have SSH access via pubkey:

ruby -e "require 'net/ssh'; Net::SSH.start('$REMOTE_HOST', '$REMOTE_USER') do |ssh| puts ssh.exec!('hostname') end"

It currently fails with the following error:
/usr/share/rubygems-integration/all/gems/net-ssh-6.1.0/lib/net/ssh/buffer.rb:316:in `set_key': rsa#set_key= is incompatible with OpenSSL 3.0 (OpenSSL::PKey::PKeyError

[Where problems could occur]

Even though the package is currently unusable, the fix could introduce problems via some subtle missuses of the new OpenSSL APIs.

[Original report]
Upstream still does not support OpenSSL 3, and due to that a bunch of tests are failing. Fedora has been seeing the same problem and they filed a bug upstream:

https://github.com/net-ssh/net-ssh/issues/843

[Lucas Kanashiro (lucaskanashiro)]

This issue is currently blocking ruby2.7 removal from Jammy. The removal requires the migration of ruby-bcrypt-pbkdf in jammy-proposed:

$ reverse-depends src:ruby2.7
Reverse-Depends
* ruby-bcrypt-pbkdf (for libruby2.7)

Packages without architectures listed are reverse-dependencies in: amd64, arm64, armhf, ppc64el, s390x

And ruby-net-ssh is the only regression blocking the ruby-bcrypt-pbkdf migration.

[Lucas Kanashiro (lucaskanashiro)]

It is also impacting vagrant already in Jammy (the release pocket). The following does not work because of this issue:

$ vagrant init debian/buster64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
$ vagrant up
Bringing machine 'default' up with 'libvirt' provider...
==> default: Box 'debian/buster64' could not be found. Attempting to find and install...
    default: Box Provider: libvirt
    default: Box Version: >= 0
==> default: Loading metadata for box 'debian/buster64'
    default: URL: https://vagrantcloud.com/debian/buster64
==> default: Adding box 'debian/buster64' (v10.20211230.1) for provider: libvirt
    default: Downloading: https://vagrantcloud.com/debian/boxes/buster64/versions/10.20211230.1/providers/libvirt.box
==> default: Successfully added box 'debian/buster64' (v10.20211230.1) for 'libvirt'!
==> default: Uploading base box image as volume into Libvirt storage...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default: -- Name: test_default
==> default: -- Description: Source: /tmp/test/Vagrantfile
==> default: -- Domain type: kvm
==> default: -- Cpus: 1
==> default: -- Feature: acpi
==> default: -- Feature: apic
==> default: -- Feature: pae
==> default: -- Clock offset: utc
==> default: -- Memory: 512M
==> default: -- Management MAC:
==> default: -- Loader:
==> default: -- Nvram:
==> default: -- Base box: debian/buster64
==> default: -- Storage pool: default
==> default: -- Image(): /var/lib/libvirt/images/test_default.img, 20G
==> default: -- Disk driver opts: cache='default'
==> default: -- Kernel:
==> default: -- Initrd:
==> default: -- Graphics Type: vnc
==> default: -- Graphics Port: -1
==> default: -- Graphics IP: 127.0.0.1
==> default: -- Graphics Password: Not defined
==> default: -- Video Type: cirrus
==> default: -- Video VRAM: 9216
==> default: -- Video 3D accel: false
==> default: -- Sound Type:
==> default: -- Keymap: en-us
==> default: -- TPM Backend: passthrough
==> default: -- TPM Path:
==> default: -- INPUT: type=mouse, bus=ps2
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 192.168.121.18:22
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Removing domain...
==> default: Deleting the machine folder
/usr/share/rubygems-integration/all/gems/net-ssh-6.1.0/lib/net/ssh/transport/kex/ecdh_sha2_nistp256.rb:21:in `generate_key!': pkeys are immutable on OpenSSL 3.0 (OpenSSL::PKey::PKeyError)
 from /usr/share/rubygems-integration/all/gems/net-ssh-6.1.0/lib/net/ssh/transport...

[Lucas Kanashiro (lucaskanashiro)]

The solution for this issue reported in ruby/openssl will fix part of the net-ssh issue:

https://github.com/ruby/openssl/issues/498

[Steve Langasek (vorlon)]

ruby-defaults 3.0 migrating to release means that ruby-net-ssh is now broken in the release pocket, so a baseline retest has let ruby-bcrypt-pbkdf so closing that task.

[Lucas Kanashiro (lucaskanashiro)]

I also filed this issue which involves some ruby-net-ssh test failures:

https://github.com/ruby/openssl/issues/500

[Lucas Kanashiro (lucaskanashiro)]

With the version in archive this is the summary of the execution of the test suite during the build:

1494 runs, 3679 assertions, 28 failures, 395 errors, 0 skips

Applying the WIP patch (attached) the summary is:

1494 runs, 3964 assertions, 16 failures, 332 errors, 0 skips

[Luis Alberto Pabón (copong)]

This currently breaks Vagrant.

[Simon Chopin (schopin)]

Here's my WIP patch, building upon Lucas' patch. If I run the tests with OPENSSL_CONF pointing to the ssl.conf file in this patch (which has the legacy provider enabled) I get

1504 runs, 4940 assertions, 4 failures, 23 errors, 0 skips

Deduplicating the errors, there are 4 types of failures that I can see:

EVP_PKEY_derive_set_peer failures, from the test_diffie_hellman_group14_sha1.rb (7 or 8 failures) -> This has me completely puzzled, as the tests work fine for the group1 data.

OpenSSL::PKey::PKeyError: EVP_PKEY_keygen: bad ffc parameters (most errors)
->I haven't looked into this one just yet

Authentication::TestKeyManager#test_identities_with_ecdsa_should_load_from_agent:
OpenSSL::PKey::PKeyError: pkeys are immutable on OpenSSL 3.0
-> This one is similar to many other already fixed

OpenSSL::PKey::DSAError: incorrect pkey type: dhpublicnumber
-> I looked quickly into it. I suspect it's a bug in the OpenSSL Ruby bindings?

I'll keep at it tomorrow.

NB: I'm running the test suite manually from upstream git with the sole Debian patch applied, which explains why the total number of tests differs.

[Simon Chopin (schopin)]

Attached is the newest version of the patch, which solves all failures mentioned above except for the EVP_PKEY_derive_set_peer diffie-hellman group 14 one:

1504 runs, 5067 assertions, 4 failures, 3 errors, 0 skips

Next steps are

* split the patch into smaller patches to ease upstream inclusion
* automatically load the custom ssl config with legacy providers if OpenSSL 3.0 is detected
* try to mock up the group 14 issue in a C PoC to get some OpenSSL upstream eyeballs on the problem

[Melvin Loos (melvin-7)]

Any update on this?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Lucas Kanashiro (lucaskanashiro)]

ruby-net-ssh/1:7.0.0~beta1-2 has the fix and it is in kinetic-proposed right now, waiting for the reverse dependencies tests to be executed to migrate to the release pocket. In the meantime, I am going to backport the needed changes to Jammy.

[Brian Murray (brian-murray)]

Hello Lucas, or anyone else affected,

Accepted ruby-net-ssh into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ruby-net-ssh/1:6.1.0-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lucas Kanashiro (lucaskanashiro)]

# Test Plan

$ lxc launch ubuntu-daily:jammy ruby-net-ssh-test
$ lxc shell ruby-net-ssh-test
# apt update && apt upgrade -y
<enable jammy-proposed>
# apt install -y ruby-net-ssh
# dpkg -l | grep ruby-net-ssh
ii ruby-net-ssh 1:6.1.0-2ubuntu0.1 all Ruby implementation of the SSH protocol
<try to access the host machine>
# ruby -e "require 'net/ssh'; Net::SSH.start('10.191.226.1', 'lucas') do |ssh| puts ssh.exec!('hostname') end"
lucas@10.191.226.1's password:
poseidon

No error as expected. I'll let other affected people test the package in jammy-proposed before flipping the tags to done.

[Lucas Kanashiro (lucaskanashiro)]

Since no one replied yet, I tried to setup a VM using vagrant as I described in comment #2, and it worked as expected:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ dpkg -l | grep ruby-net-ssh
ii ruby-net-ssh 1:6.1.0-2ubuntu0.1 all Ruby implementation of the SSH protocol
$ vagrant init debian/buster64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
$ vagrant up
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'debian/buster64' version '10.20211230.1' is up to date...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default: -- Name: test_default
==> default: -- Description: Source: /tmp/test/Vagrantfile
==> default: -- Domain type: kvm
==> default: -- Cpus: 1
==> default: -- Feature: acpi
==> default: -- Feature: apic
==> default: -- Feature: pae
==> default: -- Clock offset: utc
==> default: -- Memory: 512M
==> default: -- Management MAC:
==> default: -- Loader:
==> default: -- Nvram:
==> default: -- Base box: debian/buster64
==> default: -- Storage pool: default
==> default: -- Image(): /var/lib/libvirt/images/test_default.img, 20G
==> default: -- Disk driver opts: cache='default'
==> default: -- Kernel:
==> default: -- Initrd:
==> default: -- Graphics Type: vnc
==> default: -- Graphics Port: -1
==> default: -- Graphics IP: 127.0.0.1
==> default: -- Graphics Password: Not defined
==> default: -- Video Type: cirrus
==> default: -- Video VRAM: 9216
==> default: -- Video 3D accel: false
==> default: -- Sound Type:
==> default: -- Keymap: en-us
==> default: -- TPM Backend: passthrough
==> default: -- TPM Path:
==> default: -- INPUT: type=mouse, bus=ps2
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 192.168.121.3:22
    default: SSH username: vagrant
    default: SSH auth method: private key
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present......

[Marco Roeland (marcoroeland)]

I installed ruby-net-ssh version 1:6.1.0-2ubuntu0.1 from jammy-proposed as suggested above.
With this package installed I can again use vagrant to install a "generic/centos7" vagrant box, using the default libvirt provider. Thanks!

tags: verification-done verification-done-jammy

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ruby-net-ssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ruby-net-ssh - 1:6.1.0-2ubuntu0.1

---------------
ruby-net-ssh (1:6.1.0-2ubuntu0.1) jammy; urgency=medium

  * d/p/openssl-3/*.patch: backport upstream patches to support OpenSSL 3
    (LP: #1964025).
  * d/ruby-tests.rake: use custom OpenSSL config file if using OpenSSL 3.

 -- Lucas Kanashiro <email address hidden> Mon, 09 May 2022 18:44:20 -0300

[Alexander Adam (7ql6)]

Should these updates appear automatically at one point or do I have to activate some repos or so?

I'm still having the same issues like others as well:

https://stackoverflow.com/questions/71987581/openssl-3-0-error-when-booting-vagrantbox

[Lucas Kanashiro (lucaskanashiro)]

No, you do not need to enable any repository. ruby-net-ssh/1:6.1.0-2ubuntu0.1 landed in jammy-updates which should be enabled by default on your system.

The link to the stackoverflow page you sent is old, and at that time it was broken. This fix was even acknowledged on an upstream issue: https://github.com/hashicorp/vagrant/issues/12751#issuecomment-1132239703

[Lucas Kanashiro (lucaskanashiro)]

This fix is also released in kinetic, ruby-net-ssh/1:7.0.0~beta1-2 contains the fix.