Investigate ASLR re-randomization being disabled for children

Bug #2071815 reported by Marc Deslauriers
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openssh (Ubuntu)
Fix Released
Medium
Nick Rosbrook
Noble
Fix Committed
Medium
Unassigned

Bug Description

[Impact]

The systemd-socket-activation.patch patch has an Ubuntu delta to fix bug 2011458, but this results in ASLR not being re-randomized for children because the patch delta does "rexec_flag = 0;".

This was discovered as part of the CVE-2024-6387 discovery by Qualys, and is mentioned in the disclosure itself:

Side note: we discovered that Ubuntu 24.04 does not re-randomize the
ASLR of its sshd children (it is randomized only once, at boot time); we
tracked this down to the patch below, which turns off sshd's rexec_flag.
This is generally a bad idea, but in the particular case of this signal
handler race condition, it prevents sshd from being exploitable: the
syslog() inside the SIGALRM handler does not call any of the malloc
functions, because it is never the very first call to syslog().

This is also mentioned in the release notes of OpenSSH 9.8:

Exploitation on non-glibc systems is conceivable but has not been
examined. Systems that lack ASLR or users of downstream Linux
distributions that have modified OpenSSH to disable per-connection
ASLR re-randomisation (yes - this is a thing, no - we don't
understand why) may potentially have an easier path to exploitation.

We should investigate why that was needed, and if an alternative way of fixing the original bug can be done.

[Test Plan]

We just want to test that when a connection is accepted by sshd, the child process re-execs. There is a log message at the debug level from sshd when this happens.

1. Enable debug-level logging in sshd:

$ echo "LogLevel DEBUG" >> /etc/ssh/sshd_config.d/log-level.conf

2. Watch the logs:

$ journalctl -t sshd -b -f

3. From another host, connect to the test machine:

$ ssh <user>@<test host>

4. On the test machine, among other messages, there should be a message noting the start of the re-exec, e.g.:

sshd[2212]: debug1: rexec start in 6 out 6 newsock 6 pipe 8 sock 9

[Where problems could occur]

Through the iterations of d/p/systemd-socket-activation.patch, there have been issues related to the re-exec behavior, and how the listen fds passed by systemd are handled. See [1][2] for examples. This patch hopes to finally resolve these issues.

However, as was the case with previous bugs in this area, problems would most likely be related to incorrectly closing, or not closing, socket fds in sshd.

[1] https://bugs.launchpad.net/bugs/2020474
[2] https://bugs.launchpad.net/bugs/2011458

Related branches

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Subscribing Nick, who appears to be the original delta author.

Tobias Heider (tobhe)
tags: added: rls-oo-incoming
Changed in openssh (Ubuntu):
status: New → Confirmed
summary: - Investigate ASLR being disabled for children
+ Investigate ASLR re-randomization being disabled for children
Revision history for this message
Julian Andres Klode (juliank) wrote :

Assigning this to Nick so we don't lose track of it and can discuss it next week

tags: added: foundations-todo
removed: rls-oo-incoming
Changed in openssh (Ubuntu):
assignee: nobody → Nick Rosbrook (enr0n)
Revision history for this message
Seth Arnold (seth-arnold) wrote :
Revision history for this message
Nick Rosbrook (enr0n) wrote :

I don't think those *help*, but that's good to know for when we merge an upstream version that includes those patches.

Changed in openssh (Ubuntu):
status: Confirmed → In Progress
importance: Undecided → Medium
Nick Rosbrook (enr0n)
Changed in openssh (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:9.6p1-3ubuntu18

---------------
openssh (1:9.6p1-3ubuntu18) oracular; urgency=medium

  * d/p/systemd-socket-activation.patch: don't clear rexec_flag
    It was pointed out that this is generally not a good idea, so undo this
    change. In order to do this, we need to restore the logic to ensure
    that when we have been re-executed, we ignore $LISTEN_PID because it
    won't match, but we still need to know $LISTEN_FDS. And, do not set
    FD_CLOEXEC on the fds passed from systemd, because we want them to
    survive the re-execution. (LP: #2071815)
  * d/p/systemd-socket-activation.patch: refresh patch

 -- Nick Rosbrook <email address hidden> Fri, 26 Jul 2024 11:54:36 -0400

Changed in openssh (Ubuntu):
status: Fix Committed → Fix Released
Nick Rosbrook (enr0n)
Changed in openssh (Ubuntu Noble):
importance: Undecided → Medium
tags: removed: foundations-todo
Nick Rosbrook (enr0n)
description: updated
Nick Rosbrook (enr0n)
Changed in openssh (Ubuntu Noble):
status: New → In Progress
Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Marc, or anyone else affected,

Accepted openssh into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:9.6p1-3ubuntu13.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssh (Ubuntu Noble):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-noble
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openssh/1:9.6p1-3ubuntu13.6)

All autopkgtests for the newly accepted openssh (1:9.6p1-3ubuntu13.6) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

crmsh/unknown (i386)
debvm/unknown (amd64)
fence-agents/4.12.1-2~exp1ubuntu4 (amd64)
freedombox/unknown (amd64)
g10k/unknown (amd64)
gvfs/unknown (amd64)
libnet-scp-perl/unknown (amd64)
libnet-sftp-foreign-perl/unknown (amd64)
libnetapp-perl/unknown (amd64)
nova/unknown (i386)
pkg-perl-tools/0.78 (amd64, armhf)
rancid/unknown (amd64)
tinyssh/20240101-2 (amd64, armhf)
ubuntu-boot-test/2 (amd64)
vorta/unknown (amd64)
wcc/unknown (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :
Download full text (11.8 KiB)

I have verified the fix using openssh-server from noble-proposed.

I created a container and enable noble-proposed:

nr@six:~$ lxc launch ubuntu:noble noble
Launching noble
nr@six:~$ lxc exec noble bash
root@noble:~# cat > /etc/apt/sources.list.d/proposed.sources << EOF
> Types: deb
> URIs: http://us.archive.ubuntu.com/ubuntu/
> Suites: noble-proposed
> Components: main universe
> Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
> EOF
root@noble:~# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu noble-security InRelease
Get:5 http://us.archive.ubuntu.com/ubuntu noble-proposed InRelease [265 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu noble-proposed/main amd64 Packages [180 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu noble-proposed/main Translation-en [48.6 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu noble-proposed/main amd64 Components [22.0 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu noble-proposed/main amd64 c-n-f Metadata [3556 B]
Get:10 http://us.archive.ubuntu.com/ubuntu noble-proposed/universe amd64 Packages [650 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu noble-proposed/universe Translation-en [79.1 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu noble-proposed/universe amd64 Components [68.0 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu noble-proposed/universe amd64 c-n-f Metadata [10.7 kB]
Fetched 1326 kB in 1s (1275 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
16 packages can be upgraded. Run 'apt list --upgradable' to see them.

Then, I confirmed the bug was present with the CURRENT version:

root@noble:~# echo "LogLevel DEBUG" >> /etc/ssh/sshd_config.d/log-level.conf

In another terminal on my host, I ran:

ssh ubuntu@10.19.111.212

to initiate a session. Then, back in the container:

root@noble:~# journalctl -t sshd -b -f
Oct 23 15:59:05 noble sshd[1283]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Oct 23 15:59:07 noble sshd[1283]: debug1: ssh_packet_read_poll2: resetting read seqnr 3 [preauth]
Oct 23 15:59:07 noble sshd[1283]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Oct 23 15:59:07 noble sshd[1283]: debug1: rekey in after 134217728 blocks [preauth]
Oct 23 15:59:07 noble sshd[1283]: debug1: KEX done [preauth]
Oct 23 15:59:07 noble sshd[1283]: debug1: SSH2_MSG_EXT_INFO received [preauth]
Oct 23 15:59:07 noble sshd[1283]: debug1: kex_ext_info_check_ver: <email address hidden>=<0> [preauth]
Oct 23 15:59:08 noble sshd[1283]: debug1: userauth-request for user ubuntu service ssh-connection method none [preauth]
Oct 23 15:59:08 noble sshd[1283]: debug1: attempt 0 failures 0 [preauth]
Oct 23 15:59:08 noble sshd[1283]: debug1: PAM: initializing for "ubuntu"
Oct 23 15:59:08 noble sshd[1283]: debug1: PAM: setting PAM_RHOST to "10.19.111.1"
Oct 23 15:59:08 noble sshd[1283]: debug1: PAM: setting PAM_TTY to "ssh"
Oct 23 15:59:08 noble sshd[1283]: debug1: kex_server_update_ext_info: Sending SSH2_MSG_EXT_INFO [preauth]
Oct 23 15:59:08 noble ...

tags: added: verification-done verification-done-noble
removed: verification-needed verification-needed-noble
Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Marc, or anyone else affected,

Accepted openssh into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:9.6p1-3ubuntu13.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-noble
removed: verification-done verification-done-noble
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openssh/1:9.6p1-3ubuntu13.7)

All autopkgtests for the newly accepted openssh (1:9.6p1-3ubuntu13.7) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

crmsh/4.6.0-1ubuntu2 (amd64)
crmsh/unknown (i386)
fence-agents/unknown (amd64)
keychain/unknown (i386)
openmpi/4.1.6-7ubuntu2 (amd64)
openssh/1:9.6p1-3ubuntu13.7 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

The latest upload was restricted to a fix in d/t/sshd-socket-generator, so my previous verification stands.

tags: added: verification-done verification-done-noble
removed: verification-needed verification-needed-noble
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:9.6p1-3ubuntu13.7

---------------
openssh (1:9.6p1-3ubuntu13.7) noble; urgency=medium

  * d/t/sshd-socket-generator: run test_match_on_port test
    The test case was added to verify the fix for LP: 2076023,
    but it is not actually executed at the moment. Now that
    it does run, fix the grep commands used.

openssh (1:9.6p1-3ubuntu13.6) noble; urgency=medium

  * Explicitly listen on IPv4 by default, with socket-activated sshd
    (LP: #2080216)
    - d/systemd/ssh.socket: explicitly listen on ipv4 by default
    - d/t/sshd-socket-generator: update for new defaults and AddressFamily
    - sshd-socket-generator: handle new ssh.socket default settings
  * sshd-socket-generator: do not parse server match config
    (LP: #2076023)
  * d/p/systemd-socket-activation.patch: don't clear rexec_flag
    (LP: #2071815)
  * d/p/sshd-socket-generator.patch: add note to sshd_config
    Explain that a systemctl daemon-reload is needed for changes
    to Port et al to take effect.
    (LP: #2069041)
  * debian/openssh-server.ucf-md5sum: add new checksums for sshd_config

 -- Nick Rosbrook <email address hidden> Wed, 23 Oct 2024 14:19:51 -0400

Changed in openssh (Ubuntu Noble):
status: Fix Committed → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Update Released

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

1:9.6p1-3ubuntu13.7 was moved back to noble-proposed due to a possible regression[1] that is being investigated. Setting the bug tasks back to fix committed.

1:9.6p1-3ubuntu13.5 was restored in noble-updates.

1. https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2087551

Changed in openssh (Ubuntu Noble):
status: Fix Released → Fix Committed
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Adding block-proposed-noble to avoid accidental release while the regression in https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2087551 is investigated.

tags: added: block-proposed-noble
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.