Resource Sharing with multiple sshd services

[Luke A. Perkins]

Ubuntu: 18.04.2 LTS
OpenSSH: 7.6p1

I am having a problem starting multiple sshd processes. The default location of the sshd privilege separation directory is hard-coded to /run/sshd (see man page). If I want to have 2 sshd services using systemd, I need to write 2 service files, let's call them sshd_wan.service ans sshd_lan.service. Both of these services need to have their own "RuntimeDirectory=sshd_wan" and "RuntimeDirectory=sshd_lan". If you do not have separate RuntimeDirectory definitions for the 2 services, then when one service is killed/faults/restarts/stops/etc. the systemd (or init) process deletes the RuntimeDirectory and causes the other service to crash since a RuntimeDirectory does not exist.

The problem is the hard-coding of the sshd Privilege Separation Directory. We need to modify the OpenBSD/OpenSSH sshd code to provision command line assignment of the privilege separation directory.

I have attempted to contact the OpenSSH team (i.e. OpenSSH.com) and they say it is a Ubuntu problem. I reported this in Ubuntu bug #1831765 and Ubuntu (e.g. Paride Legovini, June 6, 2019 @ 2:55AM PDT) rejected it because I described the problem using the init.d example.

I know how to modify the sshd.c file in OpenSSH 7.6p1, the problem is getting Ubuntu and OpenSSH to admit there is a problem and it needs to be fixed.

The problem is still there regardless if you are using Upstart (i.e. init.d) or systemd.

[Robie Basak (racb)]

Thank you for taking the time to file this bug and helping to make Ubuntu better.

> ...the problem is getting Ubuntu and OpenSSH to admit there is a problem and it needs to be fixed.

It's up to individual projects to decide what configurations they want to support. Just because you can't configure your system to your exact specification doesn't necessarily mean that it's a problem for the project.

I understand what you're requesting, but I don't think Ubuntu will be prepared to maintain a patch in sshd to make the privilege separation directory configurable, assuming that upstream don't wish to do this either.

It may that there's something I'm missing and the problem can be fixed in Ubuntu, but you haven't relayed the message from upstream so I am unable to comment on that. If you'd like to expand on why exactly they think "it is a Ubuntu problem", then I can look again.

As I don't think Ubuntu will maintain the type of patch you suggest, I'm marking this bug as Won't Fix against the Ubuntu openssh package.

You might be able to use mount namespaces to give your different sshd processes different views of /run/sshd.

However, please note that you can simply comment if you have further information that you think would change this opinion, and change the status back to New yourself to request reconsideration. No need to file a new bug.

[Luke A. Perkins (public-a)]

Robbie, If I upload the sshd.c proposed change, will that be possibility? I have diffed the sshd.c code against the OpenSSH 7.6p1 source. Ubuntu has made significant and substantial changes to all of the OpenSSH source. So I know Ubuntu does not use the original OpenSSH code verbatim.

Is there anyway to change your mind from "Won't Fix" to "Investigating"?

[Robie Basak (racb)]

> If I upload the sshd.c proposed change, will that be possibility?

Please do attach the diff for reference - that is useful for the record.

I think it will still be unlikely though. Introducing new configuration options in a distribution delta is particularly painful because if upstream later implement something differently, we'll be stuck supporting an obsolete upgrade path forever. I believe this has already happened to the openssh package in Ubuntu. Just because Ubuntu patches upstream sources for other reasons doesn't automatically qualify doing so again for a new reason. The argument for a new patch will need to be made on its own merits.

I suggest that if you want to push this further, you start by identifying exactly what patch it is that upstream won't take, understand exactly why they won't take it, why existing upstream code isn't sufficient for your needs even though upstream believe it is, understand how that applies to Ubuntu's delta if the reasons are different, and present that all here.

[Luke A. Perkins (public-a)]

Robie, There are several options moving forward:
1) We need the /run/sshd file (see Ubuntu's man sshd page) to be configurable. The problem with locating the privilege separation directory in a fixed location, the systemd does not do well in multiple sshd instance assignments. The systemd will delete the processes RuntimeDirectory upon completion of the process.
2) The original OpenSSH 7.6p1 source code assigns the privilege separation directory to /var/empty (see OpenSSH man sshd page). If we assign it to /var/empty, then we get into a philosophical argument about making the /var/empty directory in an Ubuntu system.
The frustration I have with both the OpenSSH teams and the Ubuntu teams is neither want to take ownership. I am trying to provide a solution to both teams and I am getting complete rejection.
As far as the upstream support, we have 2 options, specifically:
1) Implement a command line option; I propose [-s separation_directory_name]. This would required editing only 1 file (i.e. sshd.c), so upstream modifications would be minimal.
2) Implement a sshd_config option; I propose "PrivSepDir separation_directory_name". This has less of a chance of conflicting with any upstream change. I cannot imagine a conflict but someone always has a better mouse-trap.
So how can we come to consensus on this?

[Robie Basak (racb)]

Thank you for the additional information.

> The original OpenSSH 7.6p1 source code assigns the privilege separation directory to /var/empty (see OpenSSH man sshd page).

Does that somehow mean that your problem doesn't occur if you use only the upstream source code and no distribution patches? If so, how?

> The frustration I have with both the OpenSSH teams and the Ubuntu teams is neither want to take ownership. I am trying to provide a solution to both teams and I am getting complete rejection.

Nobody owes you any duty to take ownership. Developers usually care about issues proportionately to how widely they affect users. I understand the problem you're facing, but right now it seems to affect only you, and so I don't think it warrants "taking ownership" by any team. I don't see this happening unless someone is persuaded on technical merits such as applicability to a wider use case or a lower maintenance burden to carry a patch.

Separately from that, if someone offers a patch, as you are doing, then we are grateful and we will, as a project, make a decision as to whether it will take it, decline it or require the issue to be resolved in a different way before accepting it.

> So how can we come to consensus on this?

The consensus amongst Ubuntu developers is currently "Won't Fix" for the reasons I've given already. As I said, you're welcome to continue discussion on the technical issues, but on the social side you do seem have a mistaken expectation that "Won't Fix" somehow means that some Ubuntu developer is going to "take ownership".

[Luke A. Perkins (public-a)]

> Does that somehow mean that your problem doesn't occur if you use only the upstream source code and no distribution patches? If so, how?

Good question. I have cloned the git version of OpenSSH for disco-proposed. When I compile this version of the code, the privilege separation directory is defined as "/var/empty" which would solve the problem. However, the installed version 18.04.2 LTS (bionic), has it defined as /run/sshd. The man pages for disco-proposed indicate that the privilege separation directory is at /run/sshd.

So, which git repository should I use to get the Ubuntu 18.04.2 (LTS) / OpenSSH 7.6p1 should I use?

[Luke A. Perkins (public-a)]

This is a proposed patch of "disco-proposed" of the sshd.c file. I have uploaded the original and the diff version.

[Luke A. Perkins (public-a)]

This is the git diff of sshd.c

[Luke A. Perkins (public-a)]

My recommmendation moving forward.

1) If Ubuntu wants to move the privilege separation directory from /var/empty to /run/sshd, then there needs to be a command-line option for the sshd to adjust the location of the privilege separation directory.

2) If Ubuntu keeps the privilege separation directory at /var/empty, then the man pages would need to be updated in the released openssh and no code changes would be required.

My $0.02 worth.

[Robie Basak (racb)]

On Fri, Jun 14, 2019 at 04:02:10PM -0000, Luke A. Perkins wrote:
> When I compile this version of the code, the privilege
> separation directory is defined as "/var/empty" which would solve the
> problem.

Why/how would this solve the problem?

> So, which git repository should I use to get the Ubuntu 18.04.2 (LTS) /
> OpenSSH 7.6p1 should I use?

You can use https://code.launchpad.net/ubuntu/+source/openssh to see the
sources used in Ubuntu's packaging. The applied/ubuntu/bionic-devel
branch will give you the current tree for 18.04 in Ubuntu with
distribution patches already applied.

[Robie Basak (racb)]

You're going round in circles.

Let's take a step back. Please assume that Ubuntu does not want to make
any change right now because no change is currently considered
justified.

An open question here is if Ubuntu's patches on upstream are creating
any problem that you're reporting. That's why I'm asking.

If they are, then please explain how and why.

If they are not, then I see no reason to make any change.

There's no point discussing this any further unless you can demonstrate
how Ubuntu is introducing any kind of problem that is unique to the
packaging and not upstream.