sshd will fail to start or restart if non-default Port option is incorrectly put after a non-default ListenAddress

[jhartr7]

I have a system where the default listening port is changed for sshd, done at the bottom of my sshd_config. I recently changed the ListenAddress from its default (unset, which listens on all interfaces) to bind to a specific IP. The resulting relevant snippet of my sshd_config was:

ListenAddress my_ip_here
Port 12345

I then restarted ssh with /etc/init.d/ssh restart and found that ssh was not listening.

Re-reading the manpage, I see this:

ListenAddress
             Specifies the local addresses sshd(8) should listen on. The fol‐
             lowing forms may be used:

                   ListenAddress host|IPv4_addr|IPv6_addr
                   ListenAddress host|IPv4_addr:port
                   ListenAddress [host|IPv6_addr]:port

             If port is not specified, sshd will listen on the address and all
             prior Port options specified. The default is to listen on all
             local addresses. Multiple ListenAddress options are permitted.
             Additionally, any Port options must precede this option for non-
             port qualified addresses.

On a hunch, I remove the Port line and simply change my relevant config to:

ListenAddress my_ip_here:12345

sshd then restarted just fine.

In looking at the logs, I see:

Dec 1 11:40:26 jhart-laptop kernel: [1207947.954263] init: ssh main process (26655) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.954277] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.957687] init: ssh main process (26658) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.957703] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.961094] init: ssh main process (26661) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.961115] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.964835] init: ssh main process (26664) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.964852] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.968028] init: ssh main process (26667) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.968043] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.971004] init: ssh main process (26670) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.971021] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.974044] init: ssh main process (26673) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.974087] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.977853] init: ssh main process (26676) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.977868] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.980928] init: ssh main process (26679) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.980944] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.984059] init: ssh main process (26682) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.984075] init: ssh main process ended, respawning
Dec 1 11:40:26 jhart-laptop kernel: [1207947.987118] init: ssh main process (26685) terminated with status 255
Dec 1 11:40:26 jhart-laptop kernel: [1207947.987132] init: ssh respawning too fast, stopped

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: openssh-server 1:5.9p1-5ubuntu1.7
ProcVersionSignature: Ubuntu 3.13.0-66.108~precise1-generic 3.13.11-ckt27
Uname: Linux 3.13.0-66-generic x86_64
ApportVersion: 2.0.1-0ubuntu17.13
Architecture: amd64
Date: Tue Dec 1 11:54:42 2015
InstallationMedia: Ubuntu 12.04.5 LTS "Precise Pangolin" - Release amd64 (20140807)
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/zsh
SourcePackage: openssh
UpgradeStatus: No upgrade log present (probably fresh install)

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Please attach an example sshd_config that fails as you describe, and then change the bug status back to New. Thanks!

[Robie Basak (racb)]

(ideally this would be the minimal change from the default configuration, to make it easiest to identify the cause of the problem)

[jhartr7 (jon-hart)]

Here is the default configuration file I had been using prior to modifications. AFAIK it is nearly identical to the default that comes with whatever version of OpenSSH that ships with 12.04.

[jhartr7 (jon-hart)]

I've attached my mostly default configuration that does not exhibit the problem as well as the configuration that does exhibit the problem. Obviously, to reproduce this you'll need to use a ListenAddress and perhaps a Port argument that suits your setup.

To reproduce this, take my original configuration, or simply ensure that your sshd_config has no ListenAddress or Port arguments (anywhere, in any order), and restart sshd, confirming it is listening on 22/TCP as normal:

$ sudo /etc/init.d/ssh restart
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service ssh restart

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the stop(8) and then start(8) utilities,
e.g. stop ssh ; start ssh. The restart(8) utility is also available.
ssh stop/waiting
ssh start/running, process 5968
$ sudo lsof -np 5968 |grep LISTEN
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/jhart/.gvfs
      Output information may be incomplete.
sshd 5968 root 3u IPv4 8942314 0t0 TCP *:ssh (LISTEN)
sshd 5968 root 4u IPv6 8942316 0t0 TCP *:ssh (LISTEN)

Then, take my modified configuration, or ensure that your ListenAddress specification is before the Port argument and the ListenAddress is set to something other than 0.0.0.0 (I think), and then restart sshd. You'll see that it didn't start, and is not running or listening:

$ sudo /etc/init.d/ssh restart
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service ssh restart

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the stop(8) and then start(8) utilities,
e.g. stop ssh ; start ssh. The restart(8) utility is also available.
ssh stop/waiting
ssh start/running, process 6008
$ sudo lsof -np 6008 |grep LISTEN
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/jhart/.gvfs
      Output information may be incomplete.
[ jhart@jhart-laptop (12/11/15 11:21:17) ~ <err: 1> ]
$ ps aux |grep sshd
jhart 6048 0.0 0.0 9388 924 pts/4 S+ 11:21 0:00 grep sshd

If you then swap the order of the ListenAddress and Port arguments, or use the combined ListenAddress:Port option, this works fine.

[Robie Basak (racb)]

Thank you for the update.

On looking again I noticed this from the documentation you quoted in your original report:

"If port is not specified, sshd will listen on the address and all prior Port options specified."

I'm sorry I didn't notice this the first time round. Given the word "prior" in that sentence, it seems to me that the behaviour you're reporting is as documented?

I will accept an argument that this is either counter-intuitive and shouldn't behave that way, or not documented clearly enough. But we can't fix that in Ubuntu directly, so I suggest that you engage upstream on that if you wish.

I'm marking this bug as Invalid on the basis that I believe the behaviour you're reporting is working as documented, but if you disagree please explain and change the bug status back to New.

[jhartr7 (jon-hart)]

Thanks Robie. I think you may have misunderstood what the bug is. The quote you reference is not really relevant in this case, because what I am reporting isn't that something goes wrong when a port is not specified.

The important line from the documentation is this, which I provided originally:

"Additionally, any Port options must precede this option for non-port qualified addresses."

So, yes, my configuration is invalid, but the behavior of sshd when this happens and the experience around it could use improvement. If it detects that the configuration is invalid, it should flag it as such and make it obvious from the logs/stdout/stdterr rather than letting the process try to restart and falsely report that it restarted successfully, because it didn't.

I agree that part of this could legitimately be a bug in OpenSSH.

[Robie Basak (racb)]

> So, yes, my configuration is invalid, but the behavior of sshd when this happens and the experience around it could use improvement. If it detects that the configuration is invalid, it should flag it as such and make it obvious from the logs/stdout/stdterr rather than letting the process try to restart and falsely report that it restarted successfully, because it didn't.

This seems reasonable, but as you point out it would be something that would need to be done upstream. I don't think it's appropriate for Ubuntu to deviate from upstream on this behaviour.

To set expectations I'll set this bug Won't Fix for Ubuntu, as I don't think we'd accept patches for this. But if upstream make related changes, we could change the bug status to track the availability of those changes in Ubuntu at that point.