ssh will not start at boot: systemctl status ssh -> output is listed as failed 255.

[Andrew Aldridge]

Using apt I have reinstalled and purged both ssh and openssh-server.
apt-get remove --purge ssh openssh-server
reinstalled
updated
and on boot the ctl status is failed 255

After much forum research and attempts to default solutions, the final word was that reinstall usually fixes this issue.
With the issue usually stemming from ssh service trying to start before network services are available.

Currently I have no logs available with
journalctl -u sshd | tail -100

I can provide info pertaining to this on request.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: ssh 1:6.9p1-2ubuntu0.1
ProcVersionSignature: Ubuntu 4.2.0-30.35-generic 4.2.8-ckt3
Uname: Linux 4.2.0-30-generic x86_64
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
CurrentDesktop: LXDE
Date: Wed Feb 24 14:27:55 2016
InstallationDate: Installed on 2016-02-09 (15 days ago)
InstallationMedia: Lubuntu 15.10 "Wily Werewolf" - Release amd64 (20151021)
PackageArchitecture: all
SourcePackage: openssh
UpgradeStatus: No upgrade log present (probably fresh install)

[Colin Watson (cjwatson)]

Please can you provide as much information as you have? For example, "the ctl status is failed 255" is clearly a paraphrase of something that you're seeing; it would be more useful if you could please copy and paste the original text of the error messages.

Forum research is probably leading you up entirely the wrong path; that sounds like information that predates systemd. The ssh.service unit has "After=network.target".

"journalctl -u sshd" won't find anything because that's not what the unit is called. Try "journalctl -u ssh".

[Andrew Aldridge (andrew-t-aldridge)]

Sure thing:

$ sudo systemctl status ssh
[sudo] password for andrewta:
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit) since Wed 2016-02-24 14:44:04 CST; 20s ago
  Process: 918 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
 Main PID: 918 (code=exited, status=255)

Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Main process exited, code.../a
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Unit entered failed state.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Failed with result 'exit-...'.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Service hold-off time ove...t.
Feb 24 14:44:04 janus-40 systemd[1]: Stopped OpenBSD Secure Shell server.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Start request repeated to...y.
Feb 24 14:44:04 janus-40 systemd[1]: Failed to start OpenBSD Secure Shell s...r.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Unit entered failed state.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Failed with result 'start...'.
Hint: Some lines were ellipsized, use -l to show in full.

journalctl -u ssh
-- Logs begin at Wed 2016-02-24 14:44:03 CST, end at Sat 2016-02-27 12:32:17 CST. --
Feb 24 14:44:03 janus-40 systemd[1]: Started OpenBSD Secure Shell server.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Unit entered failed state.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Failed with result 'exit-code'.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
Feb 24 14:44:03 janus-40 systemd[1]: Stopped OpenBSD Secure Shell server.
Feb 24 14:44:03 janus-40 systemd[1]: Started OpenBSD Secure Shell server.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Unit entered failed state.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Failed with result 'exit-code'.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
Feb 24 14:44:03 janus-40 systemd[1]: Stopped OpenBSD Secure Shell server.
Feb 24 14:44:03 janus-40 systemd[1]: Started OpenBSD Secure Shell server.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Unit entered failed state.
Feb 24 14:44:03 janus-40 systemd[1]: ssh.service: Failed with result 'exit-code'.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
Feb 24 14:44:04 janus-40 systemd[1]: Stopped OpenBSD Secure Shell server.
Feb 24 14:44:04 janus-40 systemd[1]: Started OpenBSD Secure Shell server.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Unit entered failed state.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Failed with result 'exit-code'.
Feb 24 14:44:04 janus-40 systemd[1]: ssh.service: Service hold-off time ov...

[Andrew Aldridge (andrew-t-aldridge)]

After boot starting ssh has never fails to start and correctly runs.

[Launchpad Janitor (janitor)]

[Expired for openssh (Ubuntu) because there has been no activity for 60 days.]

[Seth Arnold (seth-arnold)]

Tarifa on #ubuntu-server reported problems with his ssh service. The steps he found that he needed to make sshd start reliably:

systemctl disable ssh
update-rc.d ssh disable
systemctl enable ssh

"Then add both Wants=network-online.target & Requires=network-online.target in an ssh service override."

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bill Gradwohl (0cs935-bill)]

I installed version 16LTS recently and just upgraded the box to 18LTS and both display the error.

I have 2 NICs (Private / Public) with an sshd config file set to only listen to the private side. It appears sshd attempt to start after the Public NIC is up, but before the private NIC is available. The Private NIC becomes available several seconds after sshd is initiated according to journalctl.

[Bill Gradwohl (0cs935-bill)]

I created Bug 1774788 which is for rsyncd not starting at boot and was provided with a fix by Andreas Hasenack that works. I tried using that fix for ssh and it does not work.

Both services fail when an attempt is made to specify a particular NIC via their respective config files. In the case of rsync, the fix was to create the override.conf file containing:

[Unit]
After=network.target,network-online.target

via the command systemctl edit rsync.service

Using a similar procedure to create the override.conf file for ssh does not fix the ssh daemon startup issue. I'm curious as to why that is.

[Robie Basak (racb)]

> Both services fail when an attempt is made to specify a particular NIC via their respective config files.

If this is the case, then I don't think this can be treated as a bug in Ubuntu. Currently, if you change one end of the configuration that requires one thing, you need to change the other end of the configuration to make sure that the thing you've now required will be available at the right time.

I suppose there could be a Wishlist bug here that sshd will automatically notice that case and wait for the bind address to become configured. That'd best be implemented upstream and has its own problem cases (how does it know that the bind address will _ever_ be configured?)

[Bill Gradwohl (0cs935-bill)]

Robie: If the service offers a mechanism to set a specific NIC to listen on, then the service should make good on that offer when it's requested. Doesn't that make sense to you?

All services that require a NIC to function should wait for that NIC to come alive. Having a service start or attempt to start before what it needs for life support is available makes no sense. Allowing a service to start on a bogus IP address provides no "service". What's the point of having a service technically up if it's deaf to traffic?

I can't see where this qualifies as a wishlist item. The service advertises a feature and doesn't provide it. To me, that's a bug.

[Christian Ehrhardt  (paelzer)]

SSH to finally learn how to pick up addresses later is a long time demand for ssh to do that.
But it is a demand for an upstream feature IMHO.

I know it can thse day's re-pickup adresses that it lost:
- I have a KVM guest with one interface and I have set up ssh listen to only interface 192.168.122.221
  ListenAddress 192.168.122.221
- All normal/fine.
- If I take away the Network device it doesn#t work (obviously)
- If I attach the device ssh without being restarted works again.

But the problem here as already takled about is the fatal-fail of ssh to start if the ListenAddress not exists at the time.

And network-online.target might be a help for some cases, but not e.g. for networks added much later (like my virtual network example).
When I say this is a demand to upstream SSH for a long time I really mean it - this is in Ubuntu for a decade in bug 216847 and for like 3 years in upstream SSH.

I think this bug here should be used to change it to network-online.target if that really helps some cases.
The old bug I referred will continue to track until upstream implements IP_FREEBIND or any other late pickup tech (like listening to netlink events).

[Bill Gradwohl (0cs935-bill)]

For anyone that's interested ---

I added the following to root's crontab:
   @reboot sleep 120 && /bin/systemctl restart sshd

That works for me to get sshd up AND listening

[Robie Basak (racb)]

> Robie: If the service offers a mechanism to set a specific NIC to listen on, then the service should make good on that offer when it's requested. Doesn't that make sense to you?

Correct. If you configure the service to listen on a specific NIC, you should also configure the service to wait for that NIC to be up before it attempts to start.

> All services that require a NIC to function should wait for that NIC to come alive. Having a service start or attempt to start before what it needs for life support is available makes no sense.

So don't configure it like that.

> Allowing a service to start on a bogus IP address provides no "service". What's the point of having a service technically up if it's deaf to traffic?

What you're asking for is the system to algorithmically detect when it has been configured this way and defer starting of the service. This isn't a feature that currently exists. The workaround is to adjust service dependencies at the same time as you change your service configuration to add that dependency.

This problem is more complex than you might think. See https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ for some background.

> I can't see where this qualifies as a wishlist item. The service advertises a feature and doesn't provide it. To me, that's a bug.

Which service? This problem crosses an integration point: the upstream sshd daemon versus the service manager.

In any case, arguing about whether it is a bug or a feature doesn't change anything. If you'd like to see this fixed properly, please volunteer your time upstream to implement the proposed solution.