sshd will not start at boot if ListenAddress is set, because network interface is not yet up

Bug #216847 reported by sektor
192
This bug affects 22 people
Affects Status Importance Assigned to Milestone
portable OpenSSH
Unknown
Unknown
openssh (Debian)
New
Unknown
openssh (Fedora)
Unknown
Unknown
openssh (Ubuntu)
Confirmed
Low
Unassigned
Nominated for Natty by KBios

Bug Description

Binary package hint: openssh-server

The sshd will not start at boot if the ListenAddress option in /etc/ssh/sshd_config is set to an IPv4 address other then 0.0.0.0 .

I am using Ubuntu 7.10 and the version 1:4.6p1-5ubuntu0.2 of the openssh-server package.
I would expect that sshd is started after boot but it will not and I found this in /var/log/auth.log:

sshd[4527]: error: Bind to port 22 on 10.1.1.22 failed: Cannot assign requested address.
sshd[4527]: fatal: Cannot bind any address.

Once the System is started you can start/stop the sshd with the /etc/init.d/ssh script without any problems.

Revision history for this message
Mathias Gug (mathiaz) wrote : Re: [Bug 216847] [NEW] sshd will not start at boot
  • unnamed Edit (189 bytes, application/pgp-signature; name="signature.asc")

On Sun, Apr 13, 2008 at 04:23:15PM -0000, sektor wrote:
> The sshd will not start at boot if the ListenAddress option in
> /etc/ssh/sshd_config is set to an IPv4 address other then 0.0.0.0 .
>
> I am using Ubuntu 7.10 and the version 1:4.6p1-5ubuntu0.2 of the openssh-server package.

Are you using Ubuntu Desktop or Ubuntu Server ?

> I would expect that sshd is started after boot but it will not and I found this in /var/log/auth.log:
>
> sshd[4527]: error: Bind to port 22 on 10.1.1.22 failed: Cannot assign requested address.
> sshd[4527]: fatal: Cannot bind any address.

Are you using NetworkManager to get the 10.1.1.22 ip address ? What is
you network configuration (wired, wireless) ?

  status incomplete

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

Changed in openssh:
status: New → Incomplete
Revision history for this message
sektor (sektor-1) wrote : Re: sshd will not start at boot

Am Montag, den 14.04.2008, 15:19 +0000 schrieb Mathias Gug:
> Are you using Ubuntu Desktop or Ubuntu Server ?

I use the Ubuntu Desktop Edition.

> Are you using NetworkManager to get the 10.1.1.22 ip address ? What is
> you network configuration (wired, wireless) ?
>

I use the Network Manager (nm-applet 0.6.5) for my wired conecction. The
IP address is assigned over dhcp at boot time. There is no iptables
setup on my machine. The complete network configuration is Ubuntu's
installation default. The distribution is up to date.

I was able to reproduce this issue on my laptop with the same software enviroment
but over a wireless link. The wireless Interface uses the bcm43xx module.
The wireless card is a build in Broadcom BCM4306 (rev 03).

I think this is not releated to my hardware because the same thing happens
on two completely different computers.

Regards,
Aleks

Changed in openssh:
status: Incomplete → New
Revision history for this message
Colin Watson (cjwatson) wrote :

network-manager is not guaranteed to actually bring the interface up until rather later on. Until it does, you won't be able to start sshd with that configuration.

One of these days, we may be able to fix this unexpressed dependency using Upstart.

Changed in openssh:
importance: Undecided → Low
status: New → Confirmed
sektor (sektor-1)
visibility: public → private
Colin Watson (cjwatson)
summary: - sshd will not start at boot
+ sshd will not start at boot if ListenAddress is set, because network
+ interface is not yet up
visibility: private → public
Revision history for this message
Greg Porterfield (gregport) wrote :

I'm having this exact symptom but in a different environment: Ubuntu 10.04 Lucid Server.

After configuring /etc/ssh/sshd_config to have a specific ListenAddress instead of 0.0.0.0, the ssh daemon consistently fails to start at boot time. Errors from /var/log/auth.log:

Sep 23 14:26:45 a02-gsp sshd[697]: error: Bind to port 22 on 10.100.50.19 failed: Cannot assign requested address.
Sep 23 14:26:45 a02-gsp sshd[697]: fatal: Cannot bind any address.

And errors in /var/log/syslog immediately after boot:

Sep 23 14:26:45 a02-gsp init: ssh main process (739) terminated with status 255
Sep 23 14:26:45 a02-gsp init: ssh main process ended, respawning
Sep 23 14:26:45 a02-gsp init: ssh main process (744) terminated with status 255
Sep 23 14:26:45 a02-gsp init: ssh respawning too fast, stopped

So this appears to be the same problem, but the root cause would seem to be upstart in this case. Since /etc/init/ssh.conf says "start on filesystem", sshd is started before the network interfaces are configured and fails to bind to the specific address that isn't configured yet.

I've implemented the following workaround which seems to be working well: changed the start event in ssh.conf to "start on started network-interface" and added a "sleep 3" in the pre-start script (see attached patch). Without the sleep to wait for the interface to be configured, sshd would still fail to start some of the time.

If this would be more appropriate as a new bug specific to Lucid Server I'll be happy to open a new one.

tags: added: patch
Revision history for this message
KBios (kbios) wrote :

Adding random sleeps is usually not the best possible solution, as the required delay varies between systems (for example, when using a network bridge). Maybe depending on net-device-up would be better.

Revision history for this message
ilf (ilf) wrote :

Three years and six releases later, this is still present :(

Revision history for this message
Claus Allweil (clal) wrote :

I have fixed this problem for me in /etc/network/interfaces

auto eth0
iface eth0 inet dhcp
  up service ssh start

Revision history for this message
Dirk Schuster (dirk-schuster) wrote :

I tested on 12.10.

Unfortunately the solution from Claus interferes with the network manager. On my 12.10 system I wasn't able to handle the eth0 by the network manager any more.

As I don't like arvitrary sleep times, I figured out a solution just changing the start on line in /etc/init/ssh. It works very well for me in 12.10..

I add this patch in a moment. It would be fine if someone would test it and confirm it.

Users of other ubuntu versions might have to change the line accordingly.

Revision history for this message
Dirk Schuster (dirk-schuster) wrote :
Revision history for this message
Dirk Schuster (dirk-schuster) wrote :

same problem with package openssh-server of Ubuntu 13.10 (Saucy)

error messages in /var/log/auth.log after boot process:
Nov 1 17:31:46 pc sshd[1209]: error: Bind to port xxx on yyy.yyy.y.y failed: Cannot assign requested address.
Nov 1 17:31:46 pc sshd[1209]: fatal: Cannot bind any address.

reason:
/etc/ssh/sshd_config uses ListenAddress option

solution:
in /etc/init/ssh.conf
change line
start on runlevel [2345]
to
start on runlevel [2345] and net-device-up IFACE!=lo

message in /var/log/auth.log after boot process now:
Nov 1 17:39:20 pc sshd[1518]: Server listening on yyy.yyy.y.y port xxx.

What needs to be done, so that the solution will get into the package? Could anybody help?

Revision history for this message
Dirk Schuster (dirk-schuster) wrote :

same in vivid

Mai 11 12:40:44 pcds systemd[1]: Unit ssh.service entered failed state.
Mai 11 12:40:44 pcds systemd[1]: ssh.service failed.
Mai 11 12:40:44 pcds systemd[1]: ssh.service holdoff time over, scheduling ...t.
Mai 11 12:40:44 pcds systemd[1]: start request repeated too quickly for ssh...ce
Mai 11 12:40:44 pcds systemd[1]: Failed to start OpenBSD Secure Shell server.
Mai 11 12:40:44 pcds systemd[1]: Unit ssh.service entered failed state.
Mai 11 12:40:44 pcds systemd[1]: ssh.service failed.

For the first time this error was reported 7 years ago. And it lives on and on and on ...

Revision history for this message
Simon Déziel (sdeziel) wrote : Re: [Bug 216847] Re: sshd will not start at boot if ListenAddress is set, because network interface is not yet up

Wouldn't it be possible to have sshd use "IP_FREEBIND" so that it can
bind an IP that has not materialized yet?

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Changing the ssh service file to use network-online.target should also work; see http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ for more information.

tags: added: systemd-boot
Martin Pitt (pitti)
tags: removed: systemd-boot
Revision history for this message
karl forner (karl-forner) wrote :

Would be nice if it was fixed once for all. This is a critical bug when your server is in a distant location, and you cannot log on it because of that bug.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Since so many things changed since this - admittedly embarrassing old - bug was reported I thought I retest it before the next LTS comes out.

I looked at the Xenial content and found stuff like:
Not only is it specified "After network.target", but on top there is /etc/network/if-up.d/openssh-server which is intended to restart openssh if later an iface comes up, there it would be able to rebind that IP.

I tested on containers first and it worked right away, setting ListenAddress and restarting.
But well in containers devices might be "up" at a different timing, but even in a KVM Guest, setting ListenAddress again it works just fine.
Mar 24 14:54:43 testxenial-ssh sshd[1293]: Server listening on 192.168.122.131 port 22.

I wondered and wanted to make sure that the test is valid, so I spawned
trusty (LTS), vivid (last reported issue) and wily

But - for me - it was working on all of them - even with several retries it was not even racy.

Therefore I wanted to ask for a retest - at least with Xenial if possible.
Also any detail about the test environment that was used would be great - especially by the last confirmation in vivid.
Does it have to be a special network card that might initialize slowly or anything like it?

Revision history for this message
Simon Déziel (sdeziel) wrote :

Under systemd, if the ListenAddress is on an interface that is manually brought up, the ifup script doesn't help. In that situation, the invoke-rc.d reload/restart fails because the initial startup of sshd wasn't successful.

Revision history for this message
ironstorm (ironstorm-gmail) wrote :

This is still broken in 16.04 / Xenial.

I'm going to try using a root cronjob to catch and recover:

*/5 * * * * if [ $(ps -ef | grep /usr/sbin/sshd | grep -v grep | wc -l) -eq 0 ]; then /usr/sbin/service ssh start; fi

Revision history for this message
jowfdoijdfdwfwdf (dsfkljo322332) wrote :

This has been broken for almost 10 years, still broken in 16.10

It is useful to set sshd ListenAddress so ssh won't be available when connected to a different network (laptop on a public wifi etc). However with ListenAddress set, sshd always fails a rebooot/network disconnect.

Revision history for this message
Owlbsidian (owlbsidian) wrote :

For me after changing network.target to network.service on /usr/lib/systemd/system/sshd.service it started at the right time.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

As Seth mentioned in comment #13 likely network-online.target should be the solution.
@Owlbsidian - could you test and confirm with that as a fix as well?

A related fun quote from the systemd page about it:
"... please just fix your program to be friendly to dynamically changing network configuration. That way you will make your users happy because things just start to work, and you will get fewer bug reports ..."

Without that support you can still run into issues if e.g. you have a manually added interface you depend on (no service target will help you there). But that in turn would be an upstream feature request more than anything else.

Anyway, as long as ssh doesn't support that we might want to finally tackle that issue to at least improve it. Now that systemd has more means to express "online".
Maybe even by directly fixing it in Debian.

Subscribing server to track and cjwatson for his opinion (actually I see he is auto-subscribed to the pkg so he will see).

Revision history for this message
Gennady (gendelflg) wrote :

Ubuntu 17.10
Changing network.target to network-online.target in /lib/systemd/system/ssh.service worked. I also added Wants=network-online.target (don't know if it's necessary but https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ suggests it)

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks for the confirmation Gennady.

Now that there is a suggested and confirmed solution, would anybody that personally requires (alway better to make an argument) this fix mind filing this in Debian as well to be changed in ./debian/systemd/ssh.service.
I'd think at least eventually we want to be in sync on this behavior.

@CJWatson - since you usually do the openssh uploads what do you think about adding that for now on your next upload?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Upstream server about using IP_FREEBIND: https://bugzilla.mindrot.org/show_bug.cgi?id=2512

Revision history for this message
The Marauder (arn-epsilon) wrote :

10 years ago ... Same bug !

Incredible :(

Revision history for this message
Rodman (rodman-c) wrote :

This is still broken in 18.04 Server. Setting the listen address fails to load the SSHD daemon at boot time. You have to have console access and do a "service sshd restart" to fix.

How is this bug still here???

Revision history for this message
Simon Déziel (sdeziel) wrote :

@Rodman, as a workaround, maybe you could try to add an "After=systemd-networkd-wait-online.service" clause in a drop-in snippet?

Revision history for this message
Simon Déziel (sdeziel) wrote :
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Just passing by to see if anything has changed. Would be good to know if the workaround is effective for those affected.

Revision history for this message
Con (c0ntr1but3) wrote :

@ahasenack, I found this post (took plenty of Google foo) but it was helpful.

For anyone else, 18.04 LTS STILL has this issue. The following fixed it:

After=network-online.target
Wants=network-online.target

These are in my [Unit] section at the top of /etc/systemd/system/sshd.service

<code_paste_from_service_file>
[Unit]
Description=OpenBSD Secure Shell server
After=network-online.target auditd.service
Wants=network-online.target
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
</code_paste_from_service_file>

I hope this is helpful to others!

Revision history for this message
nimish (nimishp12) wrote :

Came across this issue on a system I'm rebuilding and spun up a VM running 18.04.3 to test. It seems like I got it working consistently on every reboot. I'll try and breakdown what I've tried and what worked. Everyone & Dev's can let me know if they see any issues of setting it up the way it's working.

First made a backup of the file /etc/systemd/system/sshd.service just in case even though you could use systemctl revert ssh.service to revert any changes.

What I've tried:

editing /etc/systemd/system/sshd.service and changing the following lines from:

After=network.target auditd.service

to:

After=network-online.target
Wants=network-online.target

or to:

Requires=network-online.target
After=network-online.target
Wants=network-online.target

or to:

Requires=network-online.target auditd.service
After=network-online.target auditd.service
Wants=network-online.target auditd.service

or to:

Requires=whoopsie.service
After=whoopsie.service
Wants=whoopsie.service

What I was trying to see is which service I could have ssh start after seeing as using the various network.service/targets were just not working or consistent. Using whoopsie.service worked the first time I rebooted but each reboot after that ssh failed to start.

I tried the following as well:

systemctl edit ssh.service and putting the above variations in that file and the ssh.service still failed to start.

When looking at:

systemctl show ssh or systemctl show ssh |grep -C 10 "After" to get the data I wanted to see.

I see the following "Before=" argument which has multi-user.target included. When you look at what's in /etc/systemd/system/multi-user.target.wants/ it includes links to ssh.service, networkd-dispatcher.service, networking.service & NetworkManager.service.

Names=ssh.service
Requires=sysinit.target system.slice -.mount
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=system.slice systemd-journald.socket network.target sysinit.target -.mount auditd.service basic.target

From the above data it looks like ssh.service is set to start before multi-user.target?

So I set out to change that behavior:

I tried putting the following in systemctl edit ssh.service which did not work

Requires=multi-user.target
Before=shutdown.target
After=multi-user.target
Wants=multi-user.target

Did systemctl revert ssh.service to revert to the original configs

Then I tried systemctl edit --full ssh.service and put

Requires=multi-user.target
Before=shutdown.target
After=multi-user.target
Wants=multi-user.target

reloaded the daemons systemctl daemon-reload. I don't think it totally necessary to do this if you're going to reboot anyway but why not, no harm.

Rebooted multiple times and now ssh starts on every reboot/boot.

Robie Basak (racb)
tags: added: network-online-ordering
Revision history for this message
Andy Igoshin (andy-igoshin) wrote :

/etc/systemd/system/ssh.service.d/override.conf

[Unit]
After=network-online.target auditd.service

Revision history for this message
Jean-Marie Delapierre (diesel83) wrote : Re: [Bug 216847] Re: sshd will not start at boot if ListenAddress is set, because network interface is not yet up

Le 27/02/2021 à 08:39, Andy Igoshin a écrit :
> /etc/systemd/system/ssh.service.d/override.conf
>
> [Unit]
> After=network-online.target auditd.service
>
It has worked very well for me.

Thanks.

Regards

Jean-Marie

Revision history for this message
That Man (thatmantheman1212) wrote :

I researched this issue for about 6+ hours or so and finally came across this thread. I have been able to solve the issue thanks to this thread.

# ENVIRONMENT INFO:
Ubuntu 20.04.2 LTS (Focal Fossa)
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020

# PROBLEM:
 Open /etc/ssh/sshd_config and add: "ListenAddress=192.168.5.123"
 The IP address specified is the local IP of the Ethernet adapter on the motherboard.

 # EXPECTED RESULT: Boot the server, and then be able to login to the specified IP address via SSH from a remote workstation.

 # ACTUAL RESULT: Boot the server, and I am unable to login to the specified IP address via SSH from a remote workstation.

 # NOTES: If I remove "ListenAddress=192.168.5.123" from sshd_config, reboot the server, then I am able to login via SSH from a remote workstation.

  If the server is already on, and I am logged in via SSH, and I add "ListenAddress=192.168.5.123" to sshd_config, and then I run "sudo systemctl restart sshd", and then I run "sudo systemctl status sshd", I see that sshd is listening to the IP address specified with no errors. I am then able to logout/login of the SSH session successfully.

  However, if I add "ListenAddress=192.168.5.123" to sshd_config, reboot the server, then I am unable to log into the server via SSH from a different workstation. When I check /var/log/auth.log I find this error:

   Apr 29 08:09:45 user sshd[1395]: error: Bind to port 12345 on 192.168.5.123 failed: Cannot assign requested address.
   Apr 29 08:09:45 user sshd[1395]: fatal: Cannot bind any address.

  This leads me to believe that sshd is attempting to bind to the Ethernet adapter's IP address BEFORE the Ethernet adapter receives its IP address from the DHCP server (I think). Thus resulting in this fatal error.

## SOLUTION:
 Open /etc/systemd/system/sshd.service and change:

  FROM:

   [Unit]
   After=network.target auditd.service

  TO:

   [Unit]
   After=network-online.target auditd.service

 See this article for details: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/#cutthecraphowdoimakesurethatmyservicestartsafterthenetworkisreallyonline

 With this configuration in place, I add "ListenAddress=192.168.5.123" to sshd_config, reboot the server, and then I am successfully able to login via SSH from a remote workstation.

Revision history for this message
KarlGoetz (kgoetz) wrote :
Revision history for this message
KarlGoetz (kgoetz) wrote :

cjwatson (if you're still watching) or other openssh uploaders - would you accept a patch to adjust ssh service files default behaviour to network-online instead of network ready? If yes I'll prepare a debdiff for review.

Revision history for this message
KarlGoetz (kgoetz) wrote :
Changed in openssh (Debian):
status: Unknown → New
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

It seems like I can't add another Debian bug reference to this bug, but I found this one to be a very interesting read:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982950

In a nutshell, although the "After=network-online.target" change can be considered to be a workaround here, I find it quite unlikely that the Debian maintainer (and therefore Ubuntu) will accept a patch proposing to use this statement in openssh's systemd service file. You can find a rationale by Colin in the bug I mentioned above.

In a way, I think this bug is similar to https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1918141, and we still don't seem to have a good "silver bullet" for these cases...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.