sshd will not start at boot if ListenAddress is set, because network interface is not yet up

Bug #216847 reported by sektor on 2008-04-13
176
This bug affects 18 people
Affects Status Importance Assigned to Milestone
portable OpenSSH
Unknown
Unknown
openssh (Ubuntu)
Low
Unassigned
Nominated for Natty by KBios

Bug Description

Binary package hint: openssh-server

The sshd will not start at boot if the ListenAddress option in /etc/ssh/sshd_config is set to an IPv4 address other then 0.0.0.0 .

I am using Ubuntu 7.10 and the version 1:4.6p1-5ubuntu0.2 of the openssh-server package.
I would expect that sshd is started after boot but it will not and I found this in /var/log/auth.log:

sshd[4527]: error: Bind to port 22 on 10.1.1.22 failed: Cannot assign requested address.
sshd[4527]: fatal: Cannot bind any address.

Once the System is started you can start/stop the sshd with the /etc/init.d/ssh script without any problems.

  • unnamed Edit (189 bytes, application/pgp-signature; name="signature.asc")

On Sun, Apr 13, 2008 at 04:23:15PM -0000, sektor wrote:
> The sshd will not start at boot if the ListenAddress option in
> /etc/ssh/sshd_config is set to an IPv4 address other then 0.0.0.0 .
>
> I am using Ubuntu 7.10 and the version 1:4.6p1-5ubuntu0.2 of the openssh-server package.

Are you using Ubuntu Desktop or Ubuntu Server ?

> I would expect that sshd is started after boot but it will not and I found this in /var/log/auth.log:
>
> sshd[4527]: error: Bind to port 22 on 10.1.1.22 failed: Cannot assign requested address.
> sshd[4527]: fatal: Cannot bind any address.

Are you using NetworkManager to get the 10.1.1.22 ip address ? What is
you network configuration (wired, wireless) ?

  status incomplete

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

Changed in openssh:
status: New → Incomplete

Am Montag, den 14.04.2008, 15:19 +0000 schrieb Mathias Gug:
> Are you using Ubuntu Desktop or Ubuntu Server ?

I use the Ubuntu Desktop Edition.

> Are you using NetworkManager to get the 10.1.1.22 ip address ? What is
> you network configuration (wired, wireless) ?
>

I use the Network Manager (nm-applet 0.6.5) for my wired conecction. The
IP address is assigned over dhcp at boot time. There is no iptables
setup on my machine. The complete network configuration is Ubuntu's
installation default. The distribution is up to date.

I was able to reproduce this issue on my laptop with the same software enviroment
but over a wireless link. The wireless Interface uses the bcm43xx module.
The wireless card is a build in Broadcom BCM4306 (rev 03).

I think this is not releated to my hardware because the same thing happens
on two completely different computers.

Regards,
Aleks

Changed in openssh:
status: Incomplete → New
Colin Watson (cjwatson) wrote :

network-manager is not guaranteed to actually bring the interface up until rather later on. Until it does, you won't be able to start sshd with that configuration.

One of these days, we may be able to fix this unexpressed dependency using Upstart.

Changed in openssh:
importance: Undecided → Low
status: New → Confirmed
sektor (sektor-1) on 2009-11-09
visibility: public → private
Colin Watson (cjwatson) on 2010-05-09
summary: - sshd will not start at boot
+ sshd will not start at boot if ListenAddress is set, because network
+ interface is not yet up
visibility: private → public
Greg Porterfield (gregport) wrote :

I'm having this exact symptom but in a different environment: Ubuntu 10.04 Lucid Server.

After configuring /etc/ssh/sshd_config to have a specific ListenAddress instead of 0.0.0.0, the ssh daemon consistently fails to start at boot time. Errors from /var/log/auth.log:

Sep 23 14:26:45 a02-gsp sshd[697]: error: Bind to port 22 on 10.100.50.19 failed: Cannot assign requested address.
Sep 23 14:26:45 a02-gsp sshd[697]: fatal: Cannot bind any address.

And errors in /var/log/syslog immediately after boot:

Sep 23 14:26:45 a02-gsp init: ssh main process (739) terminated with status 255
Sep 23 14:26:45 a02-gsp init: ssh main process ended, respawning
Sep 23 14:26:45 a02-gsp init: ssh main process (744) terminated with status 255
Sep 23 14:26:45 a02-gsp init: ssh respawning too fast, stopped

So this appears to be the same problem, but the root cause would seem to be upstart in this case. Since /etc/init/ssh.conf says "start on filesystem", sshd is started before the network interfaces are configured and fails to bind to the specific address that isn't configured yet.

I've implemented the following workaround which seems to be working well: changed the start event in ssh.conf to "start on started network-interface" and added a "sleep 3" in the pre-start script (see attached patch). Without the sleep to wait for the interface to be configured, sshd would still fail to start some of the time.

If this would be more appropriate as a new bug specific to Lucid Server I'll be happy to open a new one.

tags: added: patch
KBios (kbios) wrote :

Adding random sleeps is usually not the best possible solution, as the required delay varies between systems (for example, when using a network bridge). Maybe depending on net-device-up would be better.

ilf (ilf) wrote :

Three years and six releases later, this is still present :(

Claus Allweil (clal) wrote :

I have fixed this problem for me in /etc/network/interfaces

auto eth0
iface eth0 inet dhcp
  up service ssh start

Dirk Schuster (dirk-schuster) wrote :

I tested on 12.10.

Unfortunately the solution from Claus interferes with the network manager. On my 12.10 system I wasn't able to handle the eth0 by the network manager any more.

As I don't like arvitrary sleep times, I figured out a solution just changing the start on line in /etc/init/ssh. It works very well for me in 12.10..

I add this patch in a moment. It would be fine if someone would test it and confirm it.

Users of other ubuntu versions might have to change the line accordingly.

Dirk Schuster (dirk-schuster) wrote :

same problem with package openssh-server of Ubuntu 13.10 (Saucy)

error messages in /var/log/auth.log after boot process:
Nov 1 17:31:46 pc sshd[1209]: error: Bind to port xxx on yyy.yyy.y.y failed: Cannot assign requested address.
Nov 1 17:31:46 pc sshd[1209]: fatal: Cannot bind any address.

reason:
/etc/ssh/sshd_config uses ListenAddress option

solution:
in /etc/init/ssh.conf
change line
start on runlevel [2345]
to
start on runlevel [2345] and net-device-up IFACE!=lo

message in /var/log/auth.log after boot process now:
Nov 1 17:39:20 pc sshd[1518]: Server listening on yyy.yyy.y.y port xxx.

What needs to be done, so that the solution will get into the package? Could anybody help?

Dirk Schuster (dirk-schuster) wrote :

same in vivid

Mai 11 12:40:44 pcds systemd[1]: Unit ssh.service entered failed state.
Mai 11 12:40:44 pcds systemd[1]: ssh.service failed.
Mai 11 12:40:44 pcds systemd[1]: ssh.service holdoff time over, scheduling ...t.
Mai 11 12:40:44 pcds systemd[1]: start request repeated too quickly for ssh...ce
Mai 11 12:40:44 pcds systemd[1]: Failed to start OpenBSD Secure Shell server.
Mai 11 12:40:44 pcds systemd[1]: Unit ssh.service entered failed state.
Mai 11 12:40:44 pcds systemd[1]: ssh.service failed.

For the first time this error was reported 7 years ago. And it lives on and on and on ...

Wouldn't it be possible to have sshd use "IP_FREEBIND" so that it can
bind an IP that has not materialized yet?

Seth Arnold (seth-arnold) wrote :

Changing the ssh service file to use network-online.target should also work; see http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ for more information.

tags: added: systemd-boot
Martin Pitt (pitti) on 2015-05-19
tags: removed: systemd-boot
karl forner (karl-forner) wrote :

Would be nice if it was fixed once for all. This is a critical bug when your server is in a distant location, and you cannot log on it because of that bug.

Since so many things changed since this - admittedly embarrassing old - bug was reported I thought I retest it before the next LTS comes out.

I looked at the Xenial content and found stuff like:
Not only is it specified "After network.target", but on top there is /etc/network/if-up.d/openssh-server which is intended to restart openssh if later an iface comes up, there it would be able to rebind that IP.

I tested on containers first and it worked right away, setting ListenAddress and restarting.
But well in containers devices might be "up" at a different timing, but even in a KVM Guest, setting ListenAddress again it works just fine.
Mar 24 14:54:43 testxenial-ssh sshd[1293]: Server listening on 192.168.122.131 port 22.

I wondered and wanted to make sure that the test is valid, so I spawned
trusty (LTS), vivid (last reported issue) and wily

But - for me - it was working on all of them - even with several retries it was not even racy.

Therefore I wanted to ask for a retest - at least with Xenial if possible.
Also any detail about the test environment that was used would be great - especially by the last confirmation in vivid.
Does it have to be a special network card that might initialize slowly or anything like it?

Simon Déziel (sdeziel) wrote :

Under systemd, if the ListenAddress is on an interface that is manually brought up, the ifup script doesn't help. In that situation, the invoke-rc.d reload/restart fails because the initial startup of sshd wasn't successful.

ironstorm (ironstorm-gmail) wrote :

This is still broken in 16.04 / Xenial.

I'm going to try using a root cronjob to catch and recover:

*/5 * * * * if [ $(ps -ef | grep /usr/sbin/sshd | grep -v grep | wc -l) -eq 0 ]; then /usr/sbin/service ssh start; fi

This has been broken for almost 10 years, still broken in 16.10

It is useful to set sshd ListenAddress so ssh won't be available when connected to a different network (laptop on a public wifi etc). However with ListenAddress set, sshd always fails a rebooot/network disconnect.

Owlbsidian (owlbsidian) wrote :

For me after changing network.target to network.service on /usr/lib/systemd/system/sshd.service it started at the right time.

As Seth mentioned in comment #13 likely network-online.target should be the solution.
@Owlbsidian - could you test and confirm with that as a fix as well?

A related fun quote from the systemd page about it:
"... please just fix your program to be friendly to dynamically changing network configuration. That way you will make your users happy because things just start to work, and you will get fewer bug reports ..."

Without that support you can still run into issues if e.g. you have a manually added interface you depend on (no service target will help you there). But that in turn would be an upstream feature request more than anything else.

Anyway, as long as ssh doesn't support that we might want to finally tackle that issue to at least improve it. Now that systemd has more means to express "online".
Maybe even by directly fixing it in Debian.

Subscribing server to track and cjwatson for his opinion (actually I see he is auto-subscribed to the pkg so he will see).

Gennady (gendelflg) wrote :

Ubuntu 17.10
Changing network.target to network-online.target in /lib/systemd/system/ssh.service worked. I also added Wants=network-online.target (don't know if it's necessary but https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ suggests it)

Thanks for the confirmation Gennady.

Now that there is a suggested and confirmed solution, would anybody that personally requires (alway better to make an argument) this fix mind filing this in Debian as well to be changed in ./debian/systemd/ssh.service.
I'd think at least eventually we want to be in sync on this behavior.

@CJWatson - since you usually do the openssh uploads what do you think about adding that for now on your next upload?

Andreas Hasenack (ahasenack) wrote :

Upstream server about using IP_FREEBIND: https://bugzilla.mindrot.org/show_bug.cgi?id=2512

The Marauder (arn-epsilon) wrote :

10 years ago ... Same bug !

Incredible :(

Rodman (rodman-c) wrote :

This is still broken in 18.04 Server. Setting the listen address fails to load the SSHD daemon at boot time. You have to have console access and do a "service sshd restart" to fix.

How is this bug still here???

Simon Déziel (sdeziel) wrote :

@Rodman, as a workaround, maybe you could try to add an "After=systemd-networkd-wait-online.service" clause in a drop-in snippet?

Andreas Hasenack (ahasenack) wrote :

Just passing by to see if anything has changed. Would be good to know if the workaround is effective for those affected.

Con (c0ntr1but3) wrote :

@ahasenack, I found this post (took plenty of Google foo) but it was helpful.

For anyone else, 18.04 LTS STILL has this issue. The following fixed it:

After=network-online.target
Wants=network-online.target

These are in my [Unit] section at the top of /etc/systemd/system/sshd.service

<code_paste_from_service_file>
[Unit]
Description=OpenBSD Secure Shell server
After=network-online.target auditd.service
Wants=network-online.target
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
</code_paste_from_service_file>

I hope this is helpful to others!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.