/etc/init.d/ssh seems to work, but actually upstart is used.

[Mark Schouten]

While trying to get x-forwarding on Lucid to work, I edited /etc/default/ssh to add the -4 flag, as offered as a work-around in various bugreports. When trying to restart ssh using /etc/init.d/ssh, all seems to work but my flags around passed to sshd.

It seems that ssh is actually managed by upstart, which is great, but you cannot really tell. For other services, there are symlinks to /lib/init, for ssh there isn't. So I assumed that ssh is still managed using /etc/init.d/

Please clarify the use of upstart (for ssh) for users, so they don't spend hours trying to debug what shouldn't be debugged. :)

== SRU Justification ==

Impact: this bug causes quite a bit of confusion for users, which is particularly troubling considering SSH is often the only way headless systems can be accessed post-installation. The change will avoid the half-in half-out situation that some people are dealing with where they mix /etc/init.d/ssh with using the 'service' command or upstart start/stop/initctl commands.

Dev fix: A check has been added to the included init.d script which checks to see if it is being run in a chroot, and if not, it runs /lib/init/upstart-job in much the same way it is usually run when symlinked from /etc/init.d.

TEST CASE:

1. start a system with openssh-server installed
2. run 'sudo status ssh' to verify that ssh has been started by upstart.
3. run 'sudo stop ssh' to stop the upstart job.
4. run 'sudo /etc/init.d/ssh start'. It should start sshd without upstart.
5. run 'sudo status ssh' to verify that the ssh upstart job is not running
6. run 'sudo /etc/int.d/ssh stop' to stop the incorrectly managed sshd.
7. install new package with patched init.d script
8. repeat steps 2 through 4. now it should display a disclaimer like this:
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service ssh start

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start ssh

Regression Potential:

There may be users who are expecting this script to work outside of chroots. They would have to achieve this by modifying the start on or removing the /etc/init/ssh.conf. The latter case is handled directly in the code by checking for its existence. The former will also be handled on reboot, but not on upgrades. The upgrade will cause the script to be unable to stop any sshd that is running via the old method.

[rswan (robertswan71)]

So how does one start and stop ssh? I used to use the /etc/init.d script to start and stop sshd, which no longer works. If I kill the sshd process a new one starts up.

[Mark Schouten (mark-prevented)]

On Thu, May 06, 2010 at 09:31:06PM -0000, rswan wrote:
> So how does one start and stop ssh? I used to use the /etc/init.d script
> to start and stop sshd, which no longer works. If I kill the sshd
> process a new one starts up.

'stop ssh'
'start ssh'
'restart ssh'

It's rather confusing, this transition period.

--
Mark Schouten <email address hidden>

[Patrick Cervicek (patrick-ubuntu)]

Attached patch allows /etc/init/ssh.conf to use also the SSHD_OPTS from /etc/default/ssh. So it doesn't matter how sshd is started (but it should be clarified for the future!)

[Colin Watson (cjwatson)]

The Upstart maintainer explicitly asked me not to do this.

[Patrick Cervicek (patrick-ubuntu)]

off-topic:
Did the upstart maintainer say how configuration is done in upstart? When I look to /etc/init, there are also other scripts using /etc/default

$ grep -rl "/etc/default/" /etc/init
/etc/init/apport.conf
/etc/init/gssd.conf
/etc/init/idmapd.conf
/etc/init/rpc_pipefs.conf
/etc/init/hwclock.conf
/etc/init/mountall.conf
/etc/init/portmap.conf
/etc/init/statd.conf
/etc/init/hwclock-save.conf
/etc/init/gdm.conf
$

[Colin Watson (cjwatson)]

Scott's recommendation, in general, is that Upstart jobs should be
simple enough that it's reasonable to edit them directly.

/etc/default/ was introduced in the first place because /etc/init.d/
scripts were hard to edit and it was even harder to keep changes merged.
One goal of Upstart is that this should not be the case for Upstart
jobs.

On Thu, Jul 01, 2010 at 11:38:07AM -0000, Patrick Cervicek wrote:
> $ grep -rl "/etc/default/" /etc/init

You have some false positives here - a number of these are reading
system configuration files (/etc/default/locale, /etc/default/rcS)
rather than files specific to that service. My comments only apply to
the latter.

[Thierry Carrez (ttx)]

Linked to bug 617515

[Raphaël Pinson (raphink)]

The state in lucid is really problematic. Afaik, the init.d scripts are supposed to provide wrappers for upstart in the transition period, and we are using them in a lot of old scripts to manage our fleet. Reloading ssh with "/etc/init.d/ssh reload" resulted in a weird state where upstart believes ssh is stopped, while it's still up but upstart has no way to control it until the service is stopped from the init.d script.

Additionally, doing a kill -1 on sshd leads to the same bug: upstart thinks sshd is stopped when it's really reloaded.

Is there a fix planned in -updates for lucid? What is planned for future releases?

[Mark Schouten (mark-prevented)]

This upstart-'upgrade' (along with plymouth implementation) is one of the points where Canonical chose to neglect the LTS part and implement unfinished beta stuff in a version of Ubuntu on which everyone should be able to rely.

Unfortunatly, they are now busy dropping gnome, X and god knows what. So I'm afraid this will be a long running bug...

[Colin Watson (cjwatson)]

Mark, I understand you're upset, but your comment is quite unhelpful
really, and the state of GNOME/gnome-shell/Unity and X/Wayland is
entirely irrelevant to Upstart and OpenSSH. Apart from anything else
the people doing the latter have nothing, zero, zilch, nada to do with
the former. If you're upset with Canonical, I'm not sure there's any
point in taking that up with me.

I do plan to figure out some better scripting for this. The real fix
needs Upstart to be able to supervise chroots, which is hard, but it
should be possible to detect the situation in the init.d script and
arrange for some kind of sensible fallback behaviour. If I find time
for that then I will certainly backport the fix to lucid.

It's not accurate to say that the intention of the init.d script was to
provide a wrapper for Upstart in the transition period. The wrapper is
'service', as in 'service ssh start' etc. Normally, the init.d scripts
go away on transitioning to Upstart, but in this case I kept it around
because there was no other way to deal with sshd in a chroot which I
knew to be a common situation. I'm sorry I didn't quite manage to do a
complete job of it, but I really wasn't expecting it to be used as a
wrapper for Upstart when I made those changes.

Regarding what happens when you send SIGHUP to ssh, I think that's
really a separate bug from this one, and it doesn't much help to combine
them. The plan in Upstart for this is to use the proc connector rather
than the short-lived ptrace hack to supervise processes changing pids,
at which point it will be able to notice that sshd now has a new pid and
smoothly switch over to supervising it. The only connection to this bug
is that it might make sense for the init.d script to notice the
situation and do a full restart rather than sending SIGHUP.

[Raphaël Pinson (raphink)]

Thank you for your answer Colin. I reported the SIGHUP part in this bug because this seems to be what the traditional init script does when doing a reload of ssh, and this is the reason why such an operation fails to combine with Upstart.

[Clint Byrum (clint-fewbar)]

Note that this also causes libc6 upgrades to not restart the ssh service. See more info in the duplicate report bug #709468.

Submitting merge proposal which causes the init.d script to run /lib/init/upstart-job when it is not running in a chroot.

[Jamie Jamison (jamie-jamison)]

OK, if /etc/init/ssh doesn't fully replicate the functionality of the /etc/init.d/ssh script to include properly setting up chrooted environments and properly handling libc6 updates then what benefit does it provide users? Why, if you can't get Upstart to provide the same functionality as the /etc/init.d startup script, move control of sshd over to Upstart anyways? Again, what does it do for the users?

At least the fix for this bit of Upstart badness is easy enough, just remove /etc/init/ssh and start sshd with the script in init.d. Perhaps it's not as architecturally elegant as Upstart is supposed to be, but at least we all know that it works.

[Colin Watson (cjwatson)]

Jamie, making such a change at this point would break other parts of the distribution which now rely on being able to say 'start on started ssh' in Upstart jobs.

[ingo (ingo-steiner)]

> At least the fix for this bit of Upstart badness is easy enough, just remove /etc/init/ssh and start sshd with the script in init.d. Perhaps it's not as architecturally elegant as Upstart ...

Perhaps Ubuntu maintainers are not aware of the importance of sshd in general? sshd is vital on remote machines/servers which are running without screen and keyboard. ssh is the (almost) only access and that has to be reliable! It has to be up as the first daemon as soon as networking is configured and has to be the last service to be stopped on shut down - and it has to restart itself in case of unforseen interruptions.

Moving such a service to upstart at the moment is lightheaded and disqualifies Ubuntu for servers.

[Clint Byrum (clint-fewbar)]

ingo, its precisely because we understand that sshd is so important that we've moved it to upstart, which should give it the ability to respawn automatically and be started earlier in the boot.

That much has been tested and it works. This bug is only concerning an oversight in understanding of how people used /etc/init.d/ssh ... the fix is minimal and should easily be backportable to prior versions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:5.8p1-1ubuntu2

---------------
openssh (1:5.8p1-1ubuntu2) natty; urgency=low

  [ Clint Byrum ]
  * debian/openssh-server.ssh.init: Adding upstart awareness that will
    call /lib/init/upstart-job when script is run outside of a chroot.
    While this fixes LP: #531912, the change should be reverted when
    upstart gains chroot session support.

  [ Colin Watson ]
  * Only do the above if /etc/init/ssh.conf still exists, since apparently
    some people have been removing it.
 -- Colin Watson <email address hidden> Thu, 03 Mar 2011 16:18:24 +0000

[Martin Pitt (pitti)]

Accepted openssh into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Accepted openssh into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Rolf Leggewie (r0lf)]

Thanks to everyone involved in getting this fixed. I'll say that I still don't have much confidence in all of this, but judging by the test case, it seems that at least for my lucid system there is no longer a discrepancy between the status managed with init.d and upstart. Please release to -updates.

As a user I'm thankful for all the work the devs do. I think it's quite obvious from what Ubuntu has achieved that everybody involved is really trying their best and it mostly works extremely well. But I will not hide the fact that I also feel that upstart was very hastily introduced WAY before it's time. Dapper vservers are still stuck without ANY upgrade path AT ALL because of upstart. I feel this is quite outrageous given the promise of an LTS. There isn't much live left for dapper users and obviously, I'd much rather run lucid by now. I hope bug 430224 will soon get fixed for lucid.

So, thanks again to the devs for their work. Sometimes you can't win, I guess ;-)

[Rolf Leggewie (r0lf)]

got a bit mixed up with the versions. It is hardy vservers that currently have no update path to anywhere., intrepid is already EOL and an update to lucid is impossible because of bug 430224.

[Clint Byrum (clint-fewbar)]

Excerpts from Rolf Leggewie's message of Sat Jul 16 14:04:17 UTC 2011:
> got a bit mixed up with the versions. It is hardy vservers that
> currently have no update path to anywhere., intrepid is already EOL and
> an update to lucid is impossible because of bug 430224.

Rolf, I see your point, and I think we can offer a "less than perfect"
solution.

I've created a daily build PPA for the latest version of Upstart in
Ubuntu, which will build an upstart with chroot support for lucid and
maverick.

I'll just host it under my own PPA for now, but eventually we should
put this under the ubuntu-server-edgers team stewardship. Once 12.04 is
released, we can just freeze this PPA and that would cover lucid upgrades
for its lifecycle.

The PPA is

ppa:clint-fewbar/upstart

It should build upstart in the next 20 minutes or so.

Maybe you can give it a try on lucid and if it seems to work and provide
the chroot support you're looking for, we can add it to the edgers setup
and for users who have extensive chroot needs, this will work.

I already talked with the backports maintainer, Scott Kitterman, and he
felt that upstart is too core to the system to provide in the official
backports.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:5.3p1-3ubuntu7

---------------
openssh (1:5.3p1-3ubuntu7) lucid-proposed; urgency=low

  [ Clint Byrum ]
  * debian/openssh-server.ssh.init: Adding upstart awareness that will
    call /lib/init/upstart-job when script is run outside of a chroot.
    While this fixes LP: #531912, the change should be reverted when
    upstart gains chroot session support.

  [ Colin Watson ]
  * Only do the above if /etc/init/ssh.conf still exists, since apparently
    some people have been removing it.
 -- Clint Byrum <email address hidden> Fri, 10 Jun 2011 22:46:07 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:5.5p1-4ubuntu6

---------------
openssh (1:5.5p1-4ubuntu6) maverick-proposed; urgency=low

  [ Clint Byrum ]
  * debian/openssh-server.ssh.init: Adding upstart awareness that will
    call /lib/init/upstart-job when script is run outside of a chroot.
    While this fixes LP: #531912, the change should be reverted when
    upstart gains chroot session support.

  [ Colin Watson ]
  * Only do the above if /etc/init/ssh.conf still exists, since apparently
    some people have been removing it.
 -- Clint Byrum <email address hidden> Sat, 11 Jun 2011 08:06:02 -0700