"oom" change in 1:5.3p1-3ubuntu5 causes "operation not permitted"

[Alan Porter]

WHAT RECENTLY CHANGED

Recently, a security update was pushed out for the openssh-server package.

The package changes:
    -openssh-client 1:5.3p1-3ubuntu4
    -openssh-server 1:5.3p1-3ubuntu4
    -openssl 0.9.8k-7ubuntu8.4
    +openssh-client 1:5.3p1-3ubuntu5
    +openssh-server 1:5.3p1-3ubuntu5
    +openssl 0.9.8k-7ubuntu8.5

The upgrade makes a change to the /etc/init/ssh.conf file:

$ diff before/etc/init/ssh.conf after/etc/init/ssh.conf
10d9
< expect fork
15c14
< #oom never
---
> oom never
27c26
< exec /usr/sbin/sshd
---
> exec /usr/sbin/sshd -D

THE PROBLEM

I have a virtual machine at Tektonic.net. This service is a virtuozzo VM. After upgrading to the new 1:5.3p1-3ubuntu5 package, I could no longer SSH into the VM. I rebooted the machine, and SSH never allowed a connection ("connection refused").

I found this in my /var/log/syslog. The timestamp corresponds to when I did the upgrade (and I forget whether I manually did a "service ssh restart").

Jan 23 16:04:23 satu init: ssh main process (32282) terminated with status 255
Jan 23 16:04:23 satu init: Failed to spawn ssh pre-start process: unable to set oom adjustment: Operation not
permitted

WORK-AROUND

I booted the VM in "recovery mode", which allows me to directly modify the files on the VM's disk image. I reverted the /etc/init/ssh.conf to the way it was in version 1:5.3p1-3ubuntu4 (removing the "-D" and the "oom never" and adding back the "expect fork"). When I rebooted, the machine came up normally and I was able to SSH in again.

SYSTEM INFORMATION

I know that Virtuozzo machines are a little different than normal machines... they are more like a "chroot jail" than a normal machine. And I am not sure if those differences are what caused SSH to not respond. But I have installed the same upgrade on native machines and on Xen VM's with no problems.

If you need more information about this Virtuozzo VM, I am happy to provide details.

$ lsb_release -rd
Description: Ubuntu 10.04.1 LTS
Release: 10.04

$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:5.3p1-3ubuntu5
  Candidate: 1:5.3p1-3ubuntu5
  Version table:
 *** 1:5.3p1-3ubuntu5 0
        500 http://archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
        100 /var/lib/dpkg/status
     1:5.3p1-3ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ lucid/main Packages

[Alan Porter (alan.porter)]

Looking a little deeper, it looks like OpenSSH has had this "oom never" line ever since Ubuntu 10.04 came out, and that Virtuozzo containers have had to modify this file all along. I never saw it before because my Virtuozzo provider had a pristine 10.04 image that they deployed -- I assume, with the "oom never" line commented out. So this was the first time that the package had been upgraded on my system, and it asked me if I wanted to "keep my changes" or "install the package maintainer's version".

So maybe this is not so much of a bug in OpenSSH as it is a quirk in Virtuozzo (that I did not know about at the time of upgrade). Not knowing what this "oom never" option is all about, I can not make that call.

Alan Porter

[Colin Watson (cjwatson)]

"Opinion" isn't the right status here.

Alan, firstly (and tangentially, I suppose), the openssh update was a normal update, not a security update. That aside, we didn't change the "oom" line in that update, only the "expect" and "exec" lines. Here's the full diff:

  http://launchpadlibrarian.net/61846798/openssh_1%3A5.3p1-3ubuntu4_1%3A5.3p1-3ubuntu5.diff.gz

So I think what happened here is that you got dpkg's conffile prompt, and answered "install the package maintainer's version" without correctly resolving the local change that was present on your system (i.e. commenting out the "oom" line).

dpkg conffile prompts have never been particularly elegant, and at some point I think we would like to add a more convenient three-way merge facility to them (some prompts already have this due to ucf, but the package has to arrange for this manually and it makes things more complicated). This isn't really a bug in the openssh package, though.

As for the oom breakage in general, newer upstream versions of openssh do this rather more gracefully without requiring special hacks for particular container systems, so I think once you move your VM to Ubuntu 10.10 (or 12.04 LTS) or newer, this should no longer be a problem.

[Alan Porter (alan.porter)]

Agreed... this looks more like "pilot error" than anything else.

The update caught be by surprise. I should have known better, though... if my config file had not been modified by me or my Virtuozzo provider, then I would not have been asked whether or not I wanted the new change.

In either case, it's nice to have this unexpected (but perfectly legitimate) behavior documented here, so that others who are having similar issues will find this report when they do a search.

I am OK with closing this bug report. Thanks for taking the time to read and explain.

Alan