Upgrade of resolvconf from 1.63ubuntu* to 1.63ubuntu7 fails: start: Job failed to start

["Kosmonaut" Bernd Müller]

Sorry I cannot provide any further information. I just gt this bug after loging into ubuntu-desktop

ProblemType: Package
DistroRelease: Ubuntu 12.04
Package: resolvconf 1.63ubuntu7
ProcVersionSignature: Ubuntu 3.2.0-14.23-generic 3.2.3
Uname: Linux 3.2.0-14-generic x86_64
ApportVersion: 1.91-0ubuntu1
Architecture: amd64
Date: Tue Feb 7 11:28:13 2012
ErrorMessage: Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Alpha amd64 (20110705.1)
PackageArchitecture: all
SourcePackage: resolvconf
Title: package resolvconf 1.63ubuntu7 failed to install/upgrade: Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
UpgradeStatus: Upgraded to precise on 2012-01-19 (21 days ago)

[Thomas Hood (jdthood)]

The tail of DpkgTerminalLog:

resolvconf (1.63ubuntu7) wird eingerichtet ...
Neue Version der Konfigurationsdatei /etc/resolvconf/update.d/libc wird installiert ...
Neue Version der Konfigurationsdatei /etc/dhcp/dhclient-enter-hooks.d/resolvconf wird installiert ...
start: Job failed to start
invoke-rc.d: initscript resolvconf, action "start" failed.
dpkg: Fehler beim Bearbeiten von resolvconf (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück

[Stéphane Graber (stgraber)]

Was that an upgrade from 11.10 to 12.04 or an update of 12.04?
If it was an upgrade from 11.10, how did you do that upgrade? Using the update-manager or manually?

Either way, can you attach /var/log/dpkg.log to the bug report? hopefully this will help find what went wrong.

Thanks

[Thomas Hood (jdthood)]

The resolvconf runtime directories are created in the resolvconf package preinst and at boot, "by" the resolvconf upstart job.

    start on mounted MOUNTPOINT=/run
    pre-start script
        mkdir -p /run/resolvconf/interface
        touch /run/resolvconf/postponed-update
        resolvconf --enable-updates
    end script

The resolvconf package postinst sets up the /etc/resolv.conf symlink then enables, triggers and finally performs an update.

As far as I can see there is no need for the postinst to run the upstart job. I don't see why it needs to do "invoke-rc.d resolvconf start". The Debian version of the package does not do "invoke-rc.d resolvconf start" and the Debian version of the package seems to work properly.

Thus I don't understand this changelog item (1.63ubuntu2):

  * Correct wrong use of dh_installinit --upstart-only, which is only for
    upstart jobs that were never init scripts before; and don't pass
    --noscripts, because we really do want the upstart job run on install.

or (1.63ubuntu3):

  * Move the #DEBHELPER# token in debian/postinst to after the resolv.conf
    symlink is set, so the init script can actually start (since it expects
    /etc/resolv.conf to be a symlink).

Can someone explain to me why the initscript, which is a symlink to /lib/init/upstart-job, has to be run in the postinst?

If it turns out that the "invoke-rc.d resolvconf start" isn't needed then it should be removed, and once it's removed it won't fail.

But the question remains: Why does it fail?

It doesn't fail when I run it from the command line:

# invoke-rc.d resolvconf start ; echo retval $?
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service resolvconf start

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start resolvconf
resolvconf start/running
retval 0

[Thomas Hood (jdthood)]

Observation. In both this (#929552) case and in the duplicate (#929641) the system was upgraded from an earlier 1.63ubuntu* version of resolvconf. In this case it was upgraded from 1.63ubuntu3 to 1.63ubuntu7; in the dup it was upgraded from 1.63ubuntu3 to 1.63ubuntu6 and then to 1.63ubuntu7.

[Steve Langasek (vorlon)]

On Fri, Feb 10, 2012 at 07:27:10PM -0000, Thomas Hood wrote:
> The resolvconf package postinst sets up the /etc/resolv.conf symlink
> then enables, triggers and finally performs an update.

> As far as I can see there is no need for the postinst to run the upstart
> job. I don't see why it needs to do "invoke-rc.d resolvconf start".
> The Debian version of the package does not do "invoke-rc.d resolvconf
> start" and the Debian version of the package seems to work properly.

The right way to do this is by calling the init script in the maintainer
script, not by duplicating code between the init script and the maintainer
script.

*Not* starting the upstart job on install also means that stopping is not
idempotent wrt reboots after install. If the job is not running, it cannot
be stopped, so "resolvconf --disable-updates" will not be called when it
should be on shutdown.

We need to find out why the upstart job is failing and fix that, not avoid
calling the upstart job on install/upgrade.

[Thomas Hood (jdthood)]

2012/2/12 Steve Langasek <email address hidden>

> *Not* starting the upstart job on install also means that stopping is not

idempotent wrt reboots after install. If the job is not running, it cannot
> be stopped, so "resolvconf --disable-updates" will not be called when it
> should be on shutdown.
>

OK, thanks for the explanation.

(As I said earlier, the invoke-rc.d duplicates other code in the postinst.
 Since the invoke-rc.d has to stay, the other code should be removed. I'll
file a new bug report about that.)

Yes, we need to find out why the invoke-rc.d call fails. Any ideas?
--
Thomas

[Thomas Hood (jdthood)]

I wrote in comment #5:
> Observation. In both this (#929552) case and in the duplicate (#929641)
> the system was upgraded from an earlier 1.63ubuntu* version of resolvconf.

I read in initctl(8):
> COMMANDS
> start
> [...]
> IF the job is already running, start will return an error.

Resolvconf.postinst runs

    invoke-rc.d resolvconf start

which does

    /etc/init.d/resolvconf start

which, because /etc/init.d/resolvconf is a symlink to /lib/init/upstart-job, does

    start resolvconf

Now, at the time of an upgrade to the resolvconf package from any earlier version of the package that started an Upstart job, the resolvconf Upstart job will be considered by Upstart already to be running.

But since 1.63ubuntu5 the job is not stopped on upgrade.

    resolvconf (1.63ubuntu5) precise; urgency=low
      [ Steve Langasek ]
      * Call dh_installinit with -r, to not stop/start resolvconf on upgrade;
        that just leaves a window when updates aren't happening, which
        doesn't benefit us.

Hence the failure.

One solution would be to go back to dh_installinit without the "-r" option.

Suspending updates between resolvconf.old-prerm and resolvconf.new-postinst does offer the benefit that if multiple packages induce resolvconf updates during the dpkg run then these will all be processed in one go. That's nice.

I don't immediately see any problem with delaying updates until resolvconf.postinst runs, but perhaps someone else can. Steve Langasek: Did you add "-r" to "dh_installinit" in order to solve some problem?
--
Thomas

[Thomas Hood (jdthood)]

The following is a copy of Sebastian Unger (sebunger44)'s comment (#7) from #922491 which I think is another report arising from this (#929552) issue.
-------------------------------

I'm not sure whether this is related to this or not. I was about to raise a bug for it, but thought I'd comment here in case it is related to something someone did recently in regards to this bug. Here's my scenario:

I'm using oneiric-mini.iso to install from a local ubuntu mirror (more like a cache) that is constructed using reprepro. In essence it's a mirror of only the important bits of oneiric to set up an ubuntu-standard server + resolvconf from precise since the one in oneiric is totally broken.

Until very recently (1.63ubuntu3) this installed just fine (using DHCP for the single interface and getting the DNS info from that). Now I've just updated my "mirror" to the latest packages which pulled in resolvconf 1.63ubuntu6 from precise and now this no longer installs properly. In particular /etc/resolv.conf is still a file and not a sym-link after a clean expert install.

If I subsequently try to fix thinks by running dpkg-reconfigure resolvconf, it goes through a couple questions (Yes, I want to prepare /etc/resolv.conf and no I don't want to append the original contents) it fails saying (in effect) that /run/resolvconf does not exist. If I create /run/resolvconf and /run/resolvconf/interface manually and then run through dpkg-reconfigure it appears to start coming right.

I don't have the means to try this on a plain precise install at the moment.

Anybody any comments?

Seb

[Thomas Hood (jdthood)]

Re: comment #9. I copied Seb's comment here but I now think that the main problem must be that he was running 1.63ubuntu6 which failed to linkify properly. Second there was a mysterious failure of resolvconf.preinst to create /run/resolvconf, as in #927300. I have asked Seb to try 1.63ubuntu7 and report what problems remain with that.

[Thomas Hood (jdthood)]

Re: comment ##9, 10. Sebastian Unger reports that his problem was solved by 1.63ubuntu7.

[Steve Langasek (vorlon)]

On Mon, Feb 13, 2012 at 10:42:00AM -0000, Thomas Hood wrote:
> I read in initctl(8):
> > COMMANDS
> > start
> > [...]
> > IF the job is already running, start will return an error.

> Resolvconf.postinst runs

> invoke-rc.d resolvconf start

> which does

> /etc/init.d/resolvconf start

> which, because /etc/init.d/resolvconf is a symlink to /lib/init/upstart-
> job, does

> start resolvconf

> Now, at the time of an upgrade to the resolvconf package from any
> earlier version of the package that started an Upstart job, the
> resolvconf Upstart job will be considered by Upstart already to be
> running.

> But since 1.63ubuntu5 the job is not stopped on upgrade.
>
> resolvconf (1.63ubuntu5) precise; urgency=low
> [ Steve Langasek ]
> * Call dh_installinit with -r, to not stop/start resolvconf on upgrade;
> that just leaves a window when updates aren't happening, which
> doesn't benefit us.

> Hence the failure.

$ status resolvconf
resolvconf start/running
$ sudo /etc/init.d/resolvconf start
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service resolvconf start

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start resolvconf
$ echo $?
0
$

Yes, the *start* command will exit non-zero if you call it when the job is
already running. But the invoke-rc.d command will not; the upstart init
script wrapper fully implements Debian policy's requirements for init
scripts, including the requirement to not error out when starting an
already-started job.

The error message 'start: Job failed to start' indicates that the job is
stopped and cannot be started. That's not due to the dh_installinit usage.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Steve Langasek (vorlon)]

Bernd, asbenicio, could you install a test version of the upstart package as described at <http://www.theorangenotebook.com/2012/02/call-for-testing-upstart-14.html> and see if you get any useful information about this failure output in /var/log/upstart/resolvconf.log?

[Thomas Hood (jdthood)]

> Yes, the *start* command will exit non-zero if you call it when the job
> is already running. But the invoke-rc.d command will not

Darn, thought I'd found the critter. :)
--
Thomas

[Thomas Hood (jdthood)]

I just tried to reproduce this bug by installing Oneiric Server on a virtual machine, upgrading its resolvconf to 1.63ubuntu3 and then apt-get upgrading the system to Precise. No problem. Then I tried some other upgrade paths from Oneiric to Precise. No problem.

[Thomas Hood (jdthood)]

Question: Why is it that after I boot the system with resolvconf correctly installed the resolvconf Upstart job has status "stop/waiting"

     # status resolvconf
     resolvconf stop/waiting

but if I start from the command line its status becomes "start/running"?

    # start resolvconf
    resolvconf start/running
    # status resolvconf
    resolvconf start/running

What's the relevant difference, in other words, between the start command and "start on mounted MOUNTPOINT=/run"?
--
Thomas

[Thomas Hood (jdthood)]

Regarding my question, see also https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/933566/comments/1

[Steve Langasek (vorlon)]

On Thu, Feb 16, 2012 at 03:13:14PM -0000, Thomas Hood wrote:
> Question: Why is it that after I boot the system with resolvconf
> correctly installed the resolvconf Upstart job has status "stop/waiting"

> # status resolvconf
> resolvconf stop/waiting

> but if I start from the command line its status becomes "start/running"?

> # start resolvconf
> resolvconf start/running
> # status resolvconf
> resolvconf start/running

> What's the relevant difference, in other words, between the start command
> and "start on mounted MOUNTPOINT=/run"?

The only relevant difference is that somehow, resolvconf failed to start for
you at boot. Can you reproduce this on subsequent reboots? If so, please
install version 1.4-0ubuntu7 of upstart from precise and attach the
resulting /var/log/upstart/resolvconf.log file, so we can get the failure
messages.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Steve Langasek (vorlon)]

The necessary version of upstart is now in precise. Upgrading upstart and rebooting should be sufficient to get a /var/log/upstart/resolvconf.log showing the nature of the job failure.

[Thomas Hood (jdthood)]

So the answer to my question (#16) was that the resolvconf Upstart job was failing to start at boot time because of #927803.

The misbehavior reported here (#929552) occurs post-boot so I am doubtful that it has the same cause.

[Thomas Hood (jdthood)]

Once #936835 is fixed, the likelihood of the resolvconf Upstart job failing to start will be reduced and the installation failures reported here (#929552, #929641) might disappear --- without the underlying cause(s) being eliminated.

Once #936835 is fixed, that is, the resolvconf Upstart job will only fail to start if /run/resolvconf/enable-updates can't be created. That will only be the case if there was some problem mounting /run or something like that. The Upstart job will no longer fail due to an error condition returned by a resolvconf update hook script.

[Thomas Hood (jdthood)]

Kosmonaut, asbenicio: Do you get this failure with resolvconf 1.63ubuntu8?

[Launchpad Janitor (janitor)]

[Expired for resolvconf (Ubuntu) because there has been no activity for 60 days.]

[Matt (matt-gwalski)]

This appears to make a "apt-get upgrade" fail on a precise debootstrap chroot (on powerpc, at least).

[Thomas Hood (jdthood)]

Matt, can you please provide the details about the upgrade failure you experience?