package squid 2.7.STABLE7-1ubuntu12.1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1

Bug #717397 reported by Greg Rundlett on 2011-02-11
338
This bug affects 42 people
Affects Status Importance Assigned to Milestone
squid (Ubuntu)
High
Clint Byrum
Lucid
Undecided
Unassigned

Bug Description

Binary package hint: squid

Did a normal package update and got this error. Sorry I don't have more details

ProblemType: Package
DistroRelease: Ubuntu 10.04
Package: squid 2.7.STABLE7-1ubuntu12.1
ProcVersionSignature: Ubuntu 2.6.32-28.55-generic 2.6.32.27+drm33.12
Uname: Linux 2.6.32-28-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Fri Feb 11 08:43:15 2011
ErrorMessage: subprocess installed post-installation script returned error exit status 1
SourcePackage: squid
Title: package squid 2.7.STABLE7-1ubuntu12.1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1

== SRU Report ==

=== Impact ===

The bug will cause errors for anyone who has experienced bug #573853 while they try to upgrade to fix that very bug, which can be very frustrating and confusing. It will also generate a lot of apport bug reports like the ones that led to this report.

=== Dev fix ===

This should be fixed in the natty packages in the same way, though the version numbers may need to be adjusted.

=== Patch ===

See merge proposal for lucid package. Similar fix can be done for natty and maverick too. The maverick fix should accompany the fix for bug #573853

=== TEST CASE: ===

1. install squid from lucid (not lucid-updates)
2. sudo reload squid
3. upgrade to the squid in lucid-updates
4. the postinst will fail to restart squid
5. repeat steps 2 - 4, but upgrade to the fixed package instead, the postinst should succeed.

=== Regression Potential ===

Its possible that this code may introduce a new bug in the preinst that will cause some trouble, though a healthy dose of testing has been taken to try and prevent that.

Related branches

Greg Rundlett (greg.rundlett) wrote :
Clint Byrum (clint-fewbar) wrote :

Hello Greg (and duplicate reporters), thanks for taking the time to file this bug report!

So unfortunately, this is likely the result of the squid process having disappeared because of bug #573853 , which ironically was fixed in the package that you are all trying to install.

This is a temporary problem, and the way to alleviate it is to kill squid off manually (it may still be running despite upstart having lost track of it), and then start it manually.

These three commands may print errors (and may leave your system in a bad state, so please be cautious), but should lead to the upgrade finishing:

sudo stop squid
sudo killall squid
sudo start squid
sudo dpkg --configure -a

Unfortunately, given the nature of the bug fixed there's no safe way that I have come up with to plan for this possible failure and get around it in the postinst. Given that, I believe this bug has to be closed as Won't Fix.

If someone comes up with a better way to handle the upgrade when the squid job is in an unknown state, then please re-open it.

Changed in squid (Ubuntu):
status: New → Won't Fix
Clint Byrum (clint-fewbar) wrote :

Ok well after saying that, and talking a bit more with Chuck Short, there is a safe way to handle it, so I'm re-opening it as Triaged and assigning to myself.

Changed in squid (Ubuntu):
status: Won't Fix → Triaged
importance: Undecided → High
assignee: nobody → Clint Byrum (clint-fewbar)
tags: added: regression-update
Changed in squid (Ubuntu):
status: Triaged → In Progress
Clint Byrum (clint-fewbar) wrote :

The branch lp:~clint-fewbar/ubuntu/lucid/fix-upgrades-after-reload fixes the problem by checking to see if the problem the recent update is trying to correct has actually happened, and if so, doing a stop/start on the service using invoke-rc.d Between the stop/start, it also kills the errant pid (but only if invoke-rc.d worked, so policy is still respected).

I test this on a lucid vm in the following scenarios:

* bare bones no updates with just 'apt-get install squid' version 2.7.STABLE7-1ubuntu12
 - When preinst is run, the upstart pid and the pidfile agree, and so it exits doing nothing.

* same, but also running 'sudo reload squid', verifying that squid is not tracked by upstart anymore with 'status squid' showing stop/waiting, but 'ps auxw | grep squid' showing a running squid.
 - the job is stopped, the old squid is killed, squid is started again

* start squid, reload, kill -9, leaving stale pid file.
 - Code stops because /proc/`cat pidfile` does not exist.

* modify /etc/init/squid.conf to not use expect fork, and pass -N (basically manually putting in the new job file). This simulates the chance that a user may have applied the fix manually.
 - Code stops because the upstart pid matches the pidfile.

* stopped service before hand
 - pidfile is gone, code is not run

* removed upstart file
 - code checks for that, does not run.

Since creating the merge proposal I also tried testing it with a /usr/sbin/policy-rc.d that exitted with 102, the appropriate way to disable a service's stop/starting, and the code was skipped. When the policy-rc.d returned 0, the code worked as intended (invoke-rc.d's behavior was basically unchanged).

Clint Byrum (clint-fewbar) wrote :

As I finish the SRU report, I realize that we actually need to address this in natty too, and may need to drop the version comparison to make it less complex and more likely to fix the problem given scenarios I haven't thought of.

description: updated
Martin Pitt (pitti) wrote :

Keeping in queue until bug 573853 is verified and the previous squid in -proposed goes to -updates.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package squid - 2.7.STABLE9-2.1ubuntu3

---------------
squid (2.7.STABLE9-2.1ubuntu3) natty; urgency=low

  * Detect and deal with the situation where upstart has lost track of
    squid. (LP: #717397)
 -- Clint Byrum <email address hidden> Thu, 24 Feb 2011 10:54:40 -0500

Changed in squid (Ubuntu):
status: In Progress → Fix Released

Accepted squid into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in squid (Ubuntu Lucid):
status: New → Fix Committed
tags: added: verification-needed
Dave Walker (davewalker) wrote :

It doesn't seem that this package handles the state where squid is in the stop/waiting state prior to upgrade. The previous restart style stanzas are encapsulated in checking current status if statements.

Setting up squid (2.7.STABLE7-1ubuntu12.2) ...
restart: Job failed to restart
dpkg: error processing squid (--configure):
 subprocess installed post-installation script returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up squid (2.7.STABLE7-1ubuntu12.2) ...

Aptitude did successfully recover from this situation, currently unsure how apt-get would. Although this has an edge case, it's a better situation than leaving bug 725635 out in the wild.

Steve Langasek (vorlon) wrote :

Well, that's because the postinst isn't using the init script policy layer.

if status squid | grep "start/running" > /dev/null; then
        restart squid
fi

if [ "$FIXLINES" = "false" ]; then
  echo "squid.conf contains 2.2.5 syntax - cache_dir directive - not starting "
  echo "Run 'dpkg-reconfigure -plow squid' to fix this automatically,"
  echo "or fix the 'cache_dir'-entry in your squid.conf manually."
  echo "See documentation in /usr/share/doc/squid for nearer instructions."
else
  if status squid | grep "start/running" > /dev/null; then
        restart squid
  fi
fi

Why isn't this "invoke-rc.d squid restart" like it should be? Also, the fixlines check is pointless given that it's preceded by a squid restart command. This all seems to have been a wrong fix for bug #552360 and/or bug #509307.

This ought to be fixed up in SRU, probably at the same time.

On Sun, 2011-02-27 at 20:47 +0000, Steve Langasek wrote:
> Well, that's because the postinst isn't using the init script policy
> layer.
>
> if status squid | grep "start/running" > /dev/null; then
> restart squid
> fi
>
> if [ "$FIXLINES" = "false" ]; then
> echo "squid.conf contains 2.2.5 syntax - cache_dir directive - not starting "
> echo "Run 'dpkg-reconfigure -plow squid' to fix this automatically,"
> echo "or fix the 'cache_dir'-entry in your squid.conf manually."
> echo "See documentation in /usr/share/doc/squid for nearer instructions."
> else
> if status squid | grep "start/running" > /dev/null; then
> restart squid
> fi
> fi
>
> Why isn't this "invoke-rc.d squid restart" like it should be? Also, the
> fixlines check is pointless given that it's preceded by a squid restart
> command. This all seems to have been a wrong fix for bug #552360 and/or
> bug #509307.

This is particularly troubling because the 'restart' command does not
re-load the upstart job file, while invoke-rc.d's restart method, IIRC,
runs stop/start for this very reason.

That said, currently if the process is stop/waiting, its left in that
very state on upgrade. So while this is a bug, and should be fixed on
the next upload, I don't believe it is the cause of the issue.

The things that can cause a "failed to restart" will also cause a
'failed to stop' or 'failed to start'. Namely:

pre-start exits non zero
post-stop exits non zero
main process cannot be executed

There are a bunch of things in the pre-start in particular that might
fail. In this case, re-running the preinst would result in the postinst
succeeding, but squid not running.

I do think this is related, but not the same, so I've opened bug #726348
to address that issue separately.

Dave Walker (davewalker) on 2011-03-02
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package squid - 2.7.STABLE7-1ubuntu12.2

---------------
squid (2.7.STABLE7-1ubuntu12.2) lucid-proposed; urgency=low

  * Detect and deal with the situation where upstart has lost track of
    squid. (LP: #717397)
 -- Clint Byrum <email address hidden> Mon, 14 Feb 2011 14:39:47 -0800

Changed in squid (Ubuntu Lucid):
status: Fix Committed → Fix Released
Dave Walker (davewalker) wrote :

fwts (0.22.06) natty; urgency=low

  * acpi: cstates: fix progress calculation, tidy up code.

 -- Colin King <email address hidden> Fri, 4 Mar 2011 10:37:00 +0000

fwts (0.22.05) natty; urgency=low

  * Fix cstates test (LP: #728685)
  * acpi: acpidump: use more sensible heading, fix SCI text
  * hpet: Fail test and skip sanity check when base address not found
  * bios: ebda_region: close fd on error

 -- Colin King <email address hidden> Thu, 3 Mar 2011 20:42:02 +0000

fwts (0.22.04) natty; urgency=low

  * acpi: acpidump: use more sensible heading, fix SCI text
  * hpet: Fail test and skip sanity check when base address not
  * bios: ebda_region: close fd on error.

 -- Colin King <email address hidden> Thu, 3 Mar 2011 09:37:39 +0000

Dave Walker (davewalker) wrote :

Oops, wrong bug :/

Tim Nicholas (tjn) wrote :

This fix appears to have broken the upgrade process.

On two systems, it ended up failing to finish the upgrade. After killing the upgrade processes I was unable to stop the still running squid using 'service squid stop' or 'stop squid' or '/etc/init.d/squid stop' or anything else.

After that I couldn't do anything with squid as it was in a 'stop/killed' (with a PID that doesn't exist). See bug: https://bugs.launchpad.net/upstart/+bug/406397

Can I suggest an upgrade to this crazy new init system I've heard of called 'System V'. It sounds badass. And reliable/predictable.

Also, any hints as to starting the daemon without rebooting would be appreciated.

Tim Nicholas (tjn) wrote :

Oh. This was on 10.04 (LTS).

tags: added: testcase
Michael Terry (mterry) on 2012-03-22
no longer affects: squid-deb-proxy (Ubuntu)
no longer affects: squid-deb-proxy (Ubuntu Lucid)
To post a comment you must log in.