squid causing /var to stay busy during shutdown

Bug #750371 reported by ryokenau on 2011-04-04
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
squid (Ubuntu)
Undecided
Unassigned
Lucid
High
Unassigned
Maverick
High
Unassigned

Bug Description

Binary package hint: squid

I'm running Ubuntu 10.04.2 LTS with latest version of squid 2.7.STABLE7-1ubuntu12.2. This is a default install - there have been no changes to /etc/squid.conf.

When I shutdown or reboot the system, I get the following error:

 * Unmounting local filesystems...
umount2: Device or resource busy
umount: /var: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy
umount2: Device or resource busy
umount: /dev/mapper/vg01-usr busy - remounted read-only
                                                                  [fail]

When the system starts up again, I see the following message:

/dev/mapper/vg01-var: recovering journal

It appears the /var partition was not cleanly unmounted during shutdown. Obviously, there is a risk of filesystem corruption here. FYI, i have /var and other important mountpoints as separate partitions (using LVM and ext4).

To confirm my suspicions of squid still running during shutdown, I checked /var/log/squid/cache.log and there is no indication that it received the signal to terminate. In fact, cache.log hints at this problem while the system is being brought up again:

Rebuilding storage in /var/spool/squid (DIRTY)

Only when I manually run "stop squid", then cache.log would report that squid stopped successfully:

Preparing for shutdown after 0 requests
Waiting 30 seconds for active connections to finish
FD 14 Closing HTTP connection
Shutting down...
FD 15 Closing ICP connection
Closing unlinkd pipe FD 12
... <skipped> ...
logfileClose: closing log /var/log/squid/store.log
logfileClose: closing log /var/log/squid/access.log
Squid Cache (Version 2.7.STABLE7): Exiting normally.

To test whether /var was locked by squid during shutdown, I added a script to run "ls /var/run | grep squid" and "lsof | grep var" before filesystems are unmounted. And voila!

ls /var/run | grep squid:
squid.pid

lsof | grep var:
squid 971 proxy cwd DIR 251,4 4096 130613 /var/spool/squid
squid 971 proxy 5u REG 251,4 13617 128678 /var/log/squid/cache.log
squid 971 proxy 7w REG 251,4 0 128724 /var/log/squid/access.log
squid 971 proxy 9w REG 251,4 0 128789 /var/log/squid/store.log
squid 971 proxy 11w REG 251,4 48 130308 /var/spool/squid/swap.state

Hence, the next system startup would result in /var recovering journal again.

Finally, I tried running "stop squid" before issuing the shutdown command and it successfully unmounts /var and I do not get /var recovering journal on the next system startup.

ryokenau (ryokenau) on 2011-04-04
description: updated
description: updated
ryokenau (ryokenau) on 2011-04-04
description: updated
James Page (james-page) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better.

I was able to confirm this issue in the latest development release as well.

Clint Byrum (clint-fewbar) wrote :

/etc/init/squid.conf actually does have the appropriate 'stop on':

stop on runlevel [!2345]

However, umountfs/sendsigs don't wait for these things to actually die before completing the shutdown.

That is bug #688541 . So , This is a duplicate of that bug, marking as such.

Jorge Molina (jmolina) wrote :

On ubuntu 10.04.3 LTS the problem is on the "start on" line in /etc/init/squid.conf file. It is different from the one in ubuntu 11.04.

You should have something like:

start on runlevel [2345] or net-device-up IFACE!=lo
stop on runlevel[!2345]

This fixed the problem for me. Now squid shut downs properly.

The /etc/init.d/sendsig works just fine and it has nothing to do with this bug since it ignores processes managed by upstart like squid.

PD: I found a similar problems in /etc/init/sshd.conf causing problems when switching runlevels.

Jorge Molina (jmolina) wrote :

Also on the 2nd line for 'stop on' there is no space after runlevel.

Correct syntax should be:

start on runlevel [2345] or net-device-up IFACE!=lo
stop on runlevel [!2345]

I also applied codewarrior's patch to sendsig found on https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/616287

With these 2 changes I fixed 2 servers running 10.04.3 LTS that had this problem.

Clint Byrum (clint-fewbar) wrote :

Good eyes there Jorge. I missed that entirely.

It seems I inadvertently fixed this in natty at 2.7.STABLE9-2.1ubuntu6 when fixing bug #561779. But it needs to be fixed in maverick and Lucid as well. I would agree with you that this is a High Priority, as squid needs to be stopped!

Changed in squid (Ubuntu):
status: New → Fix Released
Changed in squid (Ubuntu Lucid):
status: New → Triaged
importance: Undecided → High
Changed in squid (Ubuntu Maverick):
status: New → Triaged
importance: Undecided → High
Clint Byrum (clint-fewbar) wrote :

I've uploaded fixed squids to lucid-proposed and maverick-proposed. Thanks again Jorge!

Hello ryokenau, or anyone else affected,

Accepted squid into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in squid (Ubuntu Lucid):
status: Triaged → Fix Committed
tags: added: verification-needed
Chris Halse Rogers (raof) wrote :

Hello ryokenau, or anyone else affected,

Accepted squid into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in squid (Ubuntu Maverick):
status: Triaged → Fix Committed
Jorge Molina (jmolina) wrote :

Thank you very much for your time!

ryokenau (ryokenau) wrote :

Thank you very much Clint and Chris, and especially Jorge!

I have finally found some time to get rid of my ugly workaround to this bug (which involved manually stopping squid before filesystem unmounting) and installed squid 2.7.STABLE7-1ubuntu12.4 from lucid-proposed.

Besides a "dirty" filesystem upon the first shutdown right after the update, all subsequent shutdown/reboot attempts were fine. Looks like this pesky bug is finally squashed! :)

Clint Byrum (clint-fewbar) wrote :

Thanks ryokenau, marking verification-done-lucid.

We still need somebody to verify this on maverick. Thanks!

tags: added: verification-done-lucid
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package squid - 2.7.STABLE7-1ubuntu12.4

---------------
squid (2.7.STABLE7-1ubuntu12.4) lucid-proposed; urgency=low

  * d/squid.upstart: start on was not sufficient to restart squid
    on transition from runlevel 1 to 2 (LP: #561779). Also stop on
    condition was not sufficient to stop squid because of a missing
    space. (LP: #750371)
 -- Clint Byrum <email address hidden> Sat, 27 Aug 2011 03:20:45 -0700

Changed in squid (Ubuntu Lucid):
status: Fix Committed → Fix Released
Martin Pitt (pitti) on 2011-10-13
tags: removed: verification-done-lucid
Steve Beattie (sbeattie) wrote :

I was able to reproduce this issue with squid 2.7.STABLE9-2ubuntu5.1, and have verified that the version in maverick-proposed, 2.7.STABLE9-2ubuntu5.2 appears to fix the issue. After upgrading, squid continued to function as expected. Marking verification-done.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package squid - 2.7.STABLE9-2ubuntu5.2

---------------
squid (2.7.STABLE9-2ubuntu5.2) maverick-proposed; urgency=low

  * d/squid.upstart: start on was not sufficient to restart squid
    on transition from runlevel 1 to 2 (LP: #561779). Also stop on
    condition was not sufficient to stop squid because of a missing
    space. (LP: #750371)
 -- Clint Byrum <email address hidden> Sat, 27 Aug 2011 02:21:34 -0700

Changed in squid (Ubuntu Maverick):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers