cupsd crashes regularly (daily)

Bug #1086019 reported by Steve Magoun on 2012-12-03
130
This bug affects 17 people
Affects Status Importance Assigned to Milestone
cups (Ubuntu)
High
Till Kamppeter
Quantal
High
Till Kamppeter
cups-filters (Ubuntu)
High
Till Kamppeter
Quantal
Undecided
Unassigned

Bug Description

Every day I get a 'System Problem Detected' dialog that says that cupsd has crashed. The symptom is similar to the one described in bug 1034045.

I expected this would be fixed by cups 1.6.1-0ubuntu11, but I have that version installed and the errors persist. The test case in bug 1034045 does not result in an immediate crash of cupsd. Specifically, this does not produce a crash on my system:

1. Run 'sudo rm -f /var/crash/_usr_sbin_cupsd.0.crash'
2. Run 'sudo service cupsd restart'
3. Verify that a new /var/crash/_usr_sbin_cupsd.0.crash file has been created (and if on a desktop, that a new crash notification has been shown).

[IMPACT]

For CUPS users who activate the "Show printers shared by other systems" in the server settings of system-config-printer or in the CUPS web interface, so that they automatically get shared remote CUPS printers listed in print dialogs CUPS crashes regularly, typically once a day triggered by logrotate, popping up an Apport dialog to upload the crash info to Launchpad. Otherwise all is working correctly, there is only the annoying crash report.

[TESTCASE]

Activate "Show printers shared by other systems" in the server settings of system-config-printer or run the command

cupsctl --remote-printers

I am not sure whether there must be actually a remote printer to trigger the crash. So on another computer in the same local network (or on a virtual machine if you do not have a second computer) create a CUPS queue and share it. Activate "Published shared printers connected to this system" in the server settings of system-config-printer or run the command

cupsctl --share-printers

on that computer. Now observe for some days. Probably you will get a crash report once a day.

With the proposed package the logrotate script is improved to not take a way log files from CUPS while it is running. It stops CUPS, moves the log files, and after that starts CUPS again. This way CUPS stops crashing during the logrotate once a day, so the system will behave normally now. Printing does not trigger the crash, so print jobs are always executed correctly.

You can test whether crash reports still work with the proposed package by triggering an artificial crash running the command

sudo killall -11 cupsd

Check whether the crash report process of Apport gets triggered, but DO NOT post the generated bug report on Launchpad. Remove /etc/apport/blacklist.d/cups (remainder from earlier proposed package) if this does not happen and try again.

[Regression Potential]

None. No change in any executable which could cause a regression.

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: cups 1.6.1-0ubuntu11
ProcVersionSignature: Error: [Errno 2] No such file or directory: '/proc/version_signature'
Uname: Linux 3.6.3-030603-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.6.1-0ubuntu6
Architecture: amd64
CupsErrorLog:

Date: Mon Dec 3 10:49:54 2012
InstallationDate: Installed on 2010-09-17 (808 days ago)
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Beta amd64 (20100901.1)
KernLog:
 Dec 3 10:41:14 steve-laptop kernel: [71466.813974] type=1400 audit(1354549274.550:29): apparmor="STATUS" operation="profile_replace" name="/usr/lib/cups/backend/cups-pdf" pid=15433 comm="apparmor_parser"
 Dec 3 10:41:14 steve-laptop kernel: [71466.814450] type=1400 audit(1354549274.550:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/cupsd" pid=15433 comm="apparmor_parser"
MachineType: Apple Inc. MacBookPro3,1
MarkForUpload: True
Papersize: letter
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.6.3-030603-generic root=UUID=4b3d81ed-fb5d-4946-97c0-ec537e1bfa3f ro quiet splash
SourcePackage: cups
UpgradeStatus: Upgraded to quantal on 2012-08-07 (118 days ago)
dmi.bios.date: 03/05/08
dmi.bios.vendor: Apple Inc.
dmi.bios.version: MBP31.88Z.0070.B07.0803051658
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: Mac-F4238BC8
dmi.board.vendor: Apple Inc.
dmi.board.version: PVT
dmi.chassis.asset.tag: Asset Tag#
dmi.chassis.type: 2
dmi.chassis.vendor: Apple Inc.
dmi.chassis.version: Mac-F4238BC8
dmi.modalias: dmi:bvnAppleInc.:bvrMBP31.88Z.0070.B07.0803051658:bd03/05/08:svnAppleInc.:pnMacBookPro3,1:pvr1.0:rvnAppleInc.:rnMac-F4238BC8:rvrPVT:cvnAppleInc.:ct2:cvrMac-F4238BC8:
dmi.product.name: MacBookPro3,1
dmi.product.version: 1.0
dmi.sys.vendor: Apple Inc.

Steve Magoun (smagoun) wrote :
Steve Magoun (smagoun) wrote :

I reproduced the crash again today, this time with dbus debug symbols installed. The stack trace is attached. The stack is entirely avahi + dbus code, and the signature is very similar to other crashes reported in other applications. Looks like there might be a locking problem in the way cups uses avahi?

This thread describes a similar stack trace, and has some tips about using avahi in a threaded environment:
http://marc.info/?l=freedesktop-avahi&m=122021745832173

Comment #12 of bug 524566 shows a very similar stack (without avahi), and the remainder of the bug has additional analysis.

Bug 1055060 has a crash at the same location within dbus, though the bottom of the stack is much different.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cups (Ubuntu):
status: New → Confirmed
Steve Magoun (smagoun) wrote :

I reproduced again. There is a different stack trace, but it's still triggered from avahi calling into dbus_watch_handle.

Changed in cups (Ubuntu):
importance: Undecided → High
Changed in cups (Ubuntu Quantal):
importance: Undecided → High
status: New → Confirmed
milestone: none → quantal-updates
Changed in cups (Ubuntu):
status: Confirmed → Triaged
Till Kamppeter (till-kamppeter) wrote :

I have done some tests to investigate the problem. After removing the patch for forward-porting the CUPS Broadcasting/Browsing and running the resulting CUPS for a week I did not get any crash any more, returning to the original version gave me one crash every day or every two days. Then I tried the attached patch to set the needed locks for using Avahi in a threaded environment, but instead of crashing CUPS gets stuck, still running but not responding any more. This is even worse as in the case of a crash Upstart restarts CUPS and so CUPS stays available.

So as the patch for CUPS Broadcasting/Browsing is planned to be removed again in Raring, I suggest for an SRU in Quantal only to suppress the Apport pop-ups of the crashes somehow, as a workaround. I talked with pitti on IRC about this possibility.

Setting to "Triaged" in Raring as the problem will go away with the planned removal of the patch.

Changed in cups (Ubuntu):
milestone: none → ubuntu-13.04-feature-freeze
assignee: nobody → Till Kamppeter (till-kamppeter)
Changed in cups (Ubuntu Quantal):
assignee: nobody → Till Kamppeter (till-kamppeter)
tags: added: patch
Martin Pitt (pitti) wrote :

> I suggest for an SRU in Quantal only to suppress the Apport pop-ups of the crashes somehow, as a workaround.

The simplest, but rather blunt, method is to add a file "/etc/apport/blacklist.d/cups" to a cups SRU with "/usr/sbin/cupsd". However, that would suppress *all* cups crashes, not only this one.

In order to filter out only this one in a way that completely avoids the popup (as opposed to saying "we already know about this crash"), the crash needs to be suppressed in the daemon. If you know the place where it crashes, you could temporarily install a SIGSEGV signal handler, then call the potentially crashing code path, and restore it. However, this is pretty hackish, and I wonder if there wouldn't be a more appropriate fix such as adding a NULL pointer check somewhere etc.

Till Kamppeter (till-kamppeter) wrote :

Unfortunately, the actual crash happens somewhere in Avahi or D-Bus and the stacktraces do not show through which part of CUPS the crashing piece of code was reached. Probably it is in a thread started by Avahi. So it is far from simple to find the place in CUPS where an additional NULL check is needed (this would be the real fix).

Therefore and keeping in mind that the CUPS broadcasting/browsing patch will go away in Raring, I suggest the general blocking of cupsd crash reports via /etc/apport/blacklist.d/cups. In addition, CUPS was very stable all the time, so the chance of other crashes to happen is rather low. So I think we can take the risk.

Till Kamppeter (till-kamppeter) wrote :

Uploaded a fixed package to quantal-proposed. As soon as the package gets approved it will be available for testing and a comment with instructions will be posted here. Please test it and give us fedback as your feedback is required to make the package an official update.

SRU Team: debdiff is attached.

Changed in cups (Ubuntu Quantal):
status: Confirmed → In Progress
Till Kamppeter (till-kamppeter) wrote :

Note thst the proposed package is applying a workaround which is specific to Quantal. The fix in Raring will be removing the patch for forward-porting CUPS broadcasting/browsing and replacing it by an extra daemon for Avahi browsing, making use of CUPS' Avahi broadcasts. The daemon will be part of the cups-filters package and I have nearly completed its development. So please process this SRU without the Raring task being "Fix Released".

description: updated

For me the crash always happens when the logrotate is done around 8am in the morning (beginning of /ver/log/syslog), probably the logrotating of the CUPS log files (perhaps the reload of the CUPS daemon) triggers the crash.

Restarting CUPS as in /etc/logrotate.d/cups, via

sudo invoke-rc.d --quiet cups force-reload

does not cause a crash for me, so probably manipulating the log files of CUPS (with CUPS still running), right before restarting CUPS causes the crash.

Changed in cups-filters (Ubuntu):
status: New → Triaged
Changed in cups-filters (Ubuntu Quantal):
status: New → Invalid
Changed in cups-filters (Ubuntu):
importance: Undecided → High
assignee: nobody → Till Kamppeter (till-kamppeter)
milestone: none → ubuntu-13.04-feature-freeze

I have now released cups-filters 1.0.26 upstream, with the new cups-browsed daemon added. cups-browsed browses the Bonjour broadcasts of shared remote CUPS queues and makes the queues available locally, eliminating the need of the forward-port patch for CUPS broadcasting/browsing. As soon as this package is available in Raring I will remove the patch from Raring's CUPS, eliminating the crashes.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cups - 1.6.1-0ubuntu13

---------------
cups (1.6.1-0ubuntu13) raring; urgency=low

  * debian/patches/forward-port-cups-1-5-x-cups-browsing.patch: Removed the
    forward-port of CUPS broadcasting/browsing as from cups-filters 1.0.27
    on we have cups-browsed which does Bonjour browsing for us making the
    patch unneeded. This eliminates also a crash bug in cupsd (LP: #1086019,
    LP: #1061063, LP: #1061069).
  * debian/rules: Removed "--with-remote_protocols='CUPS dnssd'" from the
    ./configure command line and removed the "CUPS" from
    "--with-local_protocols='CUPS dnssd'". These settings are not supported
    any more in stock CUPS 1.6.x (without forward-port of CUPS broadcasting/
    browsing).
  * debian/cups.postinst: Again clean /etc/cups/cupsd.conf from all keywords
    and settings which got obsolete with the dropping CUPS Broadcasting/
    Browsing in CUPS 1.6.x: BrowsePoll, BrowseAllow, BrowseDeny, BrowseOrder,
    and BrowseRemoteProtocols lines get removed and the "cups" argument gets
    removed from the BrowseLocalProtocols line.
  * debian/patches/airprint-support.patch,
    debian/patches/cupsd-no-crash-on-avahi-threaded-poll-shutdown.patch:
    Refreshed with quilt.
  * debian/patches/CVE-2012-5519.patch: Manually updated.
 -- Till Kamppeter <email address hidden> Fri, 28 Dec 2012 22:27:30 +0100

Changed in cups (Ubuntu):
status: Triaged → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cups-filters - 1.0.28-0ubuntu1

---------------
cups-filters (1.0.28-0ubuntu1) raring; urgency=low

  * New upstream release
     - cups-browsed: Added daemon to browse the Bonjour broadcasts of
       shared remote CUPS printers and automatically add local raw queues
       pointing to them, to resemble the behavior of the former CUPS
       broadcasting/browsing which was dropped in CUPS 1.6. Now remote
       printers appear as local print queues as before, but with the
       standardized Bonjour broadcasting (LP: #1086019, LP: #1061063,
       LP: #1061069).
  * debian/control: Added build dependencies on Avahi, needed for the new
    cups-browsed and also a new binary package named cups-browsed.
  * debian/cups-browsed.install: New cups-browsed binary package.
  * debian/local/cups-browsed.upstart: Upstart configuration file for
    cups-browsed.
  * debian/rules: Added everything needed so that the cups-browsed daemon gets
    started automatically.
  * debian/cups-filters.install: Include files from /usr/bin.
 -- Till Kamppeter <email address hidden> Sat, 29 Dec 2012 14:30:07 +0100

Changed in cups-filters (Ubuntu):
status: Triaged → Fix Released

Everyone who suffers this problem (probably only Quantal users), please replace your file /etc/logrotate.d/cups by the attached file:

sudo cp cups.logrotate /etc/logrotate.d/cups

Now observe for some days. Do the daily crashes go away?

Cedric M (cedric780) wrote :

cupsd crashes nearly everyday on my machine.
I just replaced /etc/logrotate.d/cups by your file and I will tell you Monday whether the crash still occurs.

Cedric M (cedric780) wrote :

Hi,
It is now Tuesday and cupsd has not crashed since I replaced '/etc/logrotate.d/cups' , my machine was online all the week end.

Cedric, thanks for the test. I will apply the new logrotate script for CUPS soon, also for newer versions to avoid crashes and problems in the future, simply by having a cleaner logrotate script.

Everyone testing my new logrotate script (comment #15), please do not forget to remove the /etc/apport/blacklist.d/cups file from the initially proposed package.

Uploaded a new fixed package to quantal-proposed, this time using the improved logrotate script. As soon as the package gets approved it will be available for testing and a comment with instructions will be posted here. Please test it and give us feedback as your feedback is required to make the package an official update.

SRU Team: debdiff is attached.

description: updated

Cedric, and everyone else who tested my improved logrotate script, as soon as the proposed package gets available please run the following commands in a terminal window to remove my script before installing the proposed package:

sudo rm /etc/logrotate.d/cups /etc/apport/blacklist.d/cups
mkdir tmpdir
cd tmpdir
apt-get download cups
sudo dpkg -i --force-confmiss cups_*.deb
cd ..
rm -rf tmpdir

Then install the package as described in the instructions which will get posted here.

Note that testing the proposed package and giving feedback here is required for the proposed package to get an official update.

description: updated

When testing using the commands above, please replace

sudo dpkg -i --force-confmiss cups_*.deb

by

sudo dpkg -i --force-confmiss cups-daemon_*.deb
sudo dpkg -i --force-confmiss cups_*.deb

Cedric M (cedric780) wrote :

I think I missed something, where do I find the 'proposed package' ?
Will automatic update propose it to me? Do I have to upgrade Ubuntu from 12.10 to 13.04 ?

Cedric, you do not need to upgrade to 13.04.

The proposed package is not yet made available. It will be announced here. I will try to make the SRU aware of the fact that they still did not approve this package.

Hello Steve, or anyone else affected,

Accepted cups into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cups/1.6.1-0ubuntu11.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cups (Ubuntu Quantal):
status: In Progress → Fix Committed
tags: added: verification-needed

Cedric, please follow the instructions of comment #25, comment #21, and comment #22 and tell us your results here. First, do the steps of BOTH comment #21 and comment #22 to clean up the configuration changes of your first testing/workaround. ONLY AFTER THAT, follow the instructions of comment #25.

Please report your results here. Thanks.

Cedric M (cedric780) wrote :

Already did #21 two weeks ago and crashes came back.

#22 does not work:
--------------------
cmarin@BAT:~/tmpdir$ sudo dpkg -i --force-confmiss cups-daemon_*.deb

dpkg: erreur de traitement de cups-daemon_*.deb (--install) :
 ne peut pas accéder à l'archive: Aucun fichier ou dossier de ce type
Des erreurs ont été rencontrées pendant l'exécution :
 cups-daemon_*.deb
---------------------
in english:
 cannot access to archive: no such file or directory

I tried to add a "apt-get download cups-daemon" but it does not work either.

Cedric, sorry, forget about comment #22. The cups-daem,on package was only introduced with 13.04. So you need to follow only the instructions of comment #21 and this you have done two weeks ago. Now you should observe the daily crashes, which you already confirmed.

So now please install the proposed package as described in comment #25. After that you should not observe the crashes any more. Please tell us whether this is the case.

Cedric M (cedric780) wrote :

#25 done via automatic update: it worked but a dialog box asked me whether to replace or not the content of a file like "logrotate.d/cups" (I didn't note it, sorry). The file changes text field was empty.

I choose "replace" as recommended by the dialog but if this file is related to the crash maybe the replace should be forced or at least it should be told that 'replace' is needed to fix a crash.

Cedric, the pop-up about replacing the file does not appear for normal users but only for you because you have manually put logrotate.d/cups file in place for doing the first tests I asked you for. Having answered "Replace" was correct. You are now using the file of the package. So now you are all set for the test. If you have no crash report tomorrow, the fix in the proposed package is verified.

Cedric M (cedric780) wrote :

Bad news : I had a cupsd crash this morning.
I will try to post the apport crash here.

Cedric M (cedric780) wrote :

Here is the crash log found on /var/crash .

Cedric, can you run

ls -l /etc/apport/blacklist.d/cups*

post the output here, and attach all files which get listed by this command.

Please attach the files one by one, do not compress or package them.

Cedric M (cedric780) wrote :

I just checked the '/etc/logrotate.d/cups' file and it has the same content as the improved script you provided.

Cedric M (cedric780) wrote :

>> ls -l /etc/apport/blacklist.d/cups*
ls: impossible d'accéder à /etc/apport/blacklist.d/cups*: Aucun fichier ou dossier de ce type

==> No such file or directory

>> ls -l /etc/apport/blacklist.d/*
-rw-r--r-- 1 root root 216 oct. 10 2012 /etc/apport/blacklist.d/apport
-rw-r--r-- 1 root root 59 oct. 11 2012 /etc/apport/blacklist.d/firefox
-rw-r--r-- 1 root root 217 oct. 1 2012 /etc/apport/blacklist.d/README.blacklist
-rw-r--r-- 1 root root 71 oct. 11 2012 /etc/apport/blacklist.d/thunderbird

Cedric, sorry, forget about the previous comment, I wanted to ask you another thing.

You are sure that your file /etc/logrotate.d/cups is exactly the same as the one I asked you for testing in comment #15.

Can you attach your /etc/logrotate.d/cups file here? Thanks.

Cedric M (cedric780) wrote :

Here is my '/etc/logrotate.d/cups' file.

Cedric M (cedric780) wrote :

My machine will be online the 3 following days and idle.
I will see Tuesday if other crashes occurred.
If you want I can also reboot the machine before leaving the office (in one hour).

Cedric, the script is the correct one. Please do the reboot and then we will see whether you get further crashes.

Cedric M (cedric780) wrote :

Hello, there is no more crash since the reboot last week.
I needed to wait more because we had a general power failure nearly at the hour where cupsd would usually crash.
( I needed to check your fix didn't cause the power failure :-P )
I think you can set the bug as fixed now.

Cedric, thank you very much. I have marked the fix as verified now.

tags: added: verification-done
removed: verification-needed

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cups - 1.6.1-0ubuntu11.4

---------------
cups (1.6.1-0ubuntu11.4) quantal-proposed; urgency=low

  * debian/cups.logrotate: Do not remove CUPS' log files while the CUPS
    daemon is running. Stop CUPS, move the files, and then start it again.
    This avoids crashes during the log rotation process (LP: #1086019).
 -- Till Kamppeter <email address hidden> Tue, 23 Apr 2013 12:12:12 +0200

Changed in cups (Ubuntu Quantal):
status: Fix Committed → Fix Released
Majestyx (majestyx) wrote :

all users can't print from libreoffice (oodt) files - only PDF printing works

To post a comment you must log in.