cupsd assert failure: *** glibc detected *** /usr/sbin/cupsd: free(): corrupted unsorted chunks: 0x00007f3dc478c0f0 ***

[Steve Langasek]

[Impact]
Affected users are shown a crash dialog roughly once a day (when cups restarts on log rotation). There is no impact on cups functionality, but a very visible impact on user experience.

[Test Case]
1. Run 'sudo rm -f /var/crash/_usr_sbin_cupsd.0.crash'
2. Run 'sudo service cupsd restart'
3. Verify that a new /var/crash/_usr_sbin_cupsd.0.crash file has been created (and if on a desktop, that a new crash notification has been shown). If this file is not created for you, then you are not affected by this bug; stop here.
4. Install cups 1.6.1-0ubuntu11 from quantal-proposed
5. Repeat steps 1 and 2
6. Verify that cupsd no longer crashes on restart (i.e., /var/crash/_usr_sbin_cupsd.0.crash is not created).

[Regression Potential]
Minimal; the change only affects the shutdown sequence of cups, so the worst case would seem to be introducing a different but related non-service-affecting crash for other users.

The cups job auto-restarted as intended after this crash, but this looks like a worrisome failure mode

ProblemType: Crash
DistroRelease: Ubuntu 12.10
Package: cups 1.6.1-0ubuntu1
ProcVersionSignature: Ubuntu 3.5.0-8.8-generic 3.5.0
Uname: Linux 3.5.0-8-generic x86_64
ApportVersion: 2.4-0ubuntu6
Architecture: amd64
AssertionMessage: *** glibc detected *** /usr/sbin/cupsd: free(): corrupted unsorted chunks: 0x00007f3dc478c0f0 ***
Date: Tue Aug 7 08:03:04 2012
ExecutablePath: /usr/sbin/cupsd
InstallationMedia: Ubuntu 10.04.1 LTS "Lucid Lynx" - Release amd64 (20100816.1)
Lpstat:
 device for davidson-printer: smb://WORKGROUP/EXCALIBUR/Printer2
 device for HP-Color-LaserJet-CM1312-MFP: hp:/usb/HP_Color_LaserJet_CM1312_MFP?serial=00CNB882VG58
 device for HP-PSC-750: ipp://207.224.24.210:631/printers/PSC_750
 device for HP-PSC-750-legal: ipp://207.224.24.210:631/printers/PSC_750
 device for PSC-750: hp:/usb/PSC_750?serial=MY2ASD4157WB
MachineType: LENOVO 3249CTO
Papersize: letter
PpdFiles:
 HP-PSC-750: HP PSC 750, hpcups 3.12.6
 HP-PSC-750-legal: HP PSC 750, hpcups 3.12.4
 HP-Color-LaserJet-CM1312-MFP: HP Color LaserJet CM1312 MFP Series Postscript (recommended)
 PSC-750: HP PSC 750, hpcups 3.12.6
 davidson-printer: HP Color LaserJet CP3505 Postscript (recommended)
ProcAttrCurrent: /usr/sbin/cupsd (enforce)
ProcCmdline: /usr/sbin/cupsd -F
ProcEnviron:
 PATH=(custom, no user)
 TERM=linux
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.5.0-8-generic root=/dev/mapper/hostname-root ro quiet splash vt.handoff=7
Signal: 6
SourcePackage: cups
StacktraceTop:
 __libc_message (do_abort=2, fmt=0x7f3dc15f50f0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:201
 malloc_printerr (action=3, str=0x7f3dc15f5270 "free(): corrupted unsorted chunks", ptr=<optimized out>) at malloc.c:5007
 ?? () from /lib/x86_64-linux-gnu/libdbus-1.so.3
 ?? () from /lib/x86_64-linux-gnu/libdbus-1.so.3
 dbus_connection_dispatch () from /lib/x86_64-linux-gnu/libdbus-1.so.3
Title: cupsd assert failure: *** glibc detected *** /usr/sbin/cupsd: free(): corrupted unsorted chunks: 0x00007f3dc478c0f0 ***
UpgradeStatus: Upgraded to quantal on 2012-06-11 (56 days ago)
UserGroups:

dmi.bios.date: 08/23/2010
dmi.bios.vendor: LENOVO
dmi.bios.version: 6QET52WW (1.22 )
dmi.board.name: 3249CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr6QET52WW(1.22):bd08/23/2010:svnLENOVO:pn3249CTO:pvrThinkPadX201:rvnLENOVO:rn3249CTO:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 3249CTO
dmi.product.version: ThinkPad X201
dmi.sys.vendor: LENOVO
modified.conffile..etc.cups.cupsd.conf: [modified]
modified.conffile..etc.default.cups: [modified]
mtime.conffile..etc.cups.cupsd.conf: 2011-08-09T08:12:10
mtime.conffile..etc.default.cups: 2010-09-25T01:44:40

[Apport retracing service (apport)]

StacktraceTop:
 __libc_message (do_abort=2, fmt=0x7f3dc15f50f0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:201
 malloc_printerr (action=3, str=0x7f3dc15f5270 "free(): corrupted unsorted chunks", ptr=<optimized out>) at malloc.c:5007
 ?? () from /tmp/tmpHaJRki/lib/x86_64-linux-gnu/libdbus-1.so.3
 ?? () from /tmp/tmpHaJRki/lib/x86_64-linux-gnu/libdbus-1.so.3
 dbus_connection_dispatch () from /tmp/tmpHaJRki/lib/x86_64-linux-gnu/libdbus-1.so.3

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Till Kamppeter (till-kamppeter)]

Reported upstream: http://www.cups.org/str.php?L4180

[Felix Möller (felix-derklecks)]

Till have the patches mentioned upstream by mike been integrated? http://www.cups.org/str.php?L4180?

[Till Kamppeter (till-kamppeter)]

Felix, I am applying them right now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cups - 1.6.1-0ubuntu9

---------------
cups (1.6.1-0ubuntu9) quantal; urgency=low

  [ Till Kamppeter ]
  * debian/patches/prevent-crash-due-to-null-host-name-or-fqdn-from-avahi.patch:
    Prevent crash due to NULL host name or FQDN from Avahi (CUPS STR #4183,
    CUPS STR #4180, LP: #1046982, LP: #1034045).
  * debian/patches/fix-crash-on-shutdown-caused-by-broken-avahi-config.patch:
    Fix crash on shutdown caused by broken Avahi config (CUPS STR #4192,
    LP: #1036974).
  * debian/patches/ipp-backend-abort-the-outer-loop-if-we-get-a-failure-from-send-document.patch,
    debian/patches/ipp-backend-could-get-stuck-in-an-endless-loop-on-certain-network-errors.patch:
    Prevent IPP backend from falling into an infinite loop in certain
    situations (CUPS STR #4194).
  * debian/patches/airprint-support.patch: Refreshed with quilt.

  [ Didier Raboud ]
  * Make sure unowned obsolete backends are removed on configure.
    (Closes: #683754)
 -- Till Kamppeter <email address hidden> Mon, 1 Oct 2012 12:27:30 +0200

[Steve Langasek (vorlon)]

This crash is still happening for me here with cups 1.6.1-0ubuntu10, so it doesn't appear to be fixed.

[Till Kamppeter (till-kamppeter)]

The upstream bug which was fixed and assumed to be the same bug as this one is probably not actually this bug and the guess went wrong due to lack of additional information around the pure stack trace.It can even be that this is not a CUPS bug but a bug in Avahi or D-Bus. The crash happens in D-Bus code which is called by Avahi code and this by CUPS code.

Bug 1061457 and bug 1051799 seem to be the same.

Steve, can you give any info about the situations when the crash occurs for you? Like on each job, on each CUPS startup/shutdown, .. Does it only occur for a certain printer (connection type), jobs from a certain app, ...?

Can you supply an error_log in debug mode (https://wiki.ubuntu.com/DebuggingPrintingProblems#CUPS_error_log) and /var/log/syslog to see what happened right before the crash.

[Steve Langasek (vorlon)]

As requested on IRC, I'm attaching printers.conf from the affected machine. This is the complete list of printer queues for this system. Only the queues on 207.224.24.210 are active. 207.224.24.210 is running CUPS 1.4.5 on Linux.

[Steve Langasek (vorlon)]

On Thu, Oct 11, 2012 at 03:59:53PM -0000, Till Kamppeter wrote:
> Steve, can you give any info about the situations when the crash occurs
> for you? Like on each job, on each CUPS startup/shutdown, .. Does it
> only occur for a certain printer (connection type), jobs from a certain
> app, ...?

It's unrelated to any printing. This crash happens daily, with no
interaction with the printer. It appears to happen at cups shutdown,
triggered by the daily log rotation.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Till Kamppeter (till-kamppeter)]

Added comment to the upstream bug http://www.cups.org/str.php?L4180 telling that the problem still persists after applying the patches to fix the other crash bugs.

[Till Kamppeter (till-kamppeter)]

Do you all have the newest avahi packages installed, version 0.6.31-1ubuntu2?

[Till Kamppeter (till-kamppeter)]

Do you have avahi-daemon installed and running?

[Steve Langasek (vorlon)]

On Thu, Oct 11, 2012 at 06:33:14PM -0000, Till Kamppeter wrote:
> Do you all have the newest avahi packages installed, version
> 0.6.31-1ubuntu2?

This is a completely up-to-date quantal system.

[Steve Langasek (vorlon)]

On Thu, Oct 11, 2012 at 06:37:30PM -0000, Till Kamppeter wrote:
> Do you have avahi-daemon installed and running?

Yes.

[Till Kamppeter (till-kamppeter)]

Can you attach your /etc/cups/cupsd.conf file?

[Till Kamppeter (till-kamppeter)]

Please run the command (install avahi-discover package if needed):

avahi-discover

Do your print queues appear under "Internet Printer" in the avahi-discover window?

[Till Kamppeter (till-kamppeter)]

Can you also attach an error_log (in debug mode, see https://wiki.ubuntu.com/DebuggingPrintingProblems#CUPS_error_log) to see the last words of cupsd before it crashes? After having set CUPS to debug logging via

cupsctl LogLevel=debug2

run

date; sudo reload cups; date

and post the output.

After that attach your error_log.

[Till Kamppeter (till-kamppeter)]

 Added Avahi and D-Bus tasks, as the crash actually happens in D-Bus code called by Avahi code called by CUPS code.

[Steve Langasek (vorlon)]

> Can you attach your /etc/cups/cupsd.conf file?

Apparently not. Launchpad is refusing to let me attach it! So these attachments will have to wait.

> Do your print queues appear under "Internet Printer" in the avahi-discover window?
Yes.

[Steve Langasek (vorlon)]

oh, of course; the file wasn't readable by my user so I couldn't attach it. Attached now.

[Steve Langasek (vorlon)]

# date; sudo reload cups; date
Fri Oct 12 13:58:28 PDT 2012
Fri Oct 12 13:58:28 PDT 2012
#

And the error log.

[Till Kamppeter (till-kamppeter)]

Did cupsd actually crash when you did the tests described in comment #26? If not, repeat the tests, perhaps trying other modes to restart or simply stop CUPS. Attach the error_log of a situation where cupsd actually crashed.

[Till Kamppeter (till-kamppeter)]

Please also attach your /var/log/syslog then to see the exact time point where cupsd crashes.

[Steve Langasek (vorlon)]

On Sat, Oct 13, 2012 at 09:42:50AM -0000, Till Kamppeter wrote:
> Did cupsd actually crash when you did the tests described in comment
> #26?

Yes, it did.

[Steve Langasek (vorlon)]

On Sat, Oct 13, 2012 at 09:43:33AM -0000, Till Kamppeter wrote:
> Please also attach your /var/log/syslog then to see the exact time point
> where cupsd crashes.

There is nothing in my /var/log/syslog related to this crash.

[Steve Langasek (vorlon)]

Hello Steve, or anyone else affected,

Accepted cups into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cups/1.6.1-0ubuntu11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Following the test case that I've added to the bug description, I can confirm that the updated cups from quantal-proposed no longer crashes for me on server shutdown. Looks like this is fixed now - thanks, Till!

[Till Kamppeter (till-kamppeter)]

Thank you very much, Steve, so I will post the patch upstream and also commit to the Debian repository of CUPS.

[Till Kamppeter (till-kamppeter)]

Patch forwarded to CUPS upstream as https://www.cups.org/str.php?L4213.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cups - 1.6.1-0ubuntu11

---------------
cups (1.6.1-0ubuntu11) quantal-proposed; urgency=low

  * debian/patches/cupsd-no-crash-on-avahi-threaded-poll-shutdown.patch:
    Fixed crash which sometimes happens on shutdown of the CUPS daemom,
    caused by a wrong shutdown sequence for shutting down the Avahi threaded
    poll (LP: #1034045).
 -- Till Kamppeter <email address hidden> Mon, 15 Oct 2012 19:43:30 +0200