Cups causes LibreOffice unittests to loop in a sbuild

[Björn Michaelsen]

When building LibreOffice 1:5.2.0-0ubuntu1 on a xenial host with a yakkety sbuild, e.g. by:

sbuild -A -d yakkety-amd64 (...).dsc

this loops/busy hangs with unittests. This was working ok up to 5.2.0~rc4 (=final), it is a regression by a LibreOffice dependency, most likely CUPS, which was updated.

I tried to inject debug symbols by running with a: --chroot-setup-command and then run something along the lines of:

 apt install cups
 wget http://launchpadlibrarian.net/278966811/cups-dbgsym_2.2~rc1-4_amd64.ddeb
 wget http://launchpadlibrarian.net/278966816/libcups2-dbgsym_2.2~rc1-4_amd64.ddeb
 dpkg -i cups-dbgsym_2.2~rc1-4_amd64.ddeb
 dpkg -i libcups2-dbgsym_2.2~rc1-4_amd64.ddeb

before starting the build proper, but I got only marginally better debug info. The hanging LibreOffice test processes usually have 2-4 child processes. The parent and one of the childs are busy, while the rest idle.

Here is a stacktrace of the busy child:

Program received signal SIGPIPE, Broken pipe.
0x00002b0a80dd615f in fgetspent (stream=0x11) at fgetspent.c:43
43 in fgetspent.c
(gdb) bt
#0 0x00002b0a80dd615f in fgetspent (stream=0x11) at fgetspent.c:43
#1 0x000000000000001e in ?? ()
#2 0x000055c4976981e0 in ?? ()
#3 0x00002b0a8aff8d20 in ipp_options () from /usr/lib/x86_64-linux-gnu/libcups.so.2
#4 0x0000000000004002 in ?? ()
#5 0x00002b0a8ada9feb in ppdCollect2 (ppd=0x2b0a8ada9bdf <cupsGetDestMediaDefault+415>, section=29, min_order=0, choices=0x2b0a8adad6f3 <cupsFileGetConf+467>) at emit.c:145
#6 0x0000000000000000 in ?? ()

Here is a stacktrace of the busy parent:

Thread 3 "CUPSManager cup" received signal SIGPIPE, Broken pipe.
0x00002b0a80dd615f in fgetspent (stream=0x11) at fgetspent.c:43
43 in fgetspent.c
(gdb) bt
#0 0x00002b0a80dd615f in fgetspent (stream=0x11) at fgetspent.c:43
#1 0x000000000000001e in ?? ()
#2 0x000055c4976981e0 in ?? ()
#3 0x00002b0a8aff8d20 in ipp_options () from /usr/lib/x86_64-linux-gnu/libcups.so.2
#4 0x0000000000004002 in ?? ()
#5 0x00002b0a8ada9feb in ppdCollect2 (ppd=0x2b0a8ada9bdf <cupsGetDestMediaDefault+415>, section=29, min_order=0, choices=0x2b0a8adad6f3 <cupsFileGetConf+467>) at emit.c:145
#6 0x0000000000000000 in ?? ()

When pressing "c" in gdb to continue, both processes stop very quickly again with the SIGPIPE.
Venturing a guess: Is the signal handling of CUPS b0rked? As fgetspent reads the shadow file[1], maybe there are missing permissions or other sandbox issues that CUPS doesnt handle error cases for properly?

[1] http://linux.die.net/man/3/fgetspent

[Till Kamppeter (till-kamppeter)]

Are there "audit" messages in the syslog file (of the host, of the chroot)?

Can you try to run CUPS in aa-complain mode of AppArmor?

Can you run CUPS in debug mode and supply the error_log file?

[Björn Michaelsen (bjoern-michaelsen)]

> Are there "audit" messages in the syslog file (of the host, of the chroot)?

Found nothing that seems relevant on the host. There is no syslog in the chroot AFAI am aware.

> Can you try to run CUPS in aa-complain mode of AppArmor?
> Can you run CUPS in debug mode and supply the error_log file?

Sorry, I dont have time for that. Reproduction steps are noted above.

[Till Kamppeter (till-kamppeter)]

To the developers of sbuild: If sbuild is used on a desktop machine (which is running CUPS) and the package built inside sbuild has a "BuildRequires: cups-daemon" making another instance of CUPS running in sbuild's chroot, could this lead to conflicts like both CUPS daemons claiming port 631 and so the later daemon (in sbuild's chroot) not starting up correctly?

The problem of two conflicting CUPS daemons should not occur on servers as on servers CUPS is usually not running (only on print servers, and I cannot imaging that the build cluster servers are also print servers). But nevertheless we should find a fix to allow developers to use sbuild on desktop machines.

[Björn Michaelsen (bjoern-michaelsen)]

This also broke winff:

(21:06:31) elbrus: Sweet5hark: winff FTBFS and seems to time out on libreoffice generating pdf's
(21:06:34) elbrus: any idea?
(21:06:43) elbrus: https://launchpadlibrarian.net/283037524/buildlog_ubuntu-yakkety-amd64.winff_1.5.3-7_BUILDING.txt.gz
(21:07:10) ***elbrus was pointed to you at #ubuntu-motu by jbicha
(21:17:08) Sweet5hark: elbrus: thats likely https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1616548 see https://git.launchpad.net/~libreoffice/ubuntu/+source/libreoffice/commit/?h=ubuntu-yakkety-5.2&id=28031ef15ac104122180669727c9420bcd51c147 show a likely workaround broken CUPS
(21:19:35) Sweet5hark: elbrus: thus you need to set SAL_DISABLE_CUPS=true in the env for cups not stalling LO in an sbuild
(21:20:57) elbrus: Sweet5hark: ok, but am I correct to say this is Ubuntu specific?
(21:21:13) elbrus: In Debian, the build was OK (in pbuilder)
(21:22:47) Sweet5hark: elbrus: yes, this is likely ubuntu specific. testbuilds up to libreoffice 5.2.0~rc4 were fine, then it broke, likely by a cups update.
(21:29:19) Sweet5hark: elbrus: you might help triaging if you downgrade cups and retry. Using LibreOffices own build as testcase is annoying due to the buildtime. but with winff you might have found a good smaller testcase for our CUPS guys ...
(21:30:16) Sweet5hark: elbrus: FWIW, "in Debian" is somewhat ambiguous, given the number of releases etc. it has. ;)
(21:32:17) elbrus: "in Debian" I mean in unstable... I just uploaded the package several days ago
(21:35:14) Sweet5hark: elbrus: FWIW, I couldnt repro that in a pbuilder, only in sbuild. I dont know exactly was the difference/root cause.
(21:37:59) elbrus: Sweet5hark: I don't have an sbuild setup...
(21:38:14) elbrus: I'll ask ginggs...
(21:38:20) elbrus: or try in my ppa
(21:38:26) elbrus: maybe that is smarter then
(21:39:36) jbicha: winff built for me in a yakkety sbuild last week
(21:40:00) elbrus: it fails in launchpad multiple times
(21:40:11) jbicha: yes I know
(21:40:27) jbicha: I test built locally and it worked, but not in the launchpad builders
(21:40:50) elbrus: jbicha: were you the one that also requested a retry?
(21:41:54) jbicha: last week I did yes
(21:42:22) ***elbrus noticed that the retry was fresher than his own retry
(21:43:07) elbrus: jbicha: you didn't try building it in a ppa did you?
(21:43:23) jbicha: no
(21:43:40) elbrus: well, I will try that tonight probably
(21:56:09) elbrus: jbicha: seems to reproduce in MY pbuilder env....
(21:56:17) elbrus: it's hanging currently
(21:56:27) jbicha: good! ;)
(21:57:02) elbrus: trying again with the CUPS env var
(22:15:03) elbrus: Sweet5hark: with your fix, winff builds in my pbuilder env.... so seems like winff is a smaller testcase for the issue
(22:20:13) ricotz: elbrus, jbicha, it likely works if the host system has a running cups instance
(22:29:42) elbrus: Sweet5hark: thanks a lot, I now have a successful build in my PPA. Will upload to Ubuntu Yakkety shortly
(22:46:18) elbrus: winff is fixed: https://launchpad.net/ubuntu/+source/winff/1.5.3-7ubuntu1/+build/10742736

As per above, this is confirmed in cups (Confirmed) and worked around in winff (Fix Commited).

[taro-k (taro110)]

Hi, I am not sure this is related or not but let me share:

Iceweasel/Firefox-esr: Temporal freeze when print/print preview
https://lists.debian.org/debian-devel/2016/09/msg00255.html

In this issue, browsers' printing function always cause
temporal freeze where Cups is never installed.
Once Cups (and the related packages) is installed,
no hangs any more even after Cups is uninstalled.

The following is the list of installed packages with the name of cups:

% aptitude search cups | grep ^i
i A libcups2 - Common UNIX Printing System(tm) - Core lib
i A libcupsfilters1 - OpenPrinting CUPS Filters - Shared library
i A libcupsimage2 - Common UNIX Printing System(tm) - Raster i

Taro

[Till Kamppeter (till-kamppeter)]

Recently, CUPS in Ubuntu was configured to run on-demand, to save resources, especially laptop or phone batteries. Perhaps one should somehow make CUPS run permanently, perhaps as long as sbuild is doing a build. Please try this.

[Sebastien Bacher (seb128)]

Could be similar to https://bugzilla.redhat.com/show_bug.cgi?id=1366775 which should be fixed with that commit
https://github.com/apple/cups/commit/0ca77b3e89dc1f75c91e1a084dba861e378c6c8d

Till, can you check if we have that commit or should backport it to the 16.10 cups version?

[Sebastien Bacher (seb128)]

that's the upstream bug
https://github.com/apple/cups/issues/4870
"libcups attempted to connect to daemon endlessly if there's no daemon running"

[Till Kamppeter (till-kamppeter)]

Seb, the final patch for https://github.com/apple/cups/issues/4870 is already backported into CUPS 2.2.0 in the cups 2.2.0-2 package in Debian (today in the morning CEST). So now I am doing

syncpackage --force -d unstable -r yakkety-proposed cups

until I do not get

syncpackage: Error: Debian version 2.2.0-2 has not been picked up by LP yet. Please try again later.

any more and then someone of the release team will let get it in right after the beta.

I hope then everything will be solved.

[Till Kamppeter (till-kamppeter)]

CUPS 2.2.0-2 is in Yakkety now. Please test whether it fixes this issue.

[Till Kamppeter (till-kamppeter)]

Anyone can check whether the CUPS currently in Yakkety and Zesty solves this problem? Thanks.

[Paul Gevers (paul-climbing)]

Winff didn't use the work-around for the version build in Yakkety, so it was fixed, yes.

[Till Kamppeter (till-kamppeter)]

Thanks, Paul Gevers, so I a closing the CUPS task now.