cups-browsed hangs on shutdown

[Jane Silber]

I have a new XPS13, running an up-to-date 16.04. The shutdown process takes ages - something is clearly getting stuck. It eventually completes, but is (or seems like) several minutes. Happy to help debug if you can give instructions on how to see what's stuck.

[Martin Pitt (pitti)]

Hey Jane! /usr/share/doc/systemd/README.Debian.gz describes some general steps to get some information about this. Particularly, please run

    systemctl start debug-shell

in a terminal, then shut down. When it's hanging, press Ctrl+Alt+F9, there should be a root shell running. What does "systemctl list-jobs" say? There's presumably one job which is stuck at "stopping", which one is it? At this point, please try if this still works:

   journalctl -b > /var/tmp/journal.txt

(it might already fail depending on how far Ubuntu has been shut down already). If so, at the next reboot you should have a /var/tmp/journal.txt, can you please attach this here?

Thanks!

[Jane Silber (silbs)]

Hi Martin

There were many services "waiting" and one that was "running" : cups-browsed.service. I did it several times and this was always the case. Nothing was listed as "stopping", although all were of type = stop.

I've also attached a journal.txt file as suggested.

Let me know if you need more info -

thanks!

[Martin Pitt (pitti)]

Hello Jane,

sorry for the delay (sprint week). I'm afraid I haven't seen this on machines I have access to, presumably this is related to having some actual printers in the office network?

Can you first please try this in the running system:

  sudo systemctl stop cups-browsed

Does that stop immediately or hang?

  → If it hangs, please Control-C it and give me the output of "sudo systemctl status cups-browsed", maybe that has an error message. You can then try "sudo systemctl kill cups-browsed" to kill it for good.

  → If it does not hang, then the hang during shutdown is perhaps related to the network already being down (let's investigate that later).

Does shutting down the machine work after manually stopping cups-browsed? If not, what's the next job that is hanging then?

Thanks!

[Jane Silber (silbs)]

It must be related to having actual printers on the network - it shuts down cleanly/quickly when I'm not in the office. I'll try the above next week when I'm back in the office.

Jane

[Norberto Bensa (nbensa)]

Hi. I did:

   sudo systemctl stop cups-browsed.service

and the command did not hang. Shutting down the machine after running the above command worked flawlessly.

This machine is notebook. I connect to the network almost exclusively using WiFi.

Can I help debug this?

Regards,
Norberto

[Martin Pitt (pitti)]

zoolook: If you have the same symptoms, these would be two interesting experiments:

 * Shut down the network manually (disconnect in NM applet), then stop cups-browsed. Does that work/hang?

 * Run these in a terminal:

   sudo mkdir -p /lib/systemd/system/cups-browsed.service.d
   printf '[Unit]\nAfter=network.target\n' | sudo tee /lib/systemd/system/cups-browsed.service.d/network.conf
   sudo systemctl daemon-reload
   sudo systemctl restart cups-browsed

  This will modify the shutdown order to force browsed to stop before shutting down the network. My hunch is that browsed wants to send something to the network on shutdown, but that is already down, so it times out there. Does shutdown work with that?

[Till Kamppeter (till-kamppeter)]

Is the fix of bug 1579905 applied? Does that fix also solve the problem here? Or is pitti's suggestion of comment #6 also needed.

I think that pitti's approach is also important as cups-browsed needs the network. At least on some systems it communicates with CUPS via localhost:631 which probably would not work if the network is already shut down. cups-browsed needs to communicate with CUPS also during its shutdown as it removes the CUPS queues it created when shutting down. So it is important that cups-browsed shuts down before CUPS and before the network.

[Martin Pitt (pitti)]

Till Kamppeter [2016-06-27 21:42 -0000]:
> Is the fix of bug 1579905 applied?

I just downloaded xenial's 1.8.3-2ubuntu3 and that has that fix (i. e.
the After=cups.service).

Unrelated note: cups-browsed.service has *both* Wants=cups.service and
Requires=cups.service, so the Wants= is entirely unnecessary as it
gets dominated by Requires=. (But this isn't causing a bug, just a
stylistic issue).

> I think that pitti's approach is also important as cups-browsed needs
> the network. At least on some systems it communicates with CUPS via
> localhost:631 which probably would not work if the network is already
> shut down.

loopback should still be up at that time, this is mostly about the
"real" network.

> cups-browsed needs to communicate with CUPS also during its
> shutdown as it removes the CUPS queues it created when shutting down. So
> it is important that cups-browsed shuts down before CUPS and before the
> network.

Right, but that's bug 1579905. Jane's journal clearly shows that cups
has *not* been stopped yet when stopping cups-browsed, so that's fine.

[Jane Silber (silbs)]

1. I am running 1.8.3-2ubuntu3 and the core issue still happens (and Martin says that version has the fix Till mentions).

2. Manually stopping cups-browsed (via "sudo systemctl stop cups-browsed.service") completes immediately, whether I am on the network or not.

After manually stopping cups-browsed, shutdown then completes as expected.

3. Separately, doing the steps Martin recommends in comment #6 to modify the shutdown order has mixed results. 2 out of 4 times shutdown quickly as expected. 2 out of 4 times, the system hung at the point of having the Unity interface and the shutdown window on the screen (i.e., it didn't even get to the point where it was originally hanging, which was on the logo splash screen). I was just booting and quickly shutting down, so this strange behaviour could be related to that.

[Till Kamppeter (till-kamppeter)]

Some more hints:

cups-browsed uses avahi-daemon for printer discovery. It does not matter whether avahi-daemon shuts down before or after cups-browsed. If avahi-daemon shuts down first, cups-browsed removes its queues triggered by them going away from DNS-SD, if cups-browsed shuts down first, it removes the queues due to its own shutdown.

cups-browsed uses D-Bus to communicate with avahi-daemon (avahi-daemon informs about devices in the network appearing and going away) and with cupsd (cupsd reports activity on the print queues via D-Bus).

cups-browsed also communicates with cupsd or cups-browsed on remote machines via the regular network (no loopback) when the legacy CUPS protocol for printer browsing and/or broadcasting is used (in cups-browsed.conf at least one of "BrowseProtocols", "BrowseRemoteProtocols", and "BrowseLocalProtocols" has "cups" set.

So for me it seems that both regular network and D-Bus must be up and running while cups-browsed is running and cups-browsed has to shut down before these two (and to start after these two).

[Till Kamppeter (till-kamppeter)]

Another hint:

https://bugs.linuxfoundation.org/show_bug.cgi?id=1266

Comment #2:

I've seen it. When I remember right it happened here on all systems when there
was the default setting "BrowseRemoteProtocols DNSSD,CUP" enabled.

Changing it to "BrowseRemoteProtocols DNSSD" fixes the hang at shutdown for me.
There's no old cups server here running offering printers via old CUPS
protocol.

[Till Kamppeter (till-kamppeter)]

Note that cups-filters 1.8.3-2ubuntu3 does not contain the fix ("Requires=cups.service" in the cups-browsed.service file). So in Xenial the bug is still there. It is tracked in bug 1579905 and there I have proposed a SRU to introduce the fix into Xenial. Please follow up there and test the SRU as soon as it is available for testing and give feedback there to make the SRU getting into Xenial's official updates.

Note that if you are not in a network with printers (comment #4) cups-browsed does not create local print queues which it has to remove on shutdown. So it will not hang if CUPS has shut down before cups-browsed. Also removing a protocol from the BrowseRemoteProtocols entry in cups-browsed.conf can lead to the case that cups-browsed does not discover any printer and therefore does not need to remove print queues on shutdown (comment #11).

Marking as duplicate of bug 1579905 ...