restarting services hangs on systemd-tty-ask-password-agent

Bug #1456789 reported by Joel Goguen
66
This bug affects 13 people
Affects Status Importance Assigned to Milestone
systemd (Debian)
Fix Released
Unknown
systemd (Ubuntu)
Undecided
Unassigned

Bug Description

I have a script run as root from cron that tweaks smb.conf and restarts smbd and winbind. Problem is, starting today after working perfectly up till yesterday, this script now hangs on Ubuntu 15.04. When run from cron, the process tree looks like this:

service smbd restart
 \_ systemctl restart smbd.service
      \_ /bin/systemd-tty-ask-password-agent --watch
      \_ /usr/bin/pkttyagent --notify-fd 5 --fallback

Run from a shell via sudo, same thing except the process tree (obviously) includes the sudo command. It times out after 5 minutes, causing the script to fail. If I manually kill the processes, it fails faster.

The simplest possible scripts that reproduce this, run as root or through cron or via sudo, are:

#!/bin/sh
service smbd restart

#!/bin/sh
systemctl restart smbd.service

Using 'service', this works perfectly on Ubuntu 14.04 and 14.10, and until this morning worked perfectly on 15.04 as well.

1) The release of Ubuntu you are using, via 'lsb_release -rd' or System -> About Ubuntu
Ubuntu 15.04
2) The version of the package you are using, via 'apt-cache policy pkgname' or by checking in Software Center
systemd 219-7ubuntu5
samba 2:4.1.13+dfsg-4ubuntu3
3) What you expected to happen
Services to restart
4) What happened instead
Services didn't restart, 'systemctl' hung

Revision history for this message
Joel Goguen (jgoguen) wrote :

Also unable to restart sssd.

Revision history for this message
Martin Pitt (pitti) wrote :

Does that happen with any service? Or is that specific to smbd and sssd? I suppose 'service' is not the culprit here, and it also happens if you directly call "sudo systemctl restart smbd.service"?

Can you please do

  sudo systemd-analyze set-log-level debug
  sudo systemctl restart systemd-udevd
  sudo systemd-analyze set-log-level debug
  sudo journalctl -b > /tmp/journal.txt

and attach /tmp/journal.txt here?

Changed in systemd (Ubuntu):
status: New → Incomplete
summary: - Unable to restart services from scripts
+ restarting services hangs on systemd-tty-ask-password-agent
Revision history for this message
Martin Pitt (pitti) wrote :

Another thing, can you please give me the output of

  systemctl list-jobs
  systemctl status|head

thanks!

Revision history for this message
Joel Goguen (jgoguen) wrote :

I haven't extensively tested all services, but it looks like anything set up similarly to smbd. Some have different behaviour, such as network-manager which immediately returns with:

% sudo restart network-manager
[sudo] password for jgoguen:
restart: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

But in general it looks like these services all fail. If the NM one is actually something else I'll happily submit another bug against the network-manager package.

The commands you asked for:

% systemctl list-jobs
No jobs running.

% systemctl status|head
● jgoguen-test-carbon
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Fri 2015-06-12 13:40:33 PDT; 11min ago
   CGroup: /
           ├─ 1 /sbin/init splash
           ├─914 /sbin/cgmanager -m name=systemd
           ├─system.slice
           │ ├─avahi-daemon.service

Revision history for this message
Martin Pitt (pitti) wrote :

> % sudo restart network-manager

That doesn't work. "restart" is an upstart specific command. You need to use the init-neutral "service" or "systemctl".

Will look at the logs soon, thanks!

Revision history for this message
Joel Goguen (jgoguen) wrote :

Ahh, yes, whoops. I tried again with 'systemctl restart network-manager' (and, just in case, I also tried 'systemctl restart network-manager.service') and the behaviour is identical to smbd et. al. - it hangs with subprocesses /bin/systemd-tty-ask-password-agent and /usr/bin/pkttyagent.

Martin Pitt (pitti)
Changed in systemd (Ubuntu):
status: Incomplete → New
status: New → Confirmed
Changed in systemd (Debian):
status: Unknown → Incomplete
Revision history for this message
Michael Thayer (michael-thayer) wrote :

Ran into something similar just now. A bit of internet searching suggested<1><2> that it was due to systemd's internal state getting out of sync, and indeed running "systemctl daemon-reload" made it go away.

<1> https://github.com/NixOS/nixpkgs/issues/2584
<2> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774153

Revision history for this message
Michael Thayer (michael-thayer) wrote :

Sorry, "systemctl daemon-reexec".

Revision history for this message
Todor Dragnev (t0d0r) wrote :

Hi,
I have same problem, when running systemctl stop/start from background bash script...

18581 pts/0 S 0:00 \_ su -
18582 pts/0 S 0:00 \_ -bash
18668 pts/0 S 0:00 \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh
18760 pts/0 S 0:00 | \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh
18762 pts/0 S 0:00 | | \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh
18763 pts/0 S 0:00 | | \_ systemctl stop redis_6379
18764 pts/0 S 0:00 | | \_ /usr/bin/systemd-tty-ask-password-agent --watch
18761 pts/0 S 0:00 | \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh

the process just hangs.

I noticed that "systemctl daemon-reexec" just kill somehow last staled process, if I have second staled process.

This is the output after daemon-reexec:

---
Warning! D-Bus connection terminated.
Disconnected from bus.
redis_6379.service - SYSV: Redis is a persistent key-value database
Loaded: loaded (/etc/rc.d/init.d/redis_6379)
Active: deactivating (stop) since Tue 2015-11-24 00:49:05 EET; 8min ago
Control: 17618 (redis_6379)
CGroup: /system.slice/redis_6379.service
├─17263 /usr/local/bin/redis-server *:6379
└─control
├─17618 /bin/sh /etc/rc.d/init.d/redis_6379 stop
└─19422 sleep 1

Nov 24 00:57:53 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:54 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:55 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:56 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:57 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:58 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:59 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:00 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:01 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:02 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:03 offmedia-bms-stage.localdomain systemd[1]: Starting SYSV: Redis is a persistent key-value database...
Info: <<<< finished action /etc/backup.d/30.redis-restore.sh: SUCCESS
Debug: send report to root
Info: FINISHED: 1 actions run. 0 fatal. 0 error. 0 warning.

Revision history for this message
Martin Pitt (pitti) wrote :

I didn't see this myself, but this could have been caused by bug 1565617 which is now fixed. Can you please try if you still get this with Ubuntu 16.04?

Changed in systemd (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Joel Goguen (jgoguen) wrote : Re: [Bug 1456789] Re: restarting services hangs on systemd-tty-ask-password-agent

I was having so many problems with Ubuntu, bugs take so long to get responses, and almost always it's solved by upgrading the package, so I saved myself a bunch of hassle by just switching to Arch.

Changed in systemd (Debian):
status: Incomplete → New
Changed in systemd (Debian):
status: New → Fix Committed
Changed in systemd (Debian):
status: Fix Committed → Fix Released
Revision history for this message
Jon Schewe (jpschewe) wrote :

I'm still seeing this with Ubuntu 16.04.2.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Same here, sample ps auxf output: http://paste.ubuntu.com/26112646/

In the upgrade there was a systemd update at the same time as the other package update, which seems to be the trigger according to the debian bug.

Changed in systemd (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
sascha arthur (sarthur) wrote :

This crap here drives me crazy.

Im having this on an apt-installation of "redis-server".

Ubuntu 18.04

systemctl / networkctl all clean.
kernel is not booted with console parameters (also with console parameters doesnt work)

I tried like everything, even replacing this stupid ..
/bin/systemd-tty-ask-password-agent with /bin/true

Im running everything as root to avoid permission issues and theres no fucking way to bypass this stupid /bin/systemd-tty-ask-password-agent

Im currently having NO WAY to install anything on the system because its stuck on this stupid process...

Fucking get rid of this shit or bring a solution to enter the password.

Already spend 2 hours on it.

Revision history for this message
niknah (hankin0) wrote :

These two worked for me...

systemctl start --no-ask-password ...

or...

service ... start

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.