upstart gets stuck when start/restart service

[Jianghua Wang]

The problem can be reproduced by the following way:
1. Firstly the restarting services works fine; taking ssh for example
  stack@DevStackOSDomU:~$ sudo restart ssh
  ssh start/running, process 25187
  stack@DevStackOSDomU:~$ status ssh
  ssh start/running, process 25187
2. restart the upstart via the command of "telinit u"
3. we can see there are two /sbin/init processes keeping running;
   stack@DevStackOSDomU:/usr/sbin$ date
   Fri Oct 16 09:58:09 UTC 2015
   stack@DevStackOSDomU:/usr/sbin$ ps -ef | grep init
   root 1 0 0 Oct10 ? 00:00:03 /sbin/init
   stack 19987 25058 0 09:58 pts/29 00:00:00 grep init
   root 27756 1 0 09:41 ? 00:00:00 /sbin/init

4. upstart lost pid tracking for the existing service
   stack@DevStackOSDomU:/usr/sbin$ status ssh
   ssh stop/waiting ---> no pid appended at here

5. then it will gets stuck when restart ssh.
  stack@DevStackOSDomU:/usr/sbin$ sudo restart ssh

System load info:
stack@DevStackOSDomU:/usr/sbin$ uname -a
Linux DevStackOSDomU 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
stack@DevStackOSDomU:/usr/sbin$ initctl --version
initctl (upstart 1.12.1)
Copyright (C) 2006-2014 Canonical Ltd., 2011 Scott James Remnant

[Jianghua Wang (wjh-fresh)]

Some more findings:
1. I think it's apparently that the hang is due to upstart lost tracking on the job's pid.
2. Before the hang occurred, there is a package upgrade. And due to there is libc upgrade (upstart depends on libc), it tries to restart the upstart via "telinit u". But don't know why the restart seems have problem to be finished; we can see there is an additional init process keeping running, where I think it should be a transient status. I guess the expected behavior is that the older init should be replaced by the new init after all things get migrated to the new init. but it seems it's not finished in time and is kept in an intermediate status.
$ ps -ef | grep init
root 1 0 0 Oct14 ? 00:00:05 /sbin/init
root 6647 1 0 Oct15 ? 00:00:00 /sbin/init

$ ps -ef | grep upstart
root 315 1 0 Oct14 ? 00:00:00 upstart-udev-bridge --daemon
root 526 1 0 Oct14 ? 00:00:00 upstart-socket-bridge --daemon
root 530 1 0 Oct14 ? 00:00:00 upstart-file-bridge --daemon
stack 2154 19493 0 02:57 pts/25 00:00:00 grep upstart
root 6781 1 0 Oct15 ? 00:00:00 upstart-udev-bridge --daemon
root 8188 1 0 Oct15 ? 00:00:00 upstart-file-bridge --daemon
root 8191 1 0 Oct15 ? 00:00:00 upstart-socket-bridge --daemon

Need some upstart experts help to dig further. Thanks.