Swift replicator dies but leaves behind pidfile

[Jill Rouleau]

Seeing this on 2 Mitaka/Trusty clouds at the moment. Swift is 2.7.1 and 2.7.0.
Juju 1.25.10 with 16.10 charms
Juju 1.25.6 with 17.02 charms

Replicators frequently fall over. Sometimes we can map these to hooks running and other times not. The main problem we see is that a pidfile is left behind so the service can not be automatically restarted without manual intervention to issue a hard stop first, or rm'ing the pidfile.

root@mycloud-os-2:~# ps auxfww|grep replicator
root 4556 0.0 0.0 10480 2072 pts/3 S+ 00:20 0:00 \_ grep --color=auto replicator
swift 8586 0.1 0.0 84848 27364 ? Ss Apr05 93:13 /usr/bin/python /usr/bin/swift-account-replicator /etc/swift/account-server.conf
swift 14251 0.2 0.1 1582612 45848 ? Ssl Apr05 147:59 /usr/bin/python /usr/bin/swift-object-replicator /etc/swift/object-server.conf
root@mycloud-os-2:~# ls /var/run/swift/
account-auditor.pid account-replicator.pid container-auditor.pid container-server.pid container-updater.pid object-replicator.pid object-updater.pid
account-reaper.pid account-server.pid container-replicator.pid container-sync.pid object-auditor.pid object-server.pid
root@mycloud-os-2:~# cat /var/run/swift/container-replicator.pid
1352
root@mycloud-os-2:~# strace -p 1352
strace: attach: ptrace(PTRACE_ATTACH, ...): No such process
root@mycloud-os-2:~# service swift-container-replicator status
swift-container-replicator start/running
root@mycloud-os-2:~# status swift-container-replicator
swift-container-replicator start/running
root@mycloud-os-2:~#
root@mycloud-os-2:~# stop swift-container-replicator
stop: Job failed while stopping
root@mycloud-os-2:~# start swift-container-replicator
swift-container-replicator start/running
root@mycloud-os-2:~# ps auxfww|grep replicator
root 29776 0.0 0.0 10480 2068 pts/3 S+ 00:59 0:00 \_ grep --color=auto replicator
swift 8586 0.1 0.0 84848 27364 ? Ss Apr05 93:16 /usr/bin/python /usr/bin/swift-account-replicator /etc/swift/account-server.conf
swift 14251 0.2 0.1 1582612 45848 ? Ssl Apr05 148:04 /usr/bin/python /usr/bin/swift-object-replicator /etc/swift/object-server.conf
swift 26898 0.1 0.0 98708 30600 ? Ss 00:35 0:02 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf

[Jill Rouleau (jillrouleau)]

This continues to be a recurrent problem. Is there any additional information we can provide that would help?

[James Page (james-page)]

Moving bug to swift package, which provides the upstream and init scripts; that said they all call swift-init and it would appear that the process tracking under both upstart and systemd is foobar as neither init system is actually tracking the process ultimately run.