swift_rsync container is in a restarting loop after stack update

[Christian Schwede]

Looks like a stack update stops the rsyncd container in an unclean way - a normal "docker stop" && "docker start" works fine, whereas a "docker kill" leaves the rsyncd.pid file, preventing the container to start later on.

I don't think we need the rsyncd PID file in the containerized environment. As seen in this case, if docker restarts the container and the file still exists, it won't start. From the rsyncd.conf manpage:

"pid file This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file."

It's set in puppet-swift: https://github.com/openstack/puppet-swift/blob/master/templates/rsyncd.conf.erb#L6

So there are multiple ways to handle this:
1. Ensure a stack update doesn't kill a container (?)
2. Remove a possibly existing .pid before starting rsyncd
3. Change the rsyncd.conf and remove the "pid file" setting (only in a containerized env - where should this be handled?)

Thoughts?

[Christian Schwede (cschwede)]

Well, scratch - puppet-swift uses puppetlabs' rsync module, and thus we can't simply update the rsyncd.conf template puppet-swift (in fact it's unused, I'll remove it). I'll add an update_task to t-h-t to fix this issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/523134

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523134
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=570c95741200a31c5e6a3ff36447625e1cb398c7
Submitter: Zuul
Branch: master

commit 570c95741200a31c5e6a3ff36447625e1cb398c7
Author: Christian Schwede <email address hidden>
Date: Mon Nov 27 14:02:42 2017 +0000

    Ensure rsyncd PID file is removed during overcloud updates

    During an overcloud update it might happen that the rsyncd container
    gets killed & restarted, leaving a PID file from rsyncd. rsyncd won't
    start because of this, which results in a container restarting loop.

    This patch ensures the PID file is absent to fix this.

    Closes-Bug: 1734674
    Change-Id: Ie5b28005f1079e432cfca6c31d28295174e06986

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/526998

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526998
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ff8122ec461daa5603fa5d3a38164fb90910c13b
Submitter: Zuul
Branch: stable/pike

commit ff8122ec461daa5603fa5d3a38164fb90910c13b
Author: Christian Schwede <email address hidden>
Date: Mon Nov 27 14:02:42 2017 +0000

    Ensure rsyncd PID file is removed during overcloud updates

    During an overcloud update it might happen that the rsyncd container
    gets killed & restarted, leaving a PID file from rsyncd. rsyncd won't
    start because of this, which results in a container restarting loop.

    This patch ensures the PID file is absent to fix this.

    Closes-Bug: 1734674
    Change-Id: Ie5b28005f1079e432cfca6c31d28295174e06986
    (cherry picked from commit 570c95741200a31c5e6a3ff36447625e1cb398c7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.7 release.

[Bogdan Dobrelya (bogdando)]

A similar bug, but for runtime https://bugs.launchpad.net/tripleo/+bug/1724559